Why, after all this time, is the testing group so small? Why is it just internal playtests and Evocati? Why don't we just open this up to everybody?

First off, we get feedback internally from the people who make Star Citizen because, A, they don't on a day-to-day basis play it. So it means that we can utilize people to come and play the game and it gives them a bit of fun. They can come and enjoy playing, shooting each other. That's fun. It's good to have a bit of a blow, a bit of steam off in the day. And everybody who is a game developer, I hope, I pray that they like games. And if they don't, they should give up the position. It should be mandated that you give up your position to somebody who does like games. But anyway, so on that front, it's good that they're the first bastion. They're the first gateway for us to be able to get feedback from. And the people are super passionate about Star Citizen and they're super passionate about what we're doing. So it's an incredible amount of feedback that we can get back. So they're the first gateway that we have. So we go, okay, let's see what the 700-odd Star Citizen developers say about it. Then when we go to Evocati, obviously Evocati are our, they're the top gun, should we say, they're the 1% that are able to come and help test the game mode. And they understand that the performance might not be the best, that they might run into crash bugs, that there might be these issues that they're going to run into, or they might have to jump through hoops or copy their account over or do X, Y, Z to be able to play the game. And it is a specific window on a Saturday or a Sunday to go and do that. And that's an incredibly valuable asset to use to be able to get that information. Now, as Sean already touched on and I touched on previously, the game mode that we're making, in terms of the environment that it's in, to understand the fun, to understand the gameplay, I think the performance has to be at a level that you can at least see past it. You can at least go, okay, I see what they're trying to do here, but I'm walking over a bit of hot coals to get there. So, oh, maybe the audio is not that great or the balance of the weapons is not that great, but they know that the balance is intrinsically not that great in this area in Star Citizen, or they know that we've disabled missiles or whatever. And I think when you have to caveat a lot of what you're trying to achieve, like healing's not in there or mounted guns are not in there or the tank's not in there, I think when you have to caveat a lot of the gameplay, going to a wider audience is always a risk. A risk because the reason is not why we're going wider. The reason is why do we do these tests in the first place? Well, we do these tests in the first place to gather information, to gather, like, is the map balanced? Advocacy versus Supreme, do we get roughly equal amounts of wins? Yes, we do. How do people feel about the balance between air versus ground versus ground versus ground? How do we see the progression? Are people actually playing through the map? Do they understand how to play the game mode? Is it doing what we want? Are people utilizing all of the area that we're giving them to play? And I think when we have that information and we can distill that information, or above everyone, what is the mean FPS of all the users that are playing it? Now, as long as we've got a big enough sample size to be able to ascertain that information, then that's all we need, because that validates what we want. There's no reason for us to do a test just because we should just do a test. We do a test because we want to gather information back to make sure that, A, we're going down the right road. B, have we made improvements with the performance? C, is the map and the creative areas of the map, are they hitting the right boxes? Are they doing what we're trying to achieve? And that's why when we go to Evocati, I think we get that information. Now, we could go to wider PTU, but I know personally, I would rather have slightly better performance before opening up to a wider audience. And we saw that with Evocati. We saw the litmus test where when we went to Evocati the first time around, performance was poor and it prevented us getting back information that we needed to progress the mode, because the only information that we got back was performance is poor. We did address that. We pushed another patch out like four weeks later. I think this was way back when. And then, lo and behold, we did get a load of feedback back, like, okay, I'm not sure about the spawn screen, or how does this work, or what? And we got some really good feedback to understand what we needed to do next. And I think, so, we will always advocate going wider to the PTU. I think there's no reason for us to go live live, because if we go live live, it's just dropping it there. We're using it as a tool to learn and iterate and understand what happens when we make these changes. And Evocati is giving us that information and internally is giving that information. We've done quite a few tests where we've gone, actually, you know what? We've had quite a lot of information from internal. Let's not go to Evocati. Like, we've already canceled one Evocati playtest because we had enough feedback internally where we were like, okay, we feel like we should address this before we go live. And that's not just performance related. So then we can iterate on it. So in terms of going wider, that would be the honest answer, is that we've got the information we need before we go to the next step.

Before we feel like we can go to that next gate, we still want to take care of a bunch of stuff, and it has everything to do with feedback. And those stages, each one of those stages just opens up wider, right? So like the internal company test, like he was mentioning, we usually get about 200, 300. It really depends on the time and what else is going on also and what our QA is doing. But we get, again, 200. When we went to Evocati, the last one anyways, we ended up with 706 unique users, 161-some-odd rounds. And mean frame time was actually reasonable at this point. So it was sitting in the 30s, which is great. But the last thing we want to do is just dump out a game mode and do whatever. And I've seen the messages. I know people are worried that we're just going to dump out the game mode and go, good luck, have fun with it. This is not what we want to do. One, we want to use it as a tool as we have been using it, but we don't want to just put it out and let it die or just leave it to not be updated or to not do anything with. So we want to make sure we've got our plan well put together long before that actually goes out. What's really important is that we're hitting our stages. And the point is, my expectation even from 18 months ago was that we were going to be past these stages long ago. But you've got to be honest. And again, we're not in a huge, huge rush on this. It wasn't ready. It just wasn't there. Performance and some of that same stuff. Rich nailed it, though, too, is that a lot of the feedback that we go over now at this point is actually properly about the game, which is great. Now we're seeing real feedback rather than, oh my God, performance is so brutal, or all my inputs are super delayed. And there was lots of feedback that seemed to be about the game, but so often it was actually just that the poor guy is running at 15 or 20 frames a second. So clearly having it running at a reasonable frame rate gives real feedback.

And to be honest, when you go wider, the barrier to entry becomes higher. So for example, any product where you have early adopters, early adopters are willing to walk over more coals. I mean, for example, if we just released Star Citizen right now as a fully fledged game, finished 1.0, we'd get so much critique, because the expectations of players going in are not what they expect of a finished retail product. Whereas because we're pre-alpha, people can go, okay, this has gone out. It's got bugs. I understand that it's got bugs. Those bugs are frustrating. It doesn't make it any less annoying. But it's the appreciation that, okay, yeah, I understand, I can move on. And I think when you get something where this is hyper-focused and it's very comparable to other games, it's very easy to just look at the barrier of, oh, controls feel sluggish, it doesn't feel great. And it's like, well, we know that Evocati understand where we're at with what's coming from the technical side of things for Star Citizen, and that it isn't just this bespoke game mode, it is a real planet. You can just fly in one direction and circumnavigate the planet for the whole half an hour of the game mode, and you'll probably never circumnavigate it because it's that big. We don't cheat it. It's the real deal. So any benefits that we're making to Star Citizen benefit Theaters, and any benefits we're making to Theaters benefit Star Citizen. So we want to make sure that it's playable, that people can understand what we want before we start ripping the plaster off, shall we say. I wouldn't be comfortable releasing Theaters to a wider audience until the performance is respectable, that you could boot it up on a mid-spec machine and not worry about frame rate. Because frame rate affects so many things. It's not just about, oh, slideshow, because it very rarely is, right, even if it's above 20. It's about the feel. It's about your turning. It's the feeling of going into ADS, going to the reload, or you feel unfair when you die, because you think performance has impacted how you've played, and in fact it does. So I think performance has such a huge layer on everything that that is the ultimate. And to be honest, there is no when until we basically fix the performance and the net sync and get that.

And we've also got a minimum set of things we want to ship with. A good example is that with Crossroads of Crime, if you've played it within Evocati or you've seen videos of it, right? It's on a planet, it's very, very infantry focused, like Richard said earlier, 70-30. I want to have at least two scenarios when this thing rolls out. And it's more to go to the tastes of the people that are involved, right? We have basically both sides of it. We could do a 50-50 one, and then we wouldn't really know. But I think having one that's based in space, that has a lot of ship combat going on, on top of then Crossroads of Crime, which is all ground-based, mostly ground-based anyways, and FPS heavy, is an important component of it too. So we've definitely got that minimum set as well as the features, because most of the feedback comes down to squad gameplay, healing, some of the stuff that they've got used to in other games, as those features come on. That just makes all the sense in the world to be putting that in.