Star Citizen Live: Q&A Tech Talk

Well, it's an interesting patch, let's just say that. So, 3.24 has a lot of new uses to some very large systems, right? And so, some foundational systems, yes, systems that have been in the game for a long long time.

Okay, getting their first update.

Yeah, exactly, and so it's been a bit of a rocky release, simply because there are some issues that happens when more players are present that affect core flow of the game, right? So anything that affects you retrieving a ship or, you know, leaving the hangar or things like that are, you know, instantly high priority. Some of them work well, you know, on internal builds, and then when we get real players on it, there's always something new. In this case, for 3.24, there's a lot of issues that happen related to cargo elevators, freight elevators, the new hangars, and net flow. And so we've been hot fixing quite a bit, figuring out what was going on, some missions that work for the same kind of reason, and so we've been kind of analyzing data and hot fixing.

You won't even say the name ATC.

Well, yeah, so let's just say that there, like I said, there are some systems in the game that are complicated, right? That date back to a very long time. And ATC is at the core of it, right? It's one of them.

Show me on the doll where ATC touched you.

Oh my god [laughter], all over. But the, and so, yeah, ATC, Transit, you know, are systems that, you know, there are any issues with these systems are debilitating instantly. And so, you know, it's not a case of if it works 60% of the time, it works every time. It's a case of if it doesn't work 1% of the time, then it's very high priority to find a fix. So we've fixed a lot of stuff in the last couple of days. There's more coming from the gameplay teams, obviously. Some changes to how we deal with obstructions in the hangars are coming as well, which has been one of the most problematic issues. Like the checks on that side were a lot too, I like to say, hardcore, which, you know, would cause a lot of side effects. It caused some hangars to break. So, you know, we're in the process now of changing all that stuff to release a little bit the hold that ATC has over cleanup and, you know, how it prevents some stowing. So there's already a bunch of odd fixes that went live for that. So hopefully that will alleviate a lot of the issues people have been running into with hangers specifically. But otherwise, like 3.24 is a super interesting patch for a lot of reasons. Also, because we've introduced new metrics to our Star Engine that have launched in 3.24. So we have now a lot more information about the state of the game servers and how they perform. So we've instrumented a lot of things in the engine. And to be clear, we've had metrics before. This is more about long term metrics that we track over a long period of time. So that allows us to really correlate important events, right? Like an area streaming in causing problems, certain game components costing us, you know, frame time in certain areas. And so I like to say that these new metrics are like if optimizing the game is like a large cornfield. The tools that we used to have to do this were very optimized, very great microscopes, right? But now with these new metrics, we have a radar that tells us exactly where to look for and drill down. And so I'm hopeful that with 24, we already got a couple of reveals just seeing, you know, the live release roll out with these metrics. So I'm hopeful that that's going to bring forward a lot more improvements. So that's why I'm excited about it.

JH:It's one of those things where it's hard to see it now, but there's so much of what we're building towards with the way that freight elevators play into Pyro and stuff that you're going to see at CitizenCon later this year, next month, six weeks, are all based on stuff that's in 3.24. I understand it's rough now and stuff, but it is a foundational patch that will be built upon and will bear fruit in future releases, all things.

BB:Yeah, one of the things that maybe I don't know if we've communicated this way. I mean, Chad probably did outline this in one of the insights of Citizen, but it's that the tech behind the way the hangars function is actually an instanced interior, right? This is the first use we have of the instanced interior system, but you can imagine what we'll be able to do with that technology moving forward. You know, I know that some of you guys have encountered ghost sounds when you go in your hangar, like you can hear other stuff from others. That's because in reality, the hangars are super posed on top of each other and instanced in place. And so when you go to your hangar, there's some, I'll just qualify it as black magic rendering calling that happens that you're basically in your hangar, that's what you see, right? Now there's a bug with audio, that's why you're either you've heard some audio from the other interiors, but the reality is all these hangars are actually on top of each other, which is when you start thinking about it, it's a bit weird. And then when you request your ATC gateway, that's when your actual instances is moved into place so you can exit and enter. And so that tech will be able to use that throughout the universe for a lot more things for instancing interiors, which is super key for a lot of the gameplay we're trying to do.

So yeah, later in the spring, end of spring, we did a couple of good tests that gave us a lot of good data, right? So what we've been trying to do on the meshing side is be a bit more pragmatic and methodical in how we approach the development. And so these tests really gave us a lot of data, right? Because of the way we conducted them, we were able to try different mesh configurations, look at performance specifically on the hybrid, which is the replication layer of service. And to really be able to hone in on what we need to improve in order to get the scale that we want. And so what came out of these tests in the spring was that there was already things that were touching on the performance of the hybrid and the game server communications that needed improvement. And so this is what we call the components called the NMQ, the network message queue. Now there's a lot of chatter online about, you know, there's a replacement thing and it's called, is it RabbitMQ, whatever. No, so we're not even in that realm. We're talking about really the networking component that transmits data between game server and client, game server and hybrid, hybrid and game servers. It's like this, think of it as the communication library underneath it. So like all games, that technology is using UDP transport to get, you know, eventually consistent updates out as fast as possible. Now, the network message queue had a couple of issues when it got a lot of data in a very short period of time. So these spikes would cause, and that's what we saw in the final server meshing tech previews, was that when large areas would stream in and a lot of players would join in and leave at the same time, the network message queue could go into a tailspin where, think of it as a drain, right? And you're trying to put stuff in the drain. And at some point, if you exceed the bandwidth of the drain, then water starts filling up. Now, the way that manifests itself in the game is interaction delays. So, you know, pressing F trying to get in the ship and it takes seconds before the system reacts, powering on ship takes forever. You know, now I'm not talking about the one offs because you can have interaction delay in your game based on your connection. That can absolutely happen. So it's influenced by your machine, your connection. So if you're playing on a 486 without the turbo button on a satellite connection, you know, you might get interaction delay, right? But in this case, we're talking about widespread, like when an entire shard gets into that state, right? So that's what we saw in the tech preview. So that we had to go a little bit back on the drawing board and actually fast track one of the refactors we wanted to do, which is the introduction of a replacement message queue. Now we call it replication message queue, replacement message queue kind of, you know, we've basically called it RMQ. And so this RMQ is a new implementation of this that has more abilities to adapt to higher bandwidth and react to these large message pushes that happen when we start to stream in large areas or when a lot of players are coming in and out of the game rapidly. And so what we're testing now is RMQ in a server mesh setting. So for you guys who don't know, 3.24, half of the servers on live currently run RMQ. The other ones are running NMQ still. We've been gathering data across both of them to see the performance improvements, the gains. So far, those have been really good, a lot of really good improvements. And so now we want to test this in a server mesh setting. So what changes in a server mesh setting is that now the hybrid is going to manage more than one game server that simulate these key areas. And so there's more traffic going through the hybrid for this at key moments. And so that's what we want to test with this next set of tech previews. So what our hope is, is that the issues that plagued the 800 players last time where there was a lot of interaction delays. Also, some of those tests had a lot of game server crashes at the time that we did the test. But in this case, we're hopeful that all these interactions delays will be basically gone from this bandwidth side of it. So that means we can get to bigger shards and the experience is going to be enjoyable across that. The thing that actually did work really well in our spring test was the actual server meshing tech. So authority transfer, management of territories, all that stuff was fantastic. A really big success. But we did need to take a little bit of a sideways step to optimize the RMQ, which is now done. And also getting mission tested right now by you guys on 3.24.

Yes. Overall, we track some of these interaction delays. And you'll see this is a common theme that I've been pushing quite a bit is that we need to measure a bit more of these things. And so on the long interaction, we've been looking a lot at the amount of events, duration of these events, which players get them. And so we're able to see a noticeable improvement on that side. There are still some interaction lag happening, but they're not systemic anymore. It's those other causes that you mentioned. Yeah, exactly. And so that's cool. The thing that's also nice about this is that we have the ability, you probably noticed, especially with 3.24, that we have a new set of ways to deploy and canary technology on live to mission test it. So not only are we doing tech reviews when it's necessary, but we also are throwing tests. So sometimes you might end up on a server that's running something new and you don't even know about it. But we know about it because we're looking at all these metrics to make sure that the new tech works as expected. And if it doesn't, we can backtrack and stop the canary, things like that. So we've been doing a lot of that for the entire 3.24 cycle.

The interesting thing is that currently for the servers running NMQ, we're not seeing heightened interaction delays. I guess just comparatively between them. We do want to keep going at it, so we're being prudent. We started with a couple of shards, then we moved on to a quarter, then to a half. As this is more stable, it's very possible that within the release of the next couple patches, all servers will be running RMQ. At this point, there's not much stopping that from happening. It's just prudence to make sure that there's no new issues that would come on. It's like if you replace the netcode for an entire game. There might be side effects. And then the typical invisible player, new crash on whatever, bad handling of slow connections.

It's responsible not to just switch everybody over.

Exactly. So NMQ does work well. We know where it falls down and at what moment. We can also identify the shards that do run into this. There was a bit in 3.23 of an accumulation in certain areas that we noticed. shards that were a lot older had this accumulated data. And it's not persistence data necessarily, but it's based on the state of the world where it would transmit more data and that would cause a heightened state of interaction delay. And so that really clued us in a bit more that our plan to do the RMQ was the right one. [...] Our goal is to roll it out to everybody as soon as we can. That's for sure.

So the network transmission layer, right? You know, game server, game client, client does interpolation. So as your ships or your actions move, your actor, so your character and your ship, when you're actually piloting a ship, the client does interpolation about your movements. So it's trying to predict what the server will say. And then the server comes in and says, yeah, that's what it is. And that comes from the network queue. So obviously when we have congestion on the network, it's going to cause some of this. But it's also server FPS, of course, has a very large effect on this. Right. And so desyncs are a combination of factors. Right. That when you see a desync, it's, you know, could be because the server is simulating a piece. And he's back in time a little bit. So he's skipping time steps. Well, he's not skipping time step. It's not the right way to do it. I'm trying to make it simple so we don't go too deep into the weeds. But think about it this way. If the server is not running at a very decent frame rate, then when those frame rates dip, the effect will be some kind of level of desync for certain kinds of items. And so with the RMQ, we're hitting on the stuff that's related to the network transmission. This will help definitely. And with the new metrics in 3.24, that helps us to really hone in on what is the cause of the cost over time on frame time. Where is our frame weight? What resources are being used? What resources are being allocated? So the thing that might not be clear when you look at the game. So the Star Engine is really good at streaming stuff in dynamically and streaming it out dynamically. That's the key feature of the engine. But that also has a certain level of cost. So depending on what players are doing in the game, the servers might be doing just fine. Right. You know, certain areas are just super easy, but then other areas stream in and then they become problematic. And so that's what we're looking at with those metrics is to see what areas and what entities in those areas are costing us the most so that we can optimize the things that really affect those desyncs in that gameplay, literally. So right now, the improvements we're getting there is several engine fixes or optimizations, I should say. And then the network queue replacement is going to help on these things quite a bit.

Well, so so you're right. Like our our game servers do perform a lot of physics calculations and a lot of physics simulation. That's the nature of the game. Right. You know, thrusters, ship simulation. Like we we track so many types of geometries in the physics engine. It's it's it's crazy. Like we have a really great physics framework in the engine. For sure, like the the most immediate gain there is going to is server meshing for sure. Like we've seen in our play test significant frame rate increase across the board for areas because you have less physical entities. Right. In general, so that means less geometry, less allocations, less simulation. You know, think of the server frame rate as a as a bunch of phases. Right. And it's the engine is extremely well multi-threaded, meaning that we can do a lot of things in parallel. But the challenge there is always determining what we can do in parallel that doesn't have impact with other things. So what can we actually simulate that doesn't depend on anything else? And so that's true of entities. So when we do an entity pass or what we call an ECU update. So that's every frame. And it's true of physics. So and it's true of networking. So all these phases at some point need to wait at the end of a frame for everybody to be completed before. And so obviously this is all bound on entity count and also physical entity count. So server meshing is the from the top is going to reduce that count by a lot because you have way less areas to simulate. So less entities to simulate means, you know, that the overall wait time across the frame will be better. And so that's the first gain for sure. Right. And then we can control that granularity as our machine technology moves up. We can control the zones, the size of the zones that we want to simulate. Now, that doesn't mean that we can't have an area that's filled with something. And it's going to have a high, you know, either network or engine cost or physics time cost.

Oh, yeah, because you know, you know, when we increase the player counts to 800, all 800 are going to go and cram themselves into the same 890 jump.

Of course, of course. And so that's a bit why you're going to see some jitters, especially, you know, with cargo. We're putting a bit more on the gameplay side. We're putting a bit more of the physics on the spotlight because we're using them to move, you know, with tractor beams, moving boxes, things like that. And so, of course, we're looking at improvements. But the first one, well, metrics has actually already given us quite a bit of very interesting things to look at. And so that's, I'm very hopeful that our radar is talking about there is going to allow us to find even more optimizations there that we can do. There's always optimizations, guys, by the way, it's always a thing. But server meshing will definitely help quite a bit. We've seen that in the last test. Like, you know, even with the very large shards, servers were running on average much higher frame rate than our current game shards. So I'm really hopeful for those gains because that's going to be a game changer for the game. Like, obviously, performance for the game. You know, if we gain, you know, one millisecond on frame time, that's one millisecond directly to your gameplay. So for sure, like server optimization for Star Citizen is at the top of my mind when I synchronize with game teams.

Correct. There's also, obviously, content optimization. So, you know, like you mentioned, there's always more than one cause, right? So optimization is, when we talk about optimization, because you mentioned one thing in your question, which was, oh, do we just get bigger servers, right? There's a limit to that, a limit that we've, you know, I don't want to say we've reached it. We're not there yet, but there's still a server size that makes sense, right? In terms of cost, cost per player and operations in general, right, of something that makes sense. And so when you do optimizations, right, you need to have a box to fit in, right? So if you're saying our budget frame is X amount of milliseconds, well, then you need to optimize towards that and you need to touch on everything. It's never just one cause or one thing. And so, obviously, how you multi-thread, how you parallelize the work across every phase, so that touches on every team. But then there's content as well, right? Like a good example is the first time we put cargo boxes in the game, I forget which patch at this point. I'm a bit losing track, but the actual one SCU box was composed of like 30-some entities. It looked really good. But, you know, when we looked at it with the scope, it's like, okay, yeah, no, we can't do that. And so content optimization is the other pass of this, right, which is reduce the entity counts, try to be smarter about, you know, how we construct objects, object containers. Streaming is also one, you know, how we stream in things. So the other area of battle on the server optimization front is once you have server frame rate, so you have entities loaded, you're simulating them every frame. That's one area to optimize. The next one is putting stuff in and out of that simulation. So streaming data in, streaming data out. And so there's these phases that every entity and game component go through. And those are, you know, these components are written by game developers, and they do different things. We have, I think, over a thousand game components. They're not all active, but registered, roughly a thousand different game components. And so each one of those have seven phases they go through in their lifecycle. Two of them are really important. It's they called register and unregister. And so this is when a game component that's attached to an entity. So entities of components and entities getting loaded. We need to run the phases on the components so that those components, when they registered, that's where they do, oh, you know, I'm a security component. I need to register with the security system and let know of the AIs in the area, what's going on and things like that. Those happen when a spawn batch appears. And so the, what people refer to as the spawn queues is when the server needs to spawn things, right? That is a queue that is basically processing these phases to put the entities into the engine for simulation. And so that's the other area of battles, optimizing these phases so that the spawn is faster and it's sharper. And despawning is the same, right? So you finish a mission, you leave away, we stream out, we need to get this out, right? And so that's the other area of optimization in battle that also has an effect on gameplay that we're really diving into with these new metrics.

Yeah, I'm happy you brought up the content optimization in addition to feature and service optimization. One of the recurring criticisms, sometimes valid and we deserve it and sometimes not, is how often we rework a thing. There's been three or four versions of the Freelancer, for instance, and there's a really good reason for that one. And that's because we discover inefficiencies in the way it was built the first time. Sometimes because of fault, sometimes because there just wasn't a better way to make it then. And there is now. So revisiting that, reworking that allows us to make it more efficient. The Freelancer famously, gosh, if old time backers will remember, I've told this story before, but if you've heard it, I'm sorry. The second version of the Freelancer had 200 rivets, little rivets that went around the panel lines. And the artist who built it for us, it was a third party client at that time because it was still early days. They copied the rivet from the concept model instead of, I don't want to spend time remaking a rivet. And then just copied and pasted a concept rivet over the entire ship 200 times. And so a single Freelancer now had a greater resource cost than the entire Idris in Squadron 42. And folks for the old hanger module days will remember, if you owned all four Freelancers and you loaded that up when they got remade, your hanger module was a slideshow. And stuff like that.

I remember also there was a thing with the original Cutlass Black. Yeah, first Cutlass that if it would be in your view, that was it. Like that thing was like, you know, I don't know how many polygons it was. It was a crazy like assembly originally. But yeah, content optimization is a big part of this. Right. And it's also the hardest because the universe is very large. Right. So when we talk about content optimization, absolutely. But we're talking about, you know, harvestable asteroids, plants, AI characters, your character, like everything, right, needs to be looked at under the scope to go like, is it under the entity budget? Is it, you know, draw calls? Because the client performance is also important, right? Like we're, you know, we're talking about server stuff now, but the, you know, your client also needs to render that stuff, right? And so content optimization is a super big part of this that, you know, we have to get a lot better at in general for, because our approach is always to make things look right and they do look right. Right. But then we schedule time to optimize it. But sometimes, you know, we have to, because of the nature of our development, we have to put it in one patch and then optimize it in the next one. And so that does cause some frame drops at some points.

We constantly make improvements to our level of detail system and stuff like that.

So in terms of the item recovery in general, there's been a lot of discussions in 3.24, which was, to be honest, very interesting for myself and the design team mostly to read, was the different opinions about the item kiosk and no drawer access to inventory. I mean, this is still going to go on right now that we have at least one release done with the current system, but I'm not saying this is the final one at all. But that's one of the reasons for why the item kiosk exists also, right, was to basically give you guys access to get your items back, right? And so I don't want to say when the feature is actually going to drop because I don't really know specifically when, but I can tell you that the adding all those kiosks in every location, getting them hooked up, you know, this is and getting the actual recovery part, we have to take it out of 3.24 because obviously 3.24 had to go at some point, right? And there was more pressing, you know, development and issues to solve with that. So we have to make that decision. But the actual kiosks themselves are already in there. I mean, you see that in the first now they're usable. And so that means that that feature is in a good place now to be implemented and added in the patch. I can't tell you which one, but I can tell you that that's we talk about that stuff all the time about, hey, when are we going to close the loop on this? When is it going to land, right? In terms of the database work too on the back ends, that's already ready to go. So it's really about, you know, when can we get cycles from the game teams to get that in there? And I do agree. That's for me is also a bit that's a pet peeve of mine, like being able to get your items back, especially when they're unique is a super key.

So, yeah, I mean, I use the term pledge item recovery, but it's really unique item recovery. That's the thing.

BB Yeah, yeah, it's, you know, the ones that you only have one of and that are important. Now, that said, there are plans, you know, for how this is going to work for Star Citizen, not just during the development, but as a whole, right? Because, you know, there's more and we're going to discuss about this. I think it's it has gone a little bit with some of the things we're going to be showing, but there is a larger plan on how purchased items are going to work and how that ties up to the gameplay of the universe. And so there's going to be more updates on that for sure.

I mean, you know, ultimately it's software, right? So for me, I'm a bit of a natural there. It's been a crazy ride for sure in the last year going from, you know, I'm going to use general numbers here, like 100, 150, you know, maybe engineers in general, to like, you know, the amount that we have at CIG and the different teams. The thing that I find the most satisfying is that we have so many incredibly talented engineers at CIG, like from our engine team, graphics, AI, audio, tools, R&D groups, gameplay teams, you know, VFX, VFX programming, all these, like our teams are so talented. It is crazy, right? And so it's really an honor to have the opportunity to try to make their lives better, right? Improve the internal processes that we have for our engineers, make sure they have career paths at CIG that they're fulfilled, that they work on things that matter, right? So one of the big roles of a CTO is to make sure that we're working on the right things, right? That we're not, you know, just developing and then hoping for a result, that we're actually driving for the results that we want, making the right calls across that. So for me, it's been very, very fulfilling. Also very difficult because my, you know, like the context switch is crazy, right? Going from all these subjects, right? But very, very fulfilling. And I'm so proud of the team that we have. Our engineering teams at CIG are our best in class, right? They're really making miracles with a smallish team, like in terms of the pure engineering teams, like making an engine, building the tech that we need for this game. I mean, the level of talent is really incredible. So I'm, it's been roughly almost a year now, you know, that I've been, I've picked up this role and it's been only, you know, positive. It's also been really positive, I think, for the Turbulent family in general, that join in the CIG family overall. It's made communications easier. It's made our missions easier to communicate. You know, it allows also the cross-culture pollination, things that, you know, Turbulent do well, things that CIG do well. We try to get the best out of it, try to influence each other out of it into one big family. So it's been, it's been a really good ride so far. I really, I'm really enjoying it. It's really, really cool. Plus, I love this project. Like, you know, there's nothing more that I want than for this, for all of our tech to come together and then to really make this game the best thing that can be for them. So it's, it's for me, it's like, I'm just ecstatic about it.

It's been nothing but positive. It's such a bullshit executive answer.

No, no, hang on, hang on. Don't say that. What's, what's been difficult is, if you want the downsides of it, is really catching up to speed to all the disciplines, right? Understanding the state of things, to be able to make calls, right? So, I mean, my, my people laugh here because at the office, you know, they lock me up in this office because I'm always on calls, not stopped from like 8 a.m. to like, I think the first time I get up during the day from my desk is at 2 p.m. And then, oh, I need to eat something and then, you know, keep going, right? And so that's the part that's difficult. The intensity is really up there, right? Really, really up there. But at the same time, it's so, I feel like all these things really move the project forward. And so it's, it's, it's motivating. But it is, it is hard on the, on the day to day, for sure.