4.0 started testing, I think, in Evocati in October, went to wave one in November, and we left it in tech preview over December and January. That's about four months of testing. Did we fix anything? What the F were we doing for four months, Benoit?

Oh, yeah. We fixed a lot of things.

All right. Well, so September, October, November, December, right? So since September, we started the Evocati in October, correct? And so for these four months, the majority of the development was aimed at finishing the features to make sure that they were server meshing compatible. So we still had a server meshing playtest at the beginning of October. That was the final one before we actually reintegrated the streams. And so then came some realizations. We were very happy with some of the tests that we've done. We've identified all the issues with our server meshing tests. We knew what we had to do and then we came back to the mainline from the 3.24 content and we started running server meshing in Pyro. And major issues started to show up. When we started the PTU at the beginning of October, the first PTU was an absolute disaster. All of the metrics that we had captured in 3.24, and we were like, yeah, these look good, were completely off the charts with the 4.0 content. Same technology, same code base, but completely new content. We realized that the size of the levels were different. The way the content was created was a bit different. And that had major implications on the streaming and replication systems. And so if some of you guys were part of the first couple of Evocati builds or large PTUs in October, the network performance was absolutely horrendous because, and I can jump into an anecdote, I do have my notes because we've got a long show to go so I'm going to be referring to some of those, but basically there was a new change in the engine where we tried to optimize some of the way we deal with streaming radius. And streaming radius is a bit of a problem in every game, right? It's basically the radius of objects and you decide what you're going to load based on this, based on your viewpoint, which is a typical issue in games, right? For performance reasons. In Star Citizen, it is a much bigger impact because the streaming radius actually affects what your client is going to load in, what the hybrid, which is the replication server in the middle, needs to replicate, and what the game server is going to also have to provide and simulate. And so there was an issue resulting from one of the engine optimizations where, now I'm going to say the final solution, not how we got there. We had to investigate this for several, several days to find it. But basically the way we compound and aggregate objects would actually inflate the bounds of things. And so the new locations in Pyro would actually have a radius of several hundred thousand kilometers. And so basically what the result was, was every game client in Pyro was basically binding a very large part of the environment. And that created the work on the hybrid service, which needs to replicate this data. Upwards of 15 to 20 megabits per client across.

Megabits?

Megabits, yeah. Which is, in the case of our meshing, substantial. And so obviously in those cases, interaction lag up the wazoo, nothing was functional. And so we lost several amounts of time there, but we learned a couple of lessons which are going to inform some of the changes we'll be doing this year. So that was the beginning of the 4.0 journey, right? The first build where we were coming in confident from our tests, first PTU, what's going on, right?

It's another thing that couldn't be discovered until scaled up.

Correct, correct. And then, well, in this case, it's a combination of scale and content, right?

With the content scaled up.

Yeah, well, in this case, it's literally scaled up, which caused the streaming problem, right? So that's a quick anecdote. What else we've been doing, starting from there, was we've established our player counts for launch, right? So from the testing, we decided to shoot for a 500 player shard with basically a 500, what we call soft cap, and 600 player cap, which gives us some flexibility in matchmaking players on and off. That number felt good for several reasons, performance and headroom, but also the ability for the game itself to support that amount of players. I've talked about this before in some of my server meshing reports, but the amount of gateways we have for hangars, how many hangar holes are there around stations for you to be able to get out without the big queue, things like that. With more players right now, the game wouldn't support it properly, so we've established those numbers. And then the team set up to finalize the latest elements that we've identified during the test to launch. So I've talked about network bind optimizations before, so that had to be completed. So this was a full solid month of work to keep optimizing the network bind so that we stopped having these quick stalls. That was a very large one. Another piece of this was the teleport API. Now I know that sounds, what is this magic, right? In a normal game server, teleporting objects is very simple because the server has all the entities in memory and you're like, yeah, just move there. Our teleport API needs to support cross-server teleportation of an object, right? So an object that's in Stanton technically can be teleported through the APIs on another server in the shard. And so the teleport API had to be kind of finished. There are several systems that use this. For example, a common example is if you die and we need to respawn your character, we need to move your player from where he died to where he needs to respawn. And that can be completely anywhere, right? So we don't know what game server will have it. And so in this case, the teleport API had to be kind of finished. So that's part of the work that was done in those four months for launch. The other one was the territory configuration. So if you followed the development during the last couple months, we went at some point from the typical small shard of like two servers and 400 players, and at some point we exploded the size of the shards. That's because our new territory configuration came into play. And that's how we were able to experiment and come to the final level of the shards that we have now, which is basically we have five servers on each side of the star system. So each star system has its own space server. We call this space server for now. It's just a cool name because they treat the open space and the jump gates. But then after that, every major planet has its own game server that's dedicated to its own. And so that really opened the shards up and we were truly now in full server meshing mode. Obviously, a lot of issues came from that, not from the configuration, but from having more servers would have ripple effects on how the entities in the game would actually behave. So we have to touch that. We also identified something that we were not able to finish for 4.0, which was the culling of instanced interiors. And I'll talk a lot more about hangars later, but basically, when you hear someone in another hangar, that's network instance culling, right? So we'll talk about that in a bit. Now, one of the big fears that we had for 4.0 was that, of course, it's the debut of server meshing. It's a brand new deployment for the game. It's a lot of technology change. But there's always a subtext that performance is going to get better if we go server meshing, right? And the reality is the initial test showed that it did not. Now, not because our assumptions were incorrect, but because as always, when you don't completely drive on the performance, it tends to get worse. And so we mobilized everybody, like the entire company, basically, all the engineers, right? Where we started a series of, I call them Joker card initiatives, right? So several teams during the last push received the Joker card. And what that allows them to do is to cut the red tape. So for example, the network team had the Joker card. They could request anything from any team and get priority. And that allowed them to really move forward on their changes and their fixes. And so we started a bunch of those Joker card initiatives on performance specifically. And so every single day we would take captures, we would review hundreds and hundreds of captures, identify code parts that needed fixing. And then literally without a big process, right? So we've cut the process down. You talked about Jiras earlier, right? This part of the process, no Jira, right? And we just called out, here's a performance hotspot. And then directors from the company, our amazing tech teams and our engineers just jump on those things and go, I know what to do, optimize, next batch. And we kept grinding like this to really make sure that we could deliver above 15 to 20 FPS per server, at least, to get started. And we want to go even further, but that was kind of our target, was to deliver something where we're no longer dipping under 10 at all, ever. And so, for example, the AI team discovered through this process that there was a very large navigation mesh generation that was directly on the main thread that cost us 15 to 20 milliseconds sometimes, which is, the frame budget is 33. But the response from the teams was amazing. They rapidly refactored and optimized really key pieces of code to shave on these things. That made a very big difference. Out of these Joker card initiatives, I'm going to jump into the important ones. Physics is a very big issue for our servers. I've said that before. Not because our physics code is terrible. In fact, it's great. But we demand a lot from it. And we started to look into, well, what's going on here? Why is it costing us so much? And so we really profiled on that side quite a bit together as a team. So one of the things to remember here is that our bugs persist, right? They persist. But what I mean by that in this case is if I spawn an object intersecting with this desk, in any other game, when the map reloads, these objects are no longer colliding and we're good. In Star Citizen, those things will stay in complete intersection in the database and in the shard and in its persistence forever. The physics code tries to solve this and that's what's costing us performance in big parts. And so we started this Joker card initiative that we call the physics expensive event resolution. And so it's about the physics engine detecting these things, which it already did, but now raising it for resolution, saying, I need a solve for this. And so a ship that intersects with a wall or that spawns in a wall, a weapon that falls through the floor and that's stuck in there, debris has been a big one. We needed a way to solve this. So what the physics expensive event system does is it raises an event for the game and the game now receives these objects and it says, oh, debris and player ships. Well, I'm going to do something with the debris. I'll either delete it, stow it, transform it, move it, right? And so as we start doing this, it's actually cleaning up all of these intersections across the shard and basically giving us back this performance step by step. And so one of the very large things that was very egregious in this was actually a bug on the ships where on explosion some of the components of the ship could stay static in the world throughout the explosion. Behind the scenes, when the ship explodes, we do a whole bunch of things. We replace some meshes with damaged meshes. Some of the elements are physically deformed and move. And we create debris. We literally spawn debris. If those components stay static and we spawn debris, we're creating intersections, debris straight intersecting into these components. And so that was a very big part of the problem. We fixed that bug, but we also added a system to basically raise the issues with the debris so that they could be cleaned up automatically. Another one was the AA turrets. So as you guys play, you destroy these AA turrets at the key locations, right? And there was a problem with the setup of these AA turrets that when they died, in the same way as a ship when it dies and the components detach, right? The AA turret had an issue in its setup where it would go like this, but overlapping each other. Basically, every time someone killed an AA turret, we added it to the server load forever. And so obviously, now that's being evicted completely from the system.