Star Citizen Live SPECIAL EDITION: A More Stable Universe

Benoit Beausejour · Chief Technology Officer

Well, so during the development of 4.0, there was a lot of new technology coming on, right? So it's a major patch that changes a lot of the fundamentals. Obviously throughout the year, we've been developing server meshing, we've done tech previews on server meshing. And so getting to the end of that process, basically exiting the tech preview streams, coming back with the 4.0 content, it was a major endeavor to synchronize all that, make sure the new content that was produced fit with the technology and that it was working. Even the results that we got with the 3.24 release, for example, were completely obscured by the new content. At the end of the year, we took a decision to split the two releases.

Jared Huckaby · Host, Community

So leave 3.24 up, put 4.0 in a preview.

Benoit Beausejour · Chief Technology Officer

Correct. Because as we were getting closer to the end of the year, and time is running out, what we knew from the integrations is that the patch was not quite ready because we needed more concurrency on it to discover more issues. And we were kind of at the maximum of what our PTU concurrency could actually deliver. But we knew it wasn't ready for a live release because we want to be fair with our assessment of where the patch was at. We knew there were some egregious issues that we had yet to discover. I'll talk a bit later about the fog of war of crashes. And so we decided to go with the lesser of the two known problems, which was leave 3.24 live for players to enjoy in a stable release, right? But at the same time, give them access to a 4.0 preview. Now, since we had taken the decision months ago that the 4.0 patch was going to be a full wipe, that gave us the ability to basically preview the patch but allowing you guys to keep your persistent progress in the preview as we move up to at some point bring 4.0 on the live environment. So overall I think that was a very difficult decision to do this option. But I think it was the best for the project because it allowed us to de-risk a little bit the .0 initial release, which had a lot of issues even during the preview. A lot of the skeleton crew teams worked throughout the break to actually document, hotfix some of those issues as well and help players that got into weird situations. And start the ball rolling for the year with already important fixes to the underlying technology.

Benoit Beausejour · Chief Technology Officer

Well, in this case, our project is different from other game projects in that we need to release the code at some point to be able to move on to bigger projects. And the PTU is a great process for us, but it does have its limits. At some point, we gain nothing from additional PTU builds because things function well at that level of concurrency and we need way more concurrency to go up. Even when we plan patches, we know at which moments in the PTU we'll have concurrency and we try to maximize those. For 4.0, we kind of ran out of gas of attention. And so we needed to get this additional hump over the edge to get the technology out, bigger concurrency and the ability to fix it in order to then plan for the release.

Benoit Beausejour · Chief Technology Officer

That was our typical decision, typically, was to just go for it. In this case, I think it shows a change of mentality, which is that we're trying to respect our live environment a bit more. And so the decision was taken to just allow players to keep playing 3.24 as even a fallback or as their primary play environment, if they preferred, while you could jump on the preview experiment and then just see, try the new content out, but then always have your stable experience to go back to.

Benoit Beausejour · Chief Technology Officer

Yeah, overall, throughout the break, during the 4.0 preview, there was actually a fairly large contingent of players still playing 3.24. Now, it might have been because of some of the issues at launch, right, so they fall back on the 3.24 patch, which was a bit the kind of the safety net that we wanted to provide. But you're right, of course, we were banking on the fact that people would go on the 4.0 preview to give us that concurrency, but at the same time, we wanted to give them an escape hatch.

Jared Huckaby · Host, Community

Sounds a little bit like we were trying to have our cake and eat it too.

Benoit Beausejour · Chief Technology Officer

A little bit, but I think previously we would have just gone with the 4.0 live at the end of December, which would have been a major issue and a lot of players wouldn't have been able to play during the break.

Benoit Beausejour · Chief Technology Officer

Well, we've been able to actually do a lot of live comparisons of the two workloads, which was very interesting. In fact, obviously the 4.0 preview workload was bigger than the 3.24 one, but we were kind of able to compare both. We did gain the ability to fix early some of the most egregious issues that would have been a real complete disaster otherwise that showed up with the concurrency. I'll talk a bit more about those. Actually, we're going to dive into that quite a bit. And so what we've gained is the ability to provide a test environment that was bigger while allowing players to keep an escape hatch and then compare both builds.

Benoit Beausejour · Chief Technology Officer

Oh, yeah. We fixed a lot of things.

Benoit Beausejour · Chief Technology Officer

All right. Well, so September, October, November, December, right? So since September, we started the Evocati in October, correct? And so for these four months, the majority of the development was aimed at finishing the features to make sure that they were server meshing compatible. So we still had a server meshing playtest at the beginning of October. That was the final one before we actually reintegrated the streams. And so then came some realizations. We were very happy with some of the tests that we've done. We've identified all the issues with our server meshing tests. We knew what we had to do and then we came back to the mainline from the 3.24 content and we started running server meshing in Pyro. And major issues started to show up. When we started the PTU at the beginning of October, the first PTU was an absolute disaster. All of the metrics that we had captured in 3.24, and we were like, yeah, these look good, were completely off the charts with the 4.0 content. Same technology, same code base, but completely new content. We realized that the size of the levels were different. The way the content was created was a bit different. And that had major implications on the streaming and replication systems. And so if some of you guys were part of the first couple of Evocati builds or large PTUs in October, the network performance was absolutely horrendous because, and I can jump into an anecdote, I do have my notes because we've got a long show to go so I'm going to be referring to some of those, but basically there was a new change in the engine where we tried to optimize some of the way we deal with streaming radius. And streaming radius is a bit of a problem in every game, right? It's basically the radius of objects and you decide what you're going to load based on this, based on your viewpoint, which is a typical issue in games, right? For performance reasons. In Star Citizen, it is a much bigger impact because the streaming radius actually affects what your client is going to load in, what the hybrid, which is the replication server in the middle, needs to replicate, and what the game server is going to also have to provide and simulate. And so there was an issue resulting from one of the engine optimizations where, now I'm going to say the final solution, not how we got there. We had to investigate this for several, several days to find it. But basically the way we compound and aggregate objects would actually inflate the bounds of things. And so the new locations in Pyro would actually have a radius of several hundred thousand kilometers. And so basically what the result was, was every game client in Pyro was basically binding a very large part of the environment. And that created the work on the hybrid service, which needs to replicate this data. Upwards of 15 to 20 megabits per client across.

Jared Huckaby · Host, Community

Megabits?

Benoit Beausejour · Chief Technology Officer

Megabits, yeah. Which is, in the case of our meshing, substantial. And so obviously in those cases, interaction lag up the wazoo, nothing was functional. And so we lost several amounts of time there, but we learned a couple of lessons which are going to inform some of the changes we'll be doing this year. So that was the beginning of the 4.0 journey, right? The first build where we were coming in confident from our tests, first PTU, what's going on, right?

Jared Huckaby · Host, Community

It's another thing that couldn't be discovered until scaled up.

Benoit Beausejour · Chief Technology Officer

Correct, correct. And then, well, in this case, it's a combination of scale and content, right?

Jared Huckaby · Host, Community

With the content scaled up.

Benoit Beausejour · Chief Technology Officer

Yeah, well, in this case, it's literally scaled up, which caused the streaming problem, right? So that's a quick anecdote. What else we've been doing, starting from there, was we've established our player counts for launch, right? So from the testing, we decided to shoot for a 500 player shard with basically a 500, what we call soft cap, and 600 player cap, which gives us some flexibility in matchmaking players on and off. That number felt good for several reasons, performance and headroom, but also the ability for the game itself to support that amount of players. I've talked about this before in some of my server meshing reports, but the amount of gateways we have for hangars, how many hangar holes are there around stations for you to be able to get out without the big queue, things like that. With more players right now, the game wouldn't support it properly, so we've established those numbers. And then the team set up to finalize the latest elements that we've identified during the test to launch. So I've talked about network bind optimizations before, so that had to be completed. So this was a full solid month of work to keep optimizing the network bind so that we stopped having these quick stalls. That was a very large one. Another piece of this was the teleport API. Now I know that sounds, what is this magic, right? In a normal game server, teleporting objects is very simple because the server has all the entities in memory and you're like, yeah, just move there. Our teleport API needs to support cross-server teleportation of an object, right? So an object that's in Stanton technically can be teleported through the APIs on another server in the shard. And so the teleport API had to be kind of finished. There are several systems that use this. For example, a common example is if you die and we need to respawn your character, we need to move your player from where he died to where he needs to respawn. And that can be completely anywhere, right? So we don't know what game server will have it. And so in this case, the teleport API had to be kind of finished. So that's part of the work that was done in those four months for launch. The other one was the territory configuration. So if you followed the development during the last couple months, we went at some point from the typical small shard of like two servers and 400 players, and at some point we exploded the size of the shards. That's because our new territory configuration came into play. And that's how we were able to experiment and come to the final level of the shards that we have now, which is basically we have five servers on each side of the star system. So each star system has its own space server. We call this space server for now. It's just a cool name because they treat the open space and the jump gates. But then after that, every major planet has its own game server that's dedicated to its own. And so that really opened the shards up and we were truly now in full server meshing mode. Obviously, a lot of issues came from that, not from the configuration, but from having more servers would have ripple effects on how the entities in the game would actually behave. So we have to touch that. We also identified something that we were not able to finish for 4.0, which was the culling of instanced interiors. And I'll talk a lot more about hangars later, but basically, when you hear someone in another hangar, that's network instance culling, right? So we'll talk about that in a bit. Now, one of the big fears that we had for 4.0 was that, of course, it's the debut of server meshing. It's a brand new deployment for the game. It's a lot of technology change. But there's always a subtext that performance is going to get better if we go server meshing, right? And the reality is the initial test showed that it did not. Now, not because our assumptions were incorrect, but because as always, when you don't completely drive on the performance, it tends to get worse. And so we mobilized everybody, like the entire company, basically, all the engineers, right? Where we started a series of, I call them Joker card initiatives, right? So several teams during the last push received the Joker card. And what that allows them to do is to cut the red tape. So for example, the network team had the Joker card. They could request anything from any team and get priority. And that allowed them to really move forward on their changes and their fixes. And so we started a bunch of those Joker card initiatives on performance specifically. And so every single day we would take captures, we would review hundreds and hundreds of captures, identify code parts that needed fixing. And then literally without a big process, right? So we've cut the process down. You talked about Jiras earlier, right? This part of the process, no Jira, right? And we just called out, here's a performance hotspot. And then directors from the company, our amazing tech teams and our engineers just jump on those things and go, I know what to do, optimize, next batch. And we kept grinding like this to really make sure that we could deliver above 15 to 20 FPS per server, at least, to get started. And we want to go even further, but that was kind of our target, was to deliver something where we're no longer dipping under 10 at all, ever. And so, for example, the AI team discovered through this process that there was a very large navigation mesh generation that was directly on the main thread that cost us 15 to 20 milliseconds sometimes, which is, the frame budget is 33. But the response from the teams was amazing. They rapidly refactored and optimized really key pieces of code to shave on these things. That made a very big difference. Out of these Joker card initiatives, I'm going to jump into the important ones. Physics is a very big issue for our servers. I've said that before. Not because our physics code is terrible. In fact, it's great. But we demand a lot from it. And we started to look into, well, what's going on here? Why is it costing us so much? And so we really profiled on that side quite a bit together as a team. So one of the things to remember here is that our bugs persist, right? They persist. But what I mean by that in this case is if I spawn an object intersecting with this desk, in any other game, when the map reloads, these objects are no longer colliding and we're good. In Star Citizen, those things will stay in complete intersection in the database and in the shard and in its persistence forever. The physics code tries to solve this and that's what's costing us performance in big parts. And so we started this Joker card initiative that we call the physics expensive event resolution. And so it's about the physics engine detecting these things, which it already did, but now raising it for resolution, saying, I need a solve for this. And so a ship that intersects with a wall or that spawns in a wall, a weapon that falls through the floor and that's stuck in there, debris has been a big one. We needed a way to solve this. So what the physics expensive event system does is it raises an event for the game and the game now receives these objects and it says, oh, debris and player ships. Well, I'm going to do something with the debris. I'll either delete it, stow it, transform it, move it, right? And so as we start doing this, it's actually cleaning up all of these intersections across the shard and basically giving us back this performance step by step. And so one of the very large things that was very egregious in this was actually a bug on the ships where on explosion some of the components of the ship could stay static in the world throughout the explosion. Behind the scenes, when the ship explodes, we do a whole bunch of things. We replace some meshes with damaged meshes. Some of the elements are physically deformed and move. And we create debris. We literally spawn debris. If those components stay static and we spawn debris, we're creating intersections, debris straight intersecting into these components. And so that was a very big part of the problem. We fixed that bug, but we also added a system to basically raise the issues with the debris so that they could be cleaned up automatically. Another one was the AA turrets. So as you guys play, you destroy these AA turrets at the key locations, right? And there was a problem with the setup of these AA turrets that when they died, in the same way as a ship when it dies and the components detach, right? The AA turret had an issue in its setup where it would go like this, but overlapping each other. Basically, every time someone killed an AA turret, we added it to the server load forever. And so obviously, now that's being evicted completely from the system.

Benoit Beausejour · Chief Technology Officer

So for 4.0, between 3.24.3 and 4.0.1, the team fixed 2,900 bugs, but that's over 3,300 tasks. So a bug can require a couple of tasks to fix. So in general, that's the number of fixes overall.

Benoit Beausejour · Chief Technology Officer

I don't do prioritization, but what I can tell you is that most of those issues are filed, they're called out, and they're raised up. But the complexity of some of them requires a specific handling. I've read some stuff over the break where we've hotfixed certain issues. There was, if I remember, a certain exploit that came out for some duping and some people thought it was a beneficial change even though it was a bug. And we fixed it. And people are like, well, why did you fix it when you could fix something else? It's all about capacity to fix. So do we have the right person available for this problem? Some problems require the experts. Our manpower is extremely specialized. And for some of the older systems, it's even worse. I call those the sandworm kind of features where basically, I've called them out before, but ATC, ASOP terminals, hangars, and transit are four of the systems that are always, oh, I'm just going to do a quick change. Never happens. Never a quick change in those systems, right?

Jared Huckaby · Host, Community

Yeah, I want to jump in here. Uno Mas in Twitch chat is yelling in all capital letters, you are the experts! True. One of the things, if you watch any of our programs over the years, I've been doing this for a decade here now, is that not everybody works on everything. Kian is great at this, doesn't know crap about that. Paul is wonderful at this, would be as lost as I would be tackling this other thing. There are a finite number of experts that are available on any one thing. And like we talked about the last time you were here, the folks who made ATC aren't here anymore. So ATC has been one of those features that had rotted, and there's this terrible Sophie's Choice, and we have a section about how these choices are made here, where it's like, okay, we could take someone like Kian or someone like Paul or whatever and put them on this, but they're going to be on this for five months because they've got to start from scratch. Whereas they could fix 15 or 20 other issues in that same time. And there's this constant calculus of, is that one worth the weight of those 20? And it's not a situation I envy you or anybody else. You said that you don't make the prioritization. You're in these conversations along with everybody else. You mean you don't make them alone. You're not the sole person who gets to decide. Many things are done in a committee because everybody's got their perspective. Everybody's got their priorities. Everybody's got their idea of what they think is the most important. I've been in chats where Chris will say, this is the most important. And Aaron will say, this is the most important. And Jens will say, this is the most important. And you're like, well, no, this is the most important. And we have the same conversation that the backers have, just three weeks earlier, is usually what I say. It's prioritization and also having the right resources and having that right resource available, because sometimes the right person for that is also the right person for this, and they've been in this for three months. And to pull them off of this right now would be devastating to that, because then you get tech debt and that begins to rot. And there are so many times where I've been in meetings or sat in chats and just ground my teeth, because we all know what needs to happen. We all know that that thing needs fixing, but there isn't the resource at the moment to go and do it. And that's at least as frustrating for us as it is for the players, probably more so for the players, if I'm being completely honest.

Benoit Beausejour · Chief Technology Officer

No, but you're right. It's all about having the right people for the right issues at the right time. And historically, if we're being frank, we've been focusing more on the features.

Jared Huckaby · Host, Community

You've been focusing more on the new stuff.

Benoit Beausejour · Chief Technology Officer

Correct. And I think that what Chris outlined in his letter from the chairman is really the direction we want to go, which is to bring back those experts into helping us fix the things that affect the players day in and day out.

Benoit Beausejour · Chief Technology Officer

Stage one is we're committing more resources to fixing issues that affect the live environment, but also that are reported by players. And so in a typical setup, we have to take those resources from features. So that means that normally if we're on 50 different features, all these engineers are working on new content, to build new components or new features, new UIs for these new gameplay elements. And this can go on for a while because there's prototyping, there's iteration, there's merging, QA, and this can take a while to bring it in. And as Chris is pointing out, it creates ripple effects. And so we have to slow down on the features to get those resources to jump into what I call these Katamari issues. Like we're going to look at a couple later, but issues that require, oh, I need a physics expert. Oh, I need an audio expert. Oh, I need a graphics expert. Oh, I need a gameplay expert. I need a guy who wrote this system. I need a guy who wrote this system. An issue that just keeps growing until it's solved. In order to do that, we need to give a Joker card to those issues. And so that's what we're going to do this year, is to slow down on features. We still need to build features, but we're going to pick the ones we can do, while still investing in correcting the problems that we know have been in the game either forever or they're new and very active. And even if they're super complicated to solve and require a refactor, right? So we'll talk about refactors, I'm sure, in the next list. But it also means, what I put in my notes, because we kind of prepared a list so that I can just organize my thoughts, increasing bug fixing quotas for teams. So in general, teams have not necessarily a quota to fit, but they have a lot of time for bug fixing. We're going to try to bump that by a lot so that we can actually jump on more bug fixes that matter. That's the thing that's key, right? Even though we fixed 3,000 issues, there are maybe 15 or 20 or 30 issues that are really affecting players day in and day out that are bugs or defects that must be resolved. And so we've put that in a list called the Director's List. So every director has a chance to basically flag an issue in their domain and say, I'm going to need a Joker card for this issue and it needs to be solved. And so that list is also going to transform itself into a release criteria. Typically when we do a PTU, we're going through these waves. The process at which we go to the next wave is always a bit, it looks good, we look at metrics, we look at disconnects and stability in general, but we don't necessarily look at quality of the game specifically because we're in development. And so it's always been a bit different for us. But I think that we need to change this process for us to move away from being an alpha game. We need to start moving away from the words constant development. In order to do that, we need to professionalize our criteria for launch. And that means that director's list needs to go down. It needs to get the solves that it needs. And those become criteria to move forward. Now, we're going to have to make decisions because we cannot fix everything at the same time. Again, it's a question of resources. The goal is to make a significant dent in that list, even if it keeps going up there.

Jared Huckaby · Host, Community

You said increase the number of engineers and programmers. You're not talking about going out and hiring 100 new people. You're talking about the people who were working on base building, the people who were working on crafting, the people who were working on resource networks. It's about taking those seniors, those people with the experience, the ones that Uno Mas said, the experts. It's about taking those experts and pulling them off the new feature development and putting them on the less sexy stuff like, hey, fix an elevator.

Benoit Beausejour · Chief Technology Officer

Well, it translates into aggressively monitoring. What we call stability internally covers some of the very simple metrics, like do servers crash, or do hybrids crash, do clients crash, and together, do these cause client disconnects? And so all these metrics go into what we call basically stability, but it goes further than that. Those are just basically the indicators, like the big level indicators. What stability is, is bare-bones things, like are you able to successfully connect to the game?

Jared Huckaby · Host, Community

Can you log in?

Benoit Beausejour · Chief Technology Officer

Correct. Can you do successful interactions? Like, can you go to your ship and press F to join your cockpit and then it takes 35 seconds? Is it consistent and reliable? Meaning, is it like this all the time? Or is it there only sometimes, right? So stability is about consistent and reliable experience interactions. It's also, when we work on stability, we're talking about improving the observability. So when you guys are playing the game, you have a very immersive experience, and issues that affect you during your gameplay are part of your bubble, right? We need to be able to see those issues from the outside. So from the server side, from the cloud environment side. And so improving observability gives us that, because there are many systems in the game that have not been instrumented to be observed, right? And so sometimes some older systems, I'm not going to name ATC, are going to have some hiccups that we can't even see, but players report to us and then we have to investigate, but we need more observability. So that's working on stability as part of that. Changing systems that typically are problematic into self-healing systems so that they can just repair themselves. So when they get into a case that we know, we just modify the system to make sure it tends towards a state that's functional. And the observability comes into play to make sure that it does function. Working on stability also means establishing budgets. So budgets for errors, budgets for performance. And so that affects everybody in the company. But if we talk purely on stability, it's all about, is the content that we've produced here or the feature we've produced here using too many resources, whether it's CPU, GPU, server calls, bandwidth, storage, networking? So reinforcing those budgets, going about finding the offenders that go above these budgets and chasing them down, solving them. So that's stability work. Reducing error rates, obviously, that's part of it, that's the result. And the challenge with working on stability, especially in light of server crashes, is that it's a fog of war. So you can launch a version of the game and there's one crash that's going around. We can identify and dedupe crashes, right? So we know what they are. So we're looking into one and then we're like, oh, if we fix this one, it's going to be very stable, right? But then when you fix that one, that allows the program to continue. And then more bugs show up, more crashes show up. So it's a fog of war. It needs constant care and attention. One thing we've done this year to help that is that we have a much more robust hotfix process that, frankly, we use a bit too much. But it does give us the ability to get some fixes out when we see big problems right away. But it's to contribute to that fog of war. So that's what working on stability is for me. It's all of these issues together.

Benoit Beausejour · Chief Technology Officer

Sure. Performance is a larger one. And so performance touches on the client, the hybrid and the server. The hybrid is the replicant that takes care of the networking. And so performance is about obviously frame rate, to begin with, on the client and on the server. We talked earlier about server optimizations. It's about loading times. So making sure that the load times are acceptable in all kinds of areas. I'm not just talking about the loading screen, I'm talking about loading an asset, streaming in an area, fully initializing a full location, right? That's the load times I'm talking about. Memory utilization, so making sure that we use memory efficiently, that we don't go above our memory budgets. Same thing here, the keyword is budgeting. So we need to have way stricter budgets. We need to work on that area. And then finally, resource utilization of your machine in general, but also network connections. So latency and bandwidth, connectivity and stability of the connection in general, that need to be improved. So that's what working on performance is. So stability is one area, performance is another one.

Benoit Beausejour · Chief Technology Officer

So it's, again, engineering, right? That has a large focus on stability and performance. But also, we have a team, an online team, that takes care of what we call site reliability. So these guys are definitely the first watchers of all these metrics. And these guys will call out issues that we need to solve, right? They will loop in engineers. And so that's the core of the people working on this. But I kind of want to push back on what you just said about, oh, the guys on the ship team don't have to do performance, right? Sometimes the way things are made in the game has a very big impact on performance. I can give examples, but areas of levels that are created with hundreds of entities instead of using a bare minimum. Or there's techniques to use, where group merging certain objects helps reduce the actual effective size of things. So the teams on content are also kind of on the hook for performance, and less so on stability, but they're on the hook for performance. And so they're not totally free, but they are less in the line of fire, because typically an engineer is needed to identify a problem first and then it might reach a content team depending on what the issue is.

Benoit Beausejour · Chief Technology Officer

It's basically the same thing almost.

Jared Huckaby · Host, Community

Why do we use both terms?

Benoit Beausejour · Chief Technology Officer

I don't know. I think you use programmer when you talk about a coder, and you talk about an engineer when he's got an architectural approach to it, I guess. But they're very similar in my mind.

Benoit Beausejour · Chief Technology Officer

It's an interesting problem, right? Because fundamentally, all these talented programmers and engineers are human beings behind it. I think that one of the challenges that the teams face regarding this, not necessarily prioritization, let's talk assignment first, is there's a certain level of ownership. I write a system, then I own it, right, in a way. And then further iterations or bug fixes on this thing, I will be the best qualified to do it because I know the code. Anybody else that needs to go in this code is going to have a little ramp up, right? And sometimes we can't pay that ramp up, especially at the pace at which, I keep saying that, in our project it's different, typically. We don't release a game before it's in a better state.

Benoit Beausejour · Chief Technology Officer

Most games don't run a live game environment in the middle of their development.

Benoit Beausejour · Chief Technology Officer

Exactly. And that, I call it the cost of shipping, is what hurts us the most, because now any priority issue for the live patch needs to become absolute. Back to assignment. One programmer writes a system, he owns it. So we're going to try to go back to this person when there's bug fixes or items to go to. They're going to have the least on-ramp time. But now sometimes it can happen that that's a junior developer. It can happen that it's a senior developer. It really depends, right? Some systems also end up with no owner when there's churn and people leave and there's replacements or people move on. There's a lot of internal movement at CIG where people want to change their specialty. And so you want to keep the distribution fair for the people in there. You don't want to assign someone to pure bug fixes for five years, because they're not going to stay with the company. You need to balance this out in terms of how you assign the work. Again, I'm not talking prioritization, but assignment. Typically, we're going to try to pair juniors with seniors to encourage growth so that these juniors can become seniors and then we can depend even more on them. That's another constraint, that sometimes you need to spend time from a senior on a specific issue to help a junior on another issue.

Jared Huckaby · Host, Community

Sometimes you'll have two people on an issue, not because they can make the baby twice as fast, but because that's how you train that junior to become a senior. And that's one less person that could be working on something else, because this is part of their training, this is their growth.

Benoit Beausejour · Chief Technology Officer

Correct. There's also, so we talk juniors, seniors, right? We make it simple, but there's also leadership, right? So lead developers, most people know this, but the lead developer is not necessarily the best developer of the team. But the lead's responsibility is management. It's making sure that the issues move and that our plan of action is sound, that it's vetted, that there's code reviews, make sure the quality is there, but also just orchestrating the team. Typically, a lead should spend roughly 40% of his time doing management. So leadership is also a key element in that. Even if you pair people and you have a million things going together, what is the drumbeat is the leadership. And so you need to have all those combinations to be able to have a healthy engineering team. You want to avoid overburdening as well. So sometimes you have some developers that come into the company and they're really good, right? But they're juniors, right? So they're learning fast. They have a lot of energy. They're going for it. And so, path of least resistance, we're going to go like, oh, well, can you look at this? Can you look at this? Can you look at this? And at some point, that person's overburdened. And we have to release our patches, so time constraints come in. And so code quality goes down, right? And that's an issue we have to battle every single day.

Benoit Beausejour · Chief Technology Officer

Well, so previously, the answer would have been junior with some support, right? That's what we would have done before. It would have been, no, we can't compromise the feature. This work is important. It's going to be transformational for the PU. This fix, or this rewrite, we're going to give a support network around someone that's in the team that's not necessarily one of our superstars. I want a team where everybody can grow into a superstar, but sometimes you have to be in the middle. The previous setup would have been something like this, where we would give a support network, but the person in charge would not necessarily have been the leads and the people that really push the needle. Those would be on transformational features for the PU. The answer this year will be no. We're using our transformational players to come and solve these problems for us, because they have to be solved. We keep talking about transit. Transit has touched a million people at CIG and we've tried to keep it running for a while. Now it's time to put a real effort behind it, which is to basically take resources from the transformational elements and go and fix the foundation.

Benoit Beausejour · Chief Technology Officer

Well, so that's a good question, right? Because sometimes the worst cases like this is when a system or an issue is almost fixed, right? Where a developer comes in and is like, oh, I have a fix for this. But what we need to do now is also look at these features to answer that question, which is, is the way the feature or the system was constructed compatible with what the PU is going to be, or what the technology behind the PU will be? A combination of the streaming technology that we've built before, that has been a very complicated and complex system even on its own without server meshing, and then the systems on their streaming and server meshing together. And so there are systems in the game that are not adapted to that approach, right? And so when we look at one of the key problems that we see in the game, we're going to look at the underlying system, and if we need to rewrite, it'll be because, yeah, it needs to be rewritten to take advantage of streaming and server meshing properly, versus trying to patch and kind of cobble it together to make it work. So that's how we make the decision. We're going to have to look deep in all these things to decide which one's needed. We've already done some of it. We're calling out some stuff already.

Benoit Beausejour · Chief Technology Officer

So that's an interesting question because we are making two games. There's some good news, right? Because a lot of the gameplay systems that power the actor and that power the core of the game are the same between the two games. And so improvements on one end result in improvements on the other one, plus or minus the online versions of these features, right? Let's say we take one team, which is the most important in this one, like the gameplay teams. That's where we have to be more fine-grained. There's this thing that we have to be careful about, which is context switching. Obviously, building a single-player cinematic game and building a multiplayer MMO-like game are similar, but the context switching between the two projects costs time. So you want to find a scheme where you have the resources on what you're trying to achieve at each milestone. Like Squadron 42 has its internal milestones and its cadence that it needs to respect this year, which is extremely aggressive, in order to get a game completely complete by the end of the year and ready for '26 release. And so there are resources that are more suited to the type of things that Squadron 42 kind of needs right now in its polish phase and its finalization phase for all the levels and all the chapters. And so obviously these people will be favored to work on Squadron 42. So there's a series of people that are on there. I don't have an exact percentage, but it's not 50-50, because there are also things that are built on the PU that are beneficial on that side. If Jens was here with me, he would say the thing we need is flex. Flex is when we need reinforcements, we can get them. We have a model now where there are a series of engineers that will work dedicated to Squadron. But we can flex on both sides depending on the milestones. And the two projects now have a cadence that's adjacent. And so we kind of know at every milestone what Star Citizen needs and what Squadron needs. And so we can flex accordingly. And a lot of those features are similar on both ends. And some of the bug fixes that we need on SC are also needed on both sides, especially actor-related bugs. It's a tricky balance, right?

Benoit Beausejour · Chief Technology Officer

Yeah, the key is in the specialization, right? So we have teams like, I can give an example, the mission system team, for example, are highly specialized in the online mission system, right? But this assignment, right? You take the resources that are good at one element that one project needs and you can dedicate them. You're not losing anything from the other side, right? The flex obviously is there in your extra kind of chunk of resources, right? Every team has core members and extra members that can flex. And so the idea with this has been to make a plan where we can actually focus on the things that matter, and we pick what we think matters, right? So our player experience team on the SC side helps us with the bug fixes that we're going to be looking at in priority. We know what those are, so we make sure those are covered. So it's about not just prioritization and assignment and just blatantly going SC, Squadron. It's really about looking at, what do we want to achieve? And on SC, we want to achieve better playability. And so we're going to commit resources to that. The flex will come only when we ship, right? That's the moment where flex actually happens, so end of milestones. And so the impact of this will kind of be minimal, because we minimize the context switch on the key resources. Leadership is a key part of this. The right leads in the right places make a big difference. Those ones need to not move.

Benoit Beausejour · Chief Technology Officer

In many areas, it's a drastic change, yes. Especially for the issues that affected a lot of players during the break. If you played during the break on the 4.0 preview, a lot of very dramatic problems have been solved in 4.0.1 or hotfixes, right? Because 4.0.1 also includes all the hotfixes from 4.0. So it is fundamentally a better patch. It regressed in key areas that are very problematic. Yes, I'm looking at the chat right now. Very drastic, with a lot of emojis. From the observability standpoint, I think that in terms of the forward-facing elements of the patch, we have regressed in 4.0.1. But from the infrastructure level, we have actually made a step forward. And so it's a 50-50 one.

Jared Huckaby · Host, Community

It's a two steps forward, one step back kind of thing?

Benoit Beausejour · Chief Technology Officer

Yes, I would say so.

Jared Huckaby · Host, Community

And the one step back just happens to be the part the players see?

Benoit Beausejour · Chief Technology Officer

Correct.

Benoit Beausejour · Chief Technology Officer

I think so. But what it needs, right, and I think Chris's letter kind of sets the commitment. That's what we needed, right? Going from the previous way of making the game to this new way, a company of our size takes time to shift gears and change direction. And so even though you're like, okay guys, this is our focus now, reorder priorities, assignments, it takes a while before it actually takes form, right? And you need the people in the teams to really grasp what we're after, right? In terms of increased playability, what it means for them in the day-to-day, right? And I think that we have the commitment from the leadership to change this and we have the means to do it. So in order to support this effort, it's not just we're talking about priorities and orders. That's one part of the puzzle. But the other part of the puzzle is to make sure these teams are properly supported to do this. It's in paying the tech debt that's been plaguing the project for a long time, that we've been kind of dealing with, putting in the corner, tackling some of it, but then letting it build up. And I think that we need to attack that head on. And to support our teams, it's about helping them improve the output, through better process, better pre-submission checks, better validation, building the tools that will allow them to have confidence that when they put in a fix, that it's fixed, or that when we say a bug is fixed, that it is fixed, which, let's just say it, has been an issue, right?

Jared Huckaby · Host, Community

Many have been the times where I've seen everybody from you to Chris to Jens in the chat, they read a report on Reddit or they see something on Twitch and YouTube and they find the Jira and the Jira says it's fixed. And it's like, I thought this said it was fixed. And that happens for a lot of reasons. We've talked a lot about how there are multiple causes for the same outcome. And yeah, that cause of that outcome was fixed, but now we've just exposed a second, different cause for the same outcome. It's frustrating.

Jared Huckaby · Host, Community

The proof will ultimately be in the action, in the month after month, patch after patch. There won't be a patch every single month. Like I said, we're currently looking at about nine or 10 on the schedule. We actually have a meeting tomorrow where we're going to kind of hammer that down and have a better idea. Certain things affect the patch cadence. But more patches this year than any year prior, but with less of the feature work, far less of the feature work, and more of the content and the bug fixes, the content that doesn't require new features. Somebody in the chat was just saying, do less events. It's not the events that are the problem. It's the events that are built on new features. Events built on existing feature work can still be fine. That's part of what we're looking at.

Benoit Beausejour · Chief Technology Officer

Ultimately, it's just the product. That's the thing that counts. What's the output? Does the output get better? Are we still dealing with the same problems that we've seen before? Did we actually solve any of those issues? Or are we still going to look at interrupted missions or any of these bugs that are plaguing us? And I think that that's the key, right? And that's how we should be measured. It's in, does this new method work? Is the product improving, right? And its quality.

Benoit Beausejour · Chief Technology Officer

So this one we're actually deeply investigating right now. It looks like, so we have, and I'll go deep with technical details here to probably explain, not just the status report. So, matchmaking. So when you click on that button, join the universe, you're talking to the matchmaking service or the matchmaking system. It will find one of the shard instances that's available. All these shards report their health to make sure that they're doing good, that if one of their servers is not good or their FPS is not in a good condition, it always reports a score. And we will always try to match you to something that has shard affinity to you.

Jared Huckaby · Host, Community

Shard affinity?

Benoit Beausejour · Chief Technology Officer

Well, it's either your player is already in that shard, that's the ultimate affinity, or you've been in that shard before. So we try to have a level of shard affinity. And now the problem with stuck at position one is that we've matchmade you on a shard. And now the matchmaking service is basically telling the shard, this guy's coming. That's a process called authorization. And for some reason, the latest update that I've got this morning on this is that the authorization call sometimes will take multiple minutes to close out. And so we don't know what the cause is on this one yet. It's kind of a newly identified condition. But that's one that we need to hotfix absolutely on live in order to get that back in there.

Benoit Beausejour · Chief Technology Officer

I can speak a bit about the hotfix, how it works, right? Now, I don't like to use the word hotfix for everything, but that's the word we have for now, right? Because not all fixes are necessarily hotfixes. Some of them just need to go out. But so what we can fix right now is a server change. So a change that goes on the game servers. That's one of the easiest things to roll out. So we can actually make a code change, and then we start deploying it, and then your game servers will slowly start updating, right? And that is the easiest one to roll out. But there's some requirements for this. There must be no data change, no asset change, and the network compatibility of the build needs to be maintained. And so that means that there are some things we cannot hotfix because it would cause a client incompatibility. And so there are some bugs that we're like, we have a fix. We can't roll it out, right? We have to wait for the next patch window. Which we are talking about, in the interest of this, to have now maintenance windows where we can actually roll out data fixes, right? Where we now need to force players to upgrade their client, but that would allow us to actually take the environment down, also do some maintenance. We rarely do live maintenance on the environment just to change things. We kind of try to keep them the way they are, but this will allow us to actually... So we're trying to nail the process now for that. And so the second kind of thing we can change is a hybrid fix. Now those are very hard to do, because a hybrid restart will disconnect all players on a shard. And so we can't upgrade a shard that's already running. And we have much fewer game shards now than before. And so that means that disrupting them will have a drastic effect on the live environment. And so those fixes are a bit of a problem. So typically it reflects in the number and the name of your shard when the hotfixes roll out.

Benoit Beausejour · Chief Technology Officer

Yes, that's what we're trying to prepare for now, is to have an official maintenance window. We're not thinking about systematically doing a downtime, but we're thinking about scheduling it, and then we'll let players know before that happens. And then we'll be using those moments to do infrastructure changes or upgrades that normally we would have to wait for.

Benoit Beausejour · Chief Technology Officer

So interestingly, that's still being investigated, but the current hypothesis on this is that the way the Pyro system was built... So first of all, the requirement when you connect, what the hell is the game doing in that loading screen? And then you open the console and it's stuff flying around. What is it doing? It's going through this context establisher set of tasks, which are all the phases for the connection. Currently, when players load into Pyro and have a long load time, it seems to be related to the way the system was built, where there's an entity in the Pyro system that holds the spacescape for the system. Now, we're peeking behind the curtains of how the levels are made, but there's a requirement for the CET connection that requires that all the entities that the client needs to load need to have an authority assigned. So this is directly a server meshing change. And so if an entity that you're loading doesn't have an authority yet, which could be any server in the mesh, if one of them doesn't have an authority yet, then the load time will have to wait for this entity to gain an authority. Now, how can this entity not have an authority? So our current hypothesis is that if the server that is hosting that Pyro spacescape entity, which has a giant radius, so every client loading into Pyro needs that entity, if the server that hosts that entity is currently rebooting because it has a crash on it, then all players will have to wait for that server to do the full provisioning loop, boot up, load the data, acquire the entity, and then players will load in. For some players, that busts the time it needs for the timeout for the login. So that's our current lead, and we think we have a fix, but obviously that's another one that we would roll out. I think it's a code fix on the hybrids that we can actually roll out for that.

Benoit Beausejour · Chief Technology Officer

I have a lot to say about that. Everybody kind of has a recipe for fixing login issues, right? A lot of them involve disconnection, shard hopping, loading into AC, rebooting twice, printing a document, I don't know, right? That is actually one of the things that we've identified as missing in our testing process. So we do some load testing before we launch a patch. And so we test with clients logging in and logging out and everything's great. What we're not testing is launch client, Persistent Universe, Alt-F4. Launch client, Persistent Universe, AC, Alt-F4. Client, Persistent Universe. A lot of the issues that we've seen in the 4.0 release came when we put in the persistent hangar flow, for example, that allows you to log into your hangar. The system for this needs to move your player, we call it unstow, your player into the shard. It also needs to unstow your hangar. Now, when players are constantly logging in and out of shards, now you've got multiple shards doing this process for you and they need to complete that process. We've identified a ton of concurrency issues with this in 4.0. Elements that we didn't test for, because you test the load, not the repeated concurrency. And so that's one thing we've identified that we absolutely need to add to our testing process. Because it's one of the other things, I've talked about this before, the difference between PTU and Live is like a 20x increase. That's also one big difference, is that on PTU players don't shard hop that much.

Jared Huckaby · Host, Community

If there's a double-edged sword of PTU, and there are many, this is the one that's always amused me. People are more patient on PTU than they are on Live, because they recognize that they're helping us, they're helping the development of the game, they're doing a job, and that patience is necessary and wonderful. But it doesn't make for an effective test of Live. And it's yet another reason we have to go to these live environments sometimes, and we don't discover these things until live environments, because Live is where Hyde comes out. So many people in PTU are Dr. Jekyll.

Benoit Beausejour · Chief Technology Officer

That's one of the key differences. A lot of the issues that are plaguing us right this second are still remnants of that. We are tracing a lot of them. I spent a lot of time on Spectrum during the break, during the workation, actually digging into a lot of those, to be able to make sure that when the team would come back we'd have some traction on some of those.

Benoit Beausejour · Chief Technology Officer

Shard lock, we never really call this shard lock. This is like a community term. You're never really locked into a shard. Your character can only be in one. And so if you get matchmade into one of, let's say, USE1B-010, everybody's favorite shard, which it is true, that shard is cursed. I mean, right now, it's always 010. I don't know why, but that one is always the one. It doesn't matter which patch, the USE 010 shard for some reason is the bane of our existence. Anyway, so you go play there, your character will get unstowed. So the hybrid will unstow your character, move it from the global database that allows you to go anywhere to that shard. Now you are, according to the community, shard locked, right? Because you're playing in that shard right now. What's supposed to happen is that if you quit the client or you get disconnected or a client crashes, when you come back, you're going to be matchmade on this shard again. Now, if that shard is no longer available during the time where you decided to come back or whatever, then that's going to give you a nice little 60K error, which will say, hey, your character is there, that shard is no longer available. Now, what that means is that behind the scenes, if the shard is crashed, there's a process that's currently processing that shard and it will stow back your character. So it's time. Time will fix it, right, in this case. During the 4.0 preview, some players got into a state where the character could not get stowed back. That was a bug that prevented that from happening. And so their character was kind of seemingly corrupted. And so we had to go in and fix a lot of those. That caused a permanent 60,030, I think is the one, where basically some players were stuck in that mode for a long time, so we had to actually do manual actions to solve that. But otherwise, most of the time, time will make your character go back to the stowed position. And then once you're stowed back, you're able to go back in there. So that's what I think the community calls shard locking.

Benoit Beausejour · Chief Technology Officer

So, brief history of transit. This is a very old system that's been in the game for a long time. And transit in the context of streaming is already a non-trivial problem to solve, and I need to dive into the why to explain a bit the complexities behind it. So the transit system is composed of multiple components. You've got peripherals. So that's, for example, the button you press to call an elevator or a tram. It's the indicators that say, hey, the tram's coming in this amount of time. Those are peripherals. You have gateways, which are locations where a carriage can stop. So, for example, if you're in Lorville and you go to the train, where the train stops, it's like a station. That's called a gateway. You have carriages, which are either elevators, trains, trams, all these things that move along the path. And then they go from gateway to gateway and they respond to orders that are submitted from peripherals. All sounds nice and dandy, what's the problem, right? So in any other game that doesn't have full-on level streaming, all of these things are always loaded. So the manager, the piece of code that actually is managing, fulfilling the orders and moving the carriages in general, or has the logic of the carriages, all this network is streamed in. It's all loaded. So it's always available. It knows where the carriages are. It knows what's happening. Everything is there. In our environment, because of our SOCS technology that allows us to have more performance on the servers because we unload the rest of the zone, that means we unload those carriages. So think about your Lorville train that you see stop at you and then leave. On the server, if nobody's on the track, it doesn't have a carriage there. It's unloaded, right? And so the current iteration of the system actually destroys that carriage and will recreate carriages when it needs it, right? And there's been a ton of issues with this. So that's the first part, right, is the streaming part. That makes it a bit more complicated. The second part is that the way the system was built initially, it would fulfill these orders, send orders basically: you call the elevator from your hangar, I need to send a lift there. The system is going to tell the lift to move. The lift moves on its own, right? So it will jump and then do the actual path. We want those to be physicalized so they're actually following certain paths. Now, the issue is that, combined with streaming, these orders and state are lost in certain conditions. And so the system just needs a whole lot of changes on its communication layer to be reliable, and it needs a fundamental change in how it fulfills these orders. Instead of just sending a message to a carriage and hoping for the best, the transit manager needs to be actually always tending towards the state that it should be in, so that it can self-heal for any kind of issues, right? Whether creating the carriage failed or streaming out and losing a piece of it, the system needs to change to be able to properly do that. So that's what the new system that we're writing is going to do. It's going to solve these communication issues with streaming and it will make the system self-healing. We also need to do fixes to the current system immediately to get that working. So now we do have a crack team on this right now, where we've significantly increased the size of the group that is looking at the current elevator issues, which we have absolutely regressed on. And they're looking at adding the self-healing component I just talked about directly in the system. So I was happy to hear that we now have proper reproductions for at least a couple of cases where the gateways are basically becoming defunct. And so you call the elevator and it never shows up. That's the number one case that we see at the moment. Or the other one is a lift shows up, you press a button and it doesn't move. So that's when the transit manager has lost this carriage or the gateway is no longer usable. And so now we have those reproductions, and they're not just happening to players. Now we have an actual case where we're able to actually go down and we've seen what actually is going on. So we can put in the self-healing things I just talked about to fix it.

Benoit Beausejour · Chief Technology Officer

They use the same system, but the streaming bubbles in Squadron 42 are a lot bigger, so they will stay streamed. Single player.

Benoit Beausejour · Chief Technology Officer

I've got to come clean first. On the ships being destroyed when stowed, that was 100% me. So that was a back-end change that we did to try to alleviate some of the load that we were seeing in some areas. And one of those side effects that was unplanned was that the location ID of ships wasn't being updated when they changed. And so the way the ASOP terminal, again, moving sands, right, does its reconciliation, which shows you what's available for you, it actually uses the location as the key element to know if a ship is actually available. Not only is it stowed, but it needs to be in the right location. Now this change that I had made for optimization did cause, for a couple of days, the issue where the locations weren't updated, and so all your ships that were stowed in other locations would show as destroyed. They were there, but they would show as destroyed in ASOP. Yeah, I'm going to take the blame for that one. That was me.

Jared Huckaby · Host, Community

Is this like if a branch falls in a forest and nobody can hear it? The ship's not technically destroyed, but if it reports destroyed and you can't call it because it reports destroyed, isn't it destroyed even if it's not destroyed?

Benoit Beausejour · Chief Technology Officer

Well, actually, claiming it would actually destroy it, too. So technically, it's destroyed.

Jared Huckaby · Host, Community

Right, but we resolved that one.

Benoit Beausejour · Chief Technology Officer

Yeah, that one's resolved.

Benoit Beausejour · Chief Technology Officer

So let's talk about hangars first, right? So, people that play more in the PTU know how the system works, but it's actually a bit special in that, first of all, the hangars are instanced. So they're instanced on your shard. You have access to your hangar based on your ownership of it. But all the hangars on a given ATC gateway, so these are the actual hangar doors on the stations, are all stacked on top of each other. So that's why there's been bugs in the past during the PTU cycles for 4.0 where you would hear, oh yeah, and that bug took a while to fix because it required a technology change. But that was also crazy. The sound was just incredible. And so what that means is that when you're in the station and you want to go to your hangar, you have two options. One of them is use the lift. Now we're just talking about the lift, right? When you call the lift, it will actually unstow your hangar, create your hangar, and then give you permission to go there, right? Right the moment you hit that button. So there's a lot of things that need to happen in quick succession. Then you go through to your hangar. There's render culling that comes into play that will make sure that you only see your hangar, even though everybody's on top of each other, that physics only affects... Now, one of the big problems that we're dealing with right now, and we've been diving into it for a while, is that the ship elevator has a couple of issues. I've seen an Avenger rising from the grave through it. I've seen ships spawning on top of each other. So there's two problems behind it. The first one is that the platform itself is moved, right? It's not a physical movement. So it's not physics driving the movement. The ship is physics enabled.

Jared Huckaby · Host, Community

That is a recipe for disaster.

Benoit Beausejour · Chief Technology Officer

Correct. When you see flapping like that, it's the combination of things, right? And so we're currently discussing a solution, because there are limitations. The way it's made is the way that they were limited to make it, but we need to fix it. Both elements are statically moved or they're both physically driven. You can't have this combination. That's what causes all these things. So that's why sometimes server FPS is good, performance is good, all your ships are going to go up that platform, everything's great. And some other times, half the ship is in the platform and that's an issue. So currently we're rejigging, rechanging the layout and the setup of the hangars to set this up properly so that it never happens again. The unstow issue is actually, we think at the moment, a bug in the timing of how, when you leave your hangar, we need to stow back this hangar, right? Because if you go to another shard, we want you to have access to your hangar. In the timing of this, if there's a ship on the pad when you leave, there's a possibility in the timing where the ship will be stowed with the hangar instead of being stowed on its own and then the hangar is clean. And that condition is not properly detected when you unstow the hangar again. And so for you as a player, you're not going to see a difference in the case where you leave with the ship on the pad, you leave your hangar, you come back on another shard and you just go to your hangar, your ship's going to be on it. But if you call a ship from the station, and the ship was carried with your hangar up and down, then ATC is not going to know that there's a ship on the pad and it will attempt to spawn it there, and that's where crazy happens. And so currently we're looking at that complete cycle to make sure that the timing is correct. There's also a couple of other issues related to this. Some corruption of data can happen in this case. And so we have an entire team dedicated to the hangars right now, just making sure these issues get solved because they're affecting everybody. I'm happy to report though that one other issue that we had with the hangars got solved, related to how you exit the hangar.

Jared Huckaby · Host, Community

Usually in a ball of flaming wreckage.

Benoit Beausejour · Chief Technology Officer

Or after 25 minutes of queuing, where the system tells you in 100 seconds you'll be able to leave. We did hotfix a change to the way that what I call the hangar queues are managed. Again, the ATC manages the queues. It's managing how you exit your hangar, which doors open, stuff like that. At concurrency, so when a lot of players are in the same location, leaving and entering, this queue and those timers would get completely out of whack. And so it would either grant too much time to someone to leave or enter, we're talking 10x the time, or give you literally three seconds to leave. And then players would get stuck in the doors, where the doors would go, and then they would try to squeeze in, and the doors would squeeze them out. And then the rest is a little bit of a party. So we hopefully have something on Live already alleviating this, but we're still tracking to confirm completely that it doesn't happen again.

Benoit Beausejour · Chief Technology Officer

And so the problem we talked about with the physicalization of the platform is definitely related, because then the obstruction code, where you can't bring down the ship elevator because there's an obstruction, initially was like, if there's a speck of dust under that platform, it's not going to move, right? Which was, let's just say, a bit too aggressive. And so I think at this point, we're doing cleanup, to just try to stow back. If one of your ships goes on a platform, we'll deal with it and stow it back. But it still needs work to make sure that even if... I mean, I remember one of the early 4.0 preview bugs was a water bottle would clip through the platform and then would prevent the hangar from functioning completely. And so now we're trying to, again, in the interest of self-healing for these systems, right? Let's not just work at preventing the condition, let's also work at what if it happens anyway, right? We need to do aggressive cleanup to keep the system working. The hangars, the elevators, the hangar doors, the ATC systems are all part, in my mind, of what I call the traversal of the game, right? And bugs that affect the traversal of the game, that prevent you from going to your ship and leaving or entering and refueling, they are of the utmost priority. They have to be solved. Because if that doesn't work, then the game doesn't work. They're critical.

Benoit Beausejour · Chief Technology Officer

Well, I've been investigating that one. I'm not responsible for it yet, until proven otherwise. But in this case, it seems that there's a condition where you can have more than one hangar in a location of the same size, which should not happen. And when the system gets in this state, it might pick one or the other, which can lead to you not having a hangar at all. So that's the thing that we're trying to find out, how it actually happens.

Benoit Beausejour · Chief Technology Officer

Well, so on the... I was going to touch a bit on ship cleanup, and not necessarily ship cleanup, but entity cleanup in general. We talked about the physics events earlier in the show, right? So the physics events are a godsend for the server performance, because we evict all these debris that are causing intersections, slowing down physics. And so that's one of the reasons why we've been able to get performance of the servers up in general. Now, one of those components... Well, the same system is actually responsible for a pink glow that shows up on a ship and will just pop it, right? And so that's part of how we try to resolve the physics intersection for ships, right? And the way it works right now is that it has a smoother number of frames where, if the physics engine is like heavy, heavy, heavy, heavy intersection, at some point it will go like, okay, something needs to happen. The way we handle that event is still a bit too aggressive right now. And so in some cases, it triggers on a heavy intersection and will pop the ship. But we don't want this to happen to attended vehicles ever. And so we're working right now at adjusting that flow so that only unattended vehicles in full intersection for a very long number of frames are cleaned up. And so right now it's a bit too aggressive for this part. But the distinct pink glow, we added that to make sure we could identify those problems, which we expected would kind of happen until we tuned the system properly.

Benoit Beausejour · Chief Technology Officer

Well, so on the inventory system, or I should say on the item system. So the way the entities are tracked, entities have entity classes that define what they are. And there are, I want to say, tags or configurations we can attach to entities for things that we want to automatically clean up. Again, we're back on cleanup because it's important. For example, AI characters will have what we call an auto-stow or auto-bury policy. Auto-stow will bring something back to an inventory after certain conditions are met. Auto-bury will destroy an entity automatically if certain conditions are met. So that gives us our ability to go, this thing is good, let's say, for 12 hours, and after that, you can get rid of it. This is true for debris or things we spawn on the side or things like that. And so a lot of people have reported losing items, for example, items that were acquired in CZs. At that point, it wasn't even a system issue. It was really those items were flagged to auto-destroy, basically. And so some players looted armor and started using those armors. And then when the conditions were met for the auto-bury, those items disappeared, right? And so that was a bad setup, right? And so now I know that a lot of those items have been re-tagged, right? And so the auto policy will not apply to them, or not if they're attended. Again, it's all about making sure that the right tags are on them. So that explains some of the inventory things. There was one that I was looking into with a player actually recently where, when you read your inventory, there's caching going on, right? And when your inventory changes, that cache is supposed to be expired. In certain locations, for some reason, that cache, especially in CZs, was causing some problems. So opening your inventory in the CZs would be empty, and then you open it again, and then you store something in it, and now other items show up. That's a telltale issue of the cache. So we're actually revamping the query cache right now for these elements to make sure it's always consistently updated. Your stuff is there. It just can't show.

Benoit Beausejour · Chief Technology Officer

LTP is a fairly complicated system, but if I can simplify it as much as I can: when you stow something in an inventory, we take that item and we process it. Let's take an easy example, let's say a single gun, a P4. You have a P4, you put it in an inventory that you own. A record is created for your, it's an ownership record that says this player owns a P4, right? Now these records we write to all the time. If you unstow that gun, meaning you put it in your hands, right? We destroy that record, because now it's in play. So you play with the gun and then you stow it back. That recreates the record. So these records should always represent what you've got in your inventory that you own. Now, where it gets complicated is that these things can be hierarchies that contain items that then need to be processed. And so we don't persist everything to LTP. Only a select set of entity classes that designers control will go into LTP. Ships are a good example, things like that. Now, when we stow a ship, we look at its entire hierarchy, and there might be things attached or loosely left in the ship that are LTP. And now they have to be added in. Also with LTP, there's also sometimes perception issues, that things should persist. A classic case is a ship left in the shard between patches. You're flying your ship in 3.24 and we release 4.0. If you leave the ship there in the shard, you don't have a ship record for it. The ship needs to be stowed in order to be there when we recover it. But we do think that there are still issues with some of the recording events in cases where ships are claimed. Because at that point, when you join into the next patch, let's say you have a ship you bought with aUEC and you store it at a location. That ship actually, its record ID comes from your long-term persistent record of that ship. In the case where you claim that ship again, we think that there might be some cases where the record data doesn't follow, and so the LTP record is not recreated. And that could be a source of some of the lost ships in the LTP data. But overall, we're not happy at all with that solution in general. And a bit like you were talking about transit, we want to do a two-pronged approach on this. One of them is, we want to find the cases that are left in the current system that don't behave the way we want them to. But we're also putting significant effort into a different system for this, which is that we don't want to have to wipe the database at all. So we want to be able to keep your inventories intact between patches, and have a forward-compatible model where we can actually keep all this data there the way it is and then migrate it.

Jared Huckaby · Host, Community

Like checking out a file from Perforce. You're not destroying it. You're just copying it out. Nobody else can write to it. Nobody else can mess with it. It's yours. And then when you stow it, it goes back in. So it's a non-destructive process.

Benoit Beausejour · Chief Technology Officer

And so our goal with this, that's a fairly large endeavor, we're not going to fix it in a second, right? We're going to still hot patch the current system to make sure that the cases that we've seen are fixed. But the longer-term fix is to make sure we only process the data that's going to be compatible. So differentiate between the serialized variables on entities that are important to a ship from those that are runtime only. That's a big difference. So those will be stored in the global database. And between patches, our goal is to just keep that database like this. So there's no, oh, we're getting records back from a thing that has been recording for three months. It's like, no, it's the exact same inventory that you have on the live patch and the new patch. So there's no loss at all. So that's our target for this new LTP solution.

Benoit Beausejour · Chief Technology Officer

Yeah, that should not happen ever. The system is supposed to be made so that we have a passenger count on each of those ship entities, and if there's a passenger, even if he's not piloting, and even if it's not his ship, we should not be despawning those ships at all. And so that's definitely on the lookout right now, as to why it happens. I mean, there's a couple of root causes. Unfortunately, it just looks the same, right? Again, if you don't have the pink glow... The pink glow is the telltale that it's physics. It's the physics detection, which we're going to ease up. If the ship just despawns out of mid-air, I've seen this happen with ATC. The timer problem that we fixed before had an issue where it could despawn your ship mid-flight when you left, which was egregious. I think that one has been solved, but the jury's still out. I want a proper confirmation on it. The one where your ship is considered unattended and you're still in it and it despawns, that needs to be solved, right, like hotfix right away.

Benoit Beausejour · Chief Technology Officer

So let's start at the beginning, right? What did we do in 4.0 for missions, right? So the way missions worked previously is that we had one server with all the entities, all the 100 players playing on it. When you played missions, there was an invisible entity created on the server somewhere, the location doesn't matter, that would manage that mission. And the way the missions work, if you can picture it in your mind, is that there's a mission management module, and then when you reach certain locations, when you stream in that location, a mission module at that location is going to actually kick in and has its own logic. If you're picking up a box, for example, let's say a box delivery mission, you get to that location, there's a mission module that pops up. It does the pickup logic. When you pick it up, it's like, oh, you need to go to this other one. Then there's another mission module at that location. Now, that's the way. And then if I add global events on top of that, there's a global events manager on top of that that was like a global thing on the server. Now, I've said global multiple times here. Every single time I said global is not compatible with server meshing, because we don't have a global entity setup, right? And so what needed to happen is a fairly large engineering task. I hate that we just said refactor, that we branded this as a refactor, because it's really a new core mission system behind the scenes, which is completely outside of the game servers and does the orchestration of what a mission is, right? And the global flow of a mission. Hey, you have something to complete. There are phases, objectives, there's state tracking, there's requirements and rewards, right? And so the mission system is what we call it. There's no longer a mission entity that's held on the server that just manages the whole thing. They all communicate to the mission system, that then takes the logic of the mission forward. Even though all these mission modules, written by the Kid Rock of missions, can do all kinds of fun things at each of the different locations. So the fundamental work was to make sure that the mission system was functional in server meshing. That was really the part of the gameplay element that was completely incompatible with server meshing. Another piece that we needed to solve in the build for 4.0, and as part of the mission system, is how markers work. And so markers are always a big issue. They've always been a big issue in Star Citizen since the very beginning. And with server meshing, these became even more problematic. So let me state the problem. You're on a Pyro game server and I'm on a Stanton game server, but I add you to my party. Now I want a party marker on you. My game server doesn't even have your star system loaded. It doesn't even know what's going on. And so for that, we needed a piece of technology that would allow the streaming system to stream your entity marker on my server so I could actually see your position changing. And so we've built all of this in place. But that's a fundamental change to how markers work. Now I'm going to dive into why we have that many bugs behind it.

Jared Huckaby · Host, Community

It's almost like we need a replication layer for the mission system itself.

Benoit Beausejour · Chief Technology Officer

Almost, for things that are tracking objectives across systems, things like that. The way we call those things, the markers are independently streamable entities. Which basically allows us to have a marker on an entity that's anywhere in the server mesh. Now that system works, right? It's in place. It's been fairly reliable, but that touches on all the markers in the game. And so a lot of issues came up for that. One of the things that we have not tackled with the refactor is how missions are built. And to Elliott's credit, I think that...

Jared Huckaby · Host, Community

We're using Elliott's name. There's James.

Benoit Beausejour · Chief Technology Officer

There's Europe, Nick, all these guys. So the credit to all these guys, right? We have not provided this team with the right tools to build compelling missions. They are dealing with an archaic scripting system that dates back to a previous era, that we are just about to finish modernizing by replacing, to give these guys modern tools and modern approaches. And so going back to markers, the markers are completely managed by game designers, and set and removed and displaced and whatever. And that makes for very complicated systems and logic that's not very robust. And so that's why, even with a robust marker system, because the markers are managed a bit ad hoc in mission scripts, it makes it really complicated to follow. I remember distinctly when we did the golden ticket event, which, that was the whole mission. There's a marker on the ticket, right? Well now, oh, the holder puts the ticket on his core armor. Now am I tracking the core armor? How does that work? The mission logic had to manage all these cases. Now we've kind of finished the foundation for making the mission system function in server meshing, and we're going to obviously keep iterating on making it robust in this setting, and fixing the missions to make sure that they also leverage the system properly. Bringing back the missions that are missing, but also improving the ones that are there so that they're even more robust. We have started the engineering work with the new StarScript foundation to rethink how missions are made from the get-go, to give Elliott, Nick, James, and all these mission designers the right tools so that they're not bogged down with details. If you saw the way they make missions now, it's involved. We want to make sure that these guys focus on the flow and not on the functionality they have to build, so that programmers can support that content. When we were talking about this change of focus, we're saying, oh, we're going to deprioritize features. We're also going to prioritize features for content as well, so that they can make content that's fun and reliable, so that missions that have been notoriously problematic, or events that have been notoriously problematic, become fun and reliable. And that's what we really want to do for the mission part.

Benoit Beausejour · Chief Technology Officer

Well, yeah, Jens did add some notes here and said, oh, if you find the Bermuda Triangle, please report it on IC. But so quantum has been, since the very beginning, when we put the first meshing test in place, that was the first thing, right? It's like, ah, quantum stops when I cross a server boundary. That's what happens, right, still. The same symptom has been, again, triggered by multiple issues.

Jared Huckaby · Host, Community

You find five causes, the sixth one shows up.

Benoit Beausejour · Chief Technology Officer

Correct, exactly. And the latest ones are related to the resource container. I don't want to dive too deep on this, but entities have classes that define what they are, and they have components on them that define their functionality. So if I have a box with a health component, then the health component manages its health, its damage, and its destruction. The entire game is made like this, with components that define things. The quantum drive is made up of multiple components, and these components need to deal with the change of authority when we cross a server boundary. There's a lot of entities intertwined for the quantum drive to function. Fuel, resource container and quantum drive. And we've been fixing issues with this transition on these entities nonstop since the beginning, because there would be a fix, then it would work for two patches, then a change to how we deal with jump points would come in and then re-break it. And so we are in deep regression on a lot of things. And so to solve that, we need a lot more focus on validating the issues. It's like, yeah, we fixed this. Okay, let me try transition one, two, three, four. I don't want to say more QA, but more QA. We need a lot more QA to validate the fixes.

Benoit Beausejour · Chief Technology Officer

Prisons is, and I've learned about this inquiring a bit about the mission, the prison bug, right, is actually a mission. And it's a very old mission, right? So it's a mission module that runs the prison logic. And so for those, I mean, I'm sure everybody's familiar, but for those who are not for some reason, there's a bug that lasted through preview in the beginning of 4.0, I think, or it was fixed just before that, where if you were a criminal and you got caught and you ended up in Klescher, you got what you deserved. Well, I don't know. But you could try the prison escape, which didn't work because the mission logic that drives the prison escape is tied to this. And so the codes weren't outputting correctly. The logic was basically made for the old mission system. It was not ported. That's part of the refactor. And so it basically depended on, when you get to that location, giving you a new mission instance for your prison experience, whereas the new system requires that these modules are persistent and stay there, which accumulates the logic change. And so the logic becomes desynchronized with the state of the mission. So what that did is that you would literally do your time, either use smuggled-in Hadanite from somebody else or mine it, pay your dues. And then the elevator will... to the prisoner who's paid, blah, blah, blah. And then you would press the elevator. Again, elevators are back. Press the elevator to leave. And then it would never detect that the action area that you're leaving is actually triggering. And so the guys have done a hotfix for this in preview, actually at the end of the preview, I think, where they've moved the logic for this leave-prison flow to the actual law component that's on your character. And so now we're guaranteed that no matter how weird this flow would get, it's actually going to be reliable.

Jared Huckaby · Host, Community

We are two hours and 30 minutes in. You just used the phrase, it's guaranteed to be reliable?

Benoit Beausejour · Chief Technology Officer

Let me rewind. It should be a lot more reliable, because your player's always streamed in. I have a fairly good confidence in this fix. As long as the elevator works.

Benoit Beausejour · Chief Technology Officer

Mining has actually been a dreaded gameplay loop in 4.0. That's one of the reasons, when we did the evaluation with our 3.24 or 4.0 preview, whatever, we did take a look at mining. Like, is mining functional? Is mining in the Prospector functional? Is mining in the MOLE functional? Is mining in the ROC functional? Is hand mining functional? It actually was intermittent in the first report, but we've identified what we think is causing this delay. It's basically the conversion from the render node object, because the mining and harvestables are always in large quantities. They could be a very large performance hog. And so they're highly virtualized. And so there's a conversion from render node to entity that is, for some reason, very slow. And so we're diving in to try to figure out what's going on. But that's what's happening. So that's why it takes time for them to show up properly.

Benoit Beausejour · Chief Technology Officer

There was a bug that caused everybody in a given mission to get the reward for the missions, right? So that means if a mission payout was 10K, then everybody would get 10K, everybody participating.

Jared Huckaby · Host, Community

It was amazing.

Benoit Beausejour · Chief Technology Officer

Well, so the funny bit is I talked to Rich Tyrer about it and he's like, I couldn't catch it before they fixed it. That's what I want. And so ultimately, it wasn't exactly what we wanted, but that's the direction we want to go in. We want to encourage you guys to play together. And so having those rewards divided is kind of a counterpoint to get people to play together. Why would I group with you if I'm going to get half the payout? And so the thing is the system doesn't work the way we want it to. So the fix did go through because it was an easy fix and we had the right persons in place to do it. But that's not where we want this to be. We do want to improve that part.

Benoit Beausejour · Chief Technology Officer

Well, so we're still investigating the cause, but for now, right, we think it's related to a specific driver version from NVIDIA, related to the upgrades for the 5090 cards. So we're not quite sure exactly, but when you update your NVIDIA drivers, there are two drivers, one for Direct3D and one for Vulkan. It doesn't look like it, but there are two. And so we think that there's an issue related to the latest version of the driver for Vulkan specifically. And so you could downgrade your drivers and it probably solves the issue in the meantime. But we're still investigating to make sure it's completely not related to this. But other games are also having the same issue.

Jared Huckaby · Host, Community

Yeah, there was one or two drivers ago where I saw somebody, they had glowing heads. All the character heads were glowing, and they were on Vulkan.

Benoit Beausejour · Chief Technology Officer

We don't think it's a third-party app in this case, though. You do need to be careful with third-party apps that inject into the 3D pipeline, for multiple reasons, one of them being that the EAC sandbox can catch it, but also because they can cause GPU crashes.

Benoit Beausejour · Chief Technology Officer

The first thing I would say is that the Issue Council at CIG is of extreme importance, right? Even though for players, if you don't feel like your issue is being actioned or that there's retroaction from the team, you can be certain that we have, especially Bayer's group is at the forefront of this, but Issue Council reports are significantly of high quality in general when they do pass the filters, of course. And so we use that data every day to confirm issues. A lot of the issues we have on our famous director's list that we talked about are not internal Jiras. They're IC reports, right? And so that process is super key for us. It's a very difficult signal-to-noise ratio process, though, because the level of effort put into reporting is not necessarily smooth. The quality range is large, is what I'm trying to say. Some reports are like, game, no work. Others are extremely filled with evidence, and those are the ones we want, obviously. We do want to improve the dupe system. So I've seen several backers on the ETF and elsewhere that are very active at calling out dupes. And for now, I think that their voices are kind of muddled by people saying, oh, I can reproduce, right? Instead of saying, oh, I confirmed this dupe, right? So we're actively looking at how we can improve that flow, because when we identify a duplicate, it's a major win. So when you contribute to the Issue Council, a new report, depending on the issue, takes a lot of effort for it to reach us, because you need to get reproductions on it. Reproducing another issue is a great contribution, but identifying a dupe is probably the best contribution you can make, because that helps us a lot. And for now, that's not leveled enough. So we want to change the way the Issue Council works to make that a lot more at the forefront, for sure. But the prioritization and the upvotes have historically been used, but in this new process, we're also trying to lean into this a lot more. And so the please-fix hammer button, right, to upvote an issue, we're going to try to give it a voice, and actually, some of the internal discussions we're having, I can share some thoughts if you guys are interested, are even to have community-identified issues directly in the director's list, like make the community a director in this list.

Benoit Beausejour · Chief Technology Officer

No, that's a balancing act, right? That's why, for now, I feel like the dupe markers are kind of more experienced testers that are helping us. But de-duping will happen typically when we actually intake a bug from the community, and we're going to try to repro it internally. That's where we're like, oh, it's been marked as a dupe, but it actually is a new symptom, or it's the same case as another one or not.

Benoit Beausejour · Chief Technology Officer

No, not so far. Not so far as to say, oh yeah, there's this going on. The worst things that we've seen so far in IC would be things like brigading a specific issue. And that's also why we've been kind of careful with rewards for the Issue Council. It's not because we don't want to reward players that are doing stuff there, like we are doing some of them, but we're always trying to stay asynchronous so that you can't go, oh, they're going to reward for this, so I'm going to go and create reports, or I'm going to create dupes, or I'm going to create whatever.

Jared Huckaby · Host, Community

Obviously, you're the person who would know, and it's not that I don't believe you. What I've seen and what I'm reading in chat right now... Would you commit to maybe taking a look at that and seeing if maybe it is being manipulated?

Benoit Beausejour · Chief Technology Officer

Absolutely. Yeah, I can absolutely look at it. Especially if there's a sense that that's happening.

Benoit Beausejour · Chief Technology Officer

On error codes, I'm going to go and merge pop-ups into that. Because the error pop-ups date back to the very original start of the project. Nobody's touched these things in forever. So the nicely misaligned cancel button, it's my favorite pet peeve, right? Why haven't we fixed that button? Well, we're fixing it. So we actually have a CL right now to change all these pop-ups to make them at least readable and aligned, right? We start with that.

Jared Huckaby · Host, Community

No, you start with the elevators, then you fix the button.

Benoit Beausejour · Chief Technology Officer

So the error codes themselves, right? So there's already good categorization for issues. So every kind of, like 30,000 errors, 40,000 errors, 60,000 errors are all families of errors. Unfortunately, over the years, we've added more systems, we've grown the infrastructure, and there's a lot more causes for exit or fatal errors than there are actual codes assigned in those categories. And so what we're doing now is kind of seeing if we can surface more information. 30,009 is typically related to your character during login, but there's a million causes that could cause a 30,009, whether it's that the shard can't unstow your player or your player is missing a component. And we're trying to surface more of that, so that there's more of these cases, so that you, as a player, know a bit more about it. Also, the wording for some of those errors is being revamped, because they're not always very clear. And so adding more details and more granularity to our error codes, changing the wording, making sure they're more effective at representing what they are, and eliminating a bunch of the... There's a couple cases where we fall to a default error code for kind of unknown conditions. That's what we need to flesh out, is the unknown. What is the unknown condition? Try to surface more error codes from the other systems to get more of that information in place. So we need more clarity on that side, so that players aren't going, oh, again, a 19K error, what does that mean, right? I go in Spectrum a lot, in SC testing chat, and I see players going, what's the fix to 19K errors, right? And a raw 19K error is your client can't talk to the backend service at all. So it's typically a client-side problem. Firewall, machine connectivity, something like that. But the error message itself is really unclear. It's like, oh, we failed to log you in. Well, what can I do as a player to fix that? So we're trying to get a lot more clarity on these error messages. That's part of the improvements of these things that have been plaguing us forever. These goddamn pop-ups are one of them. Hey, you want a funny one? Here's a funny one on these pop-ups, right? Because we're all complaining about them. But we lost those pop-ups during the 4.0 preview, right? Everybody was having a black screen. Everybody was complaining about a black screen. I got a black screen, right? Well, what actually happened is that someone that I shall not name, right, but a developer, fixed another issue where, when you die, we need to move the camera and move your character, and it looks like crap. So we didn't want to see that. And so the way to solve that was to fade out the game UI, including the pop-ups. That was it. And so in this case, if you got disconnected, then the pop-up would fade to black.

Benoit Beausejour · Chief Technology Officer

I mean, I think I want players to know that we know about the state of the game. We're not oblivious to it. We really care about making the work of our teams shine. And we're going to work our ass off to get this game stable and solid and to be the game that we want it to be. I think it's about time we do that change.