Star Citizen Live | Tech Talk w/ Benoit Beausejour

Well, I want to go a bit further out than just the beginning of the year, right? Because let's take same date last year. So let's go full one year back. The state of the game one year in the past was still single server DGS, right? So we're talking about maximum 100 players per game shard, one server simulating all of the universe, no second star system, right? So only one star system. Server performance, not good.

Four or five frames.

Yeah, if you're lucky. When you load it in. Yeah, the parameters for us to launch a patch was basically seven-ish, which is horrible if you think about it. And so that was last year, right? So we start from far, right?

We were still in the clutches of 30K.

We're still in the clutches of 30K, correct. So we did have some level of server recovery, but a lot of issues related to it that could have side effects. Some of them are still present. We'll talk about some of those. Fast forward to now, the assessment is a lot better, I think, in my mind. Obviously, we've had the release of 4.0. It's been a bit of a roller coaster. I'm not saying it's been just a straight line. We've taken some dips and then improved the stability and the quality of the game overall. Obviously, we have the 4.0 release, which came with static server meshing, which was the element that we needed to free up server performance. So on average now, when we look at server perf, other than some egregious problems, which we've identified, we get per territory stats now. So basically, we don't just get like server FPS is X, right? We're going to know Stanton server FPS is X or Hurston server FPS is Y. And so we can compare depending on which territory, what performance we're getting. And FPS is just one of those parameters, right? There's a lot more. But let's stick to FPS just because it's a simple mental model, because it does have an effect on your gameplay. So nowadays, on average, server FPS is up 20 easily. Obviously, some zones, like let's say when we launched, I think, Stormageddon, there were some issues affecting those zones that made those servers dip below that quite a bit. And so they drag the average down. But overall, the game server performance is way, way better. And I think having 600, 700, we've pushed up to 800 players per shard, which is something we're pushing at right now. We'll talk about this if we talk about server meshing a bit more. And so we're going to be pushing this really hard moving forward. But I think that if you compare last year to this year, in my mind, it's not the same game. Completely not the same game. Now, if we look at even the stats from our players, session length is up like 10x. It's not even comparable. It means people play for longer periods of time and they stay online. I think server recovery is a big help for that. The stability of the game server has also improved dramatically. This is all because of the efforts of the changes that we've done, not only to the priority calls, like you just explained, which is taking where we're going and trying to slowly shift it to change direction. Because a team of this size doesn't switch on a dime. It takes a long time for these changes to trickle in. But now that we've got kind of the big organizational changes in, we're starting to see it and reap the rewards of it, which I hope is showing in the game. Anyway, it's showing for me. So I'm seeing it. So compare one year back to now, I think it's night and day. There's still a long ways to go. I'm not saying we're happy with this. I'm just saying.

Well, so there's a series of things company-wide that have had to change for this to happen. The first one, we talked a little bit about this in January, which was the idea of having hero cards or joker cards, I call them, on specific issues. And it was funny because people were like, give the joker card to this other bug. And I was like, yeah, that's basically what we do. But we've extended that concept a bit further. And I hate that we changed the name, but we did change the name, and we call those hero squads now. So what we've done is that we've taken a subject, for example, transit, which we've extensively talked about in January, and we've made a hero squad around it. So that means we've dedicated resources just to all the bugs related to one system. And we've changed the methodology. So the way that we used to deal with bug fixes in the past, it's basically engineers get the bug report. They put it in their task list. At some point, they emit the fix, and they ship it in the patch. Now, as the patch release comes closer and closer, those bugs get verified and or need to be accelerated. And so it's always a bit of a, the fixes are valid, they're correct, but we don't have time to test them fully. We've had a lot of regressions, right? And so regression was becoming a problem because we were going too fast to try to solve some issues. And so some systems like transit are complicated enough that you can't just emit the fix.

I want to pause you there. What is regression?

Regression is when I say I fixed bug X and I've created bug Y. So it's basically when we're introducing new bugs or breaking other things because we try a fix. And so regression was a big problem on the project in general. That was in part because of the speed at which things were going, the code review process, things like this. So these hero squads, basically, we flipped a bit the relationship. So we decided that fixing issues, let's say, I'm going to keep transit as my hero squad example because it's an easy one to reason, but we've put resources dedicated to it. And then we got the QA teams to basically drive the priorities within the squad. So instead of saying, hey, production management is going to drive the priorities, we got the QA leads to do that. And so now these guys were completely driven by player impact. And so when they look at issues, they're like, is this impacting players or is it a fringe issue? And they decided on the priority. And it turned out that that priority accuracy was a lot better. And we asked those teams to be relentless into what they want to reach. And so classify the issues in order. And then the moment we get a green light from these guys, it's a celebration because they are hardcore.

No, that one's a bit of a separate one, actually. So the hero squads is really about getting our QA team to drive the priority on a specific subject and drive down, right? One of the changes in methodology for the devs on those teams is a bit of a change in approach. In code review in these hero squads, we no longer just accept the fix based on code change. We ask the developer to explain what the problem is: can you reason why this is happening? Second, propose a change. And then the leads can validate this change before we start even writing any code for it. And then we verify the change with QA. And then that's when we actually allow the change to come into the stream at all. So we do a lot more work before the work is actually started, a lot more work validation before it actually gets submitted. On those teams, developers working on those hero push squads have a much more methodology to apply to solve that. Because basically, in the past, the regression rate was too high. And so the big advantage of that setup is that those teams are faster to react, because they're dedicated on one subject. And so if there is a regression, because they still happen, they're faster to pick it up. But then the fixes are slower to put in. But that still has been very effective. So transit is one of our hero push squads, which to me has been kind of successful since the beginning of the year. I think the transit system, we're going to talk a bit more about it later, it's in a much better shape. The embedded green light is what you just talked about, which is, that was a major organizational change. And so in the past, the way that game devs at CIG on Star Citizen would work is that they would submit and test later. Now that sounds crazy, but it is what it is. And it was what it was. And so at the beginning of the year, we completely changed this. And so we added this embedded green light process. So what is an embedded? So embedded are QA testers that are embedded in a dev team. Everybody has a view of there's a floor of QA people and then they will receive a build and they test the build and stuff like that. But there's another class of testers that are embedded in teams and they become specialists in what that team is doing. So if you have an embedded tester in the FPS weapon team, those people know all about FPS weapons, how to test them, what change affects what, and so they can judge what they have to test. So the embedded green light is we gave that team the power to red light any change before it's submitted. And so that's a big culture shift because it used to be hot and fast. Get it in. We'll test and fix later if there's issues. And we trust devs. We don't trust devs. We need to verify. We need to check it. And the whole point of this was to hit on regression or side effects because our rate of regression and side effects was extremely high.

No, I mean, we've had embedded QA for years. This was about empowering them, in order to be a bulwark against letting the stuff through.

This is not about BO. That's what you're referring to. But hygiene is a collection of initiatives related to cleaning up errors, logs, and in general, just reports we get from the game engine that can signify errors or not. So if anybody's ever opened a console in Star Citizen, you probably think, whoa, what the hell is going on? Because there's tons of messages going through, all kinds of stuff. Well, that's what hygiene is meant to clean up in a way. Now, these things will take us a long time to clean up, arguably. But we do have the same problem of signal-to-noise ratio in other areas. So when environment artists are working in the editor, they get level warnings. Well, there's a ton of those warnings that are either irrelevant...

Asserts.

Asserts, warnings and errors. And so the issue is a level artist is going to get some warning. It's like, can I do anything about this? Oh, there's 50,000 of them. Probably I don't have to worry about this. The reality is one of those is actually an issue. And so the hygiene initiatives is like a collection of things that we need to change in how we build the game to clean up these things so that the errors are meaningful. We have the same thing on the DataCore side. So for those who don't know, the game data, so all the records we use to configure the components or all the game data, is just stored in a database called DataCore. There's a ton of errors in DataCore that are reported, schema violations, things like that. They're not fatal errors, but they're still there. And we allow them to be there. And that's historical. And so the hygiene initiative is a means for us to reach zero errors in those areas so that we can then prevent them from even being submitted in the first place. So hygiene is basically pre-submit verification on things and concerted efforts to solve these things so that we can be in a clean house. It's like we want to clean the house so that we can do better.

Somebody in the chat was saying errors that should be warnings, warnings that should be infos.

Yeah. Bad classifications of things. So we even have now systems put in place in the assert dialogues that pop up so that you can report a bad error. Literally, now we get reports for bad errors so that devs can go in and fix the wording or change the severity or things like this.

Yeah, so that's exactly right. I feel like an AI when I say that now: you're absolutely right. But yeah, we changed the way the assert dialogues work. So asserts, for those who don't know, is like when I'm a programmer and I'm asserting in an area of the code that this value must be set to 1. That means that if the value comes in and it's not 1, then we'll have an error that will show up. That's the assert error. And typically, you need to action asserts or something else is going to go wrong. And so historically, the assert dialogue was just a global thing. And then we still have those get assigned to dev teams. So they are classified by code owners and by dev team. And so you can know if it's an audio assert or if it's a programming assert or whatever, which is great. But we allowed people to ignore those either completely or just skip them. And on the other hand, when you're actually testing the game or playing the game, you get these asserts. So you want to ignore them because you're doing something else. You're not in assert fixing mode. We've changed the relationship towards asserts inside the company so that now you can ignore errors for any other team but yours. So if you get an assert error, you need to fix it. That's it. Because the thing that we were seeing in the company was basically teams going like, oh, when are these... I'm going to pick on audio. Sorry, Graham and team, I'm not picking on you guys, it's just random. But like, when are the audio guys going to fix their asserts, right? When, as you know, the reality is we all have our own asserts to fix. And so we changed that relationship as well.

We even found some, I'm not going to blame anybody, but we've got analytics inside the company, so we know who's turning the asserts off, right? So we found some shit.

Yeah, so stability score is, I don't think we've ever discussed this publicly. So we have this measure, which is an indication that we use, and the team that actually comes up with this would already tell you it's alchemy, right? It's not an exact science, but it does give us a pretty good idea. So we try to establish a stability score for every build, right? We actually auto-calculate it based on the data that we get.

Not patch, every build.

Every build. Yeah, every build. So we send a build to PTU or QA or staging environments or whatever, and we always have a stability score that's calculated over time. And so what the stability score is, I don't want to go into the alchemy of it, but we're trying to figure out basically the stability of the environment. And so if you're connected to a game server and you get disconnected, we're going to count that. Now, the reason for your disconnection will be used as part of that. How long the build has been running is also part of that equation. And the concurrency of it is also part of it. And so with these black magic numbers, we basically get a score out of it. That's one thing. Now we've put labels on. So we used to use the score internally as just a number like, oh, it's a 0.08 build. And then at some point, I got a bit fed up with that because I'm like, what does that mean? And so the values for this are optimal, excellent, stable, unstable, volatile, crippled, or critically unstable. So that's the range, right? So we go from optimal, which is a great stability score, to critically unstable, and all of the things in between. And this is based on all the stats that we get. And we kind of auto-calculate it. Now, we still have Jeffrey Pease, internal alchemist, always playing around with these numbers or adjusting the score, telling us how bad we're doing. But it is actually pretty accurate in terms of what the perceived stability will be from a player's perspective. And so if I compare, what was September 11 last year? Was it like 3.23 or maybe 3.24 at that point? 3.23 or 3.24, I forget which one. I would say the stability score for 3.23.x was probably in the kind of unstable range, right? That would be a normal range for the 3.2x, this kind of thing. So the stability score, server score for 4.3.0, for example, is between optimal and excellent. And so that means the rate of server crashes and disconnections overall is really good. Now, this score is not just server crashes. It's also client crashes or client disconnections. So if everybody gets disconnected because one of our cloud routers dies, then the stability score of the build will be terrible, even though the servers are up and they're running real fine. So it kind of encompasses everything. We're trying to get one number for all of it.

That was critically unstable. Yeah, it's funny because actually the server, if you just took the server score for 3.18, it wasn't that bad, but it's everything before. That's why it was so critical in the 3.18 era. It's because it was really a massive backend problem at the time. So yeah, let's not talk about that. We're past that now.

So yeah, so that's the stability score, right? So that's an interesting measure. And that proves that we've kind of really improved the crash rate on game servers, so the server errors that you get. Now, we need to do more on the stability score because with static server meshing, even the stability score is sometimes skewed. We have run into area-local crashes that would crash loop a server nonstop. And so that really contributes badly to the overall score of the build. But that's because it's one area. So we're discussing how we can do more territory-based scoring as well, so that we can get a better idea. So basically, what I'm saying is we can't get an actual single number for all of the stability stuff. There's always some factors; we've had AMD driver or NVIDIA driver issues that affected stability scores in some builds and so on.

No. We always try to shoot for better than live when we can. But there's a lot of things that we look at to be able to ship a patch. Now, obviously, for this year, it was a very big change for the development teams because moving to a four-week cycle means that we are always shipping. We have a ship date that we give ourselves and a schedule. And then we've got our gate reviews. And everything is chained up to get there. Well, the closer you get to this date, from the player's perspective, when you guys see daily PTUs go, go, go, and it's always update, update, update, that's because we're getting close to the release. And so we're trying to get to the date, and we're trying to validate all the things we need to validate as fast as possible. The focus is really on shipping. So even minor issues become top priority. And so it's been really difficult for the teams this year to keep up with this because we finish a patch like, yay, OK, we're late for the gate review. Literally, it's going like this. Obviously, we balance this across multiple teams, but there are teams that unfortunately are on everything, and so they're always on it.

Well, so this one is a bit more subjective because we don't specifically have a score for it. But if I go back to last year, the way we would develop the game, and I don't want to go too deep in the discussion about source control and streams and all that stuff because that's just a rabbit hole. But in general, when we work, we have a mainstream where we work in. In the past, this was a melting pot of everything. So core technology, editor, tooling, audio teams, content teams, gameplay teams, art, everything in one stream. And the fallout from that is that you go a bit faster because you're building things. You're improving the tools as you're improving the gameplay, as you're improving everything. But the reality is we also had major internal stability issues, which is, imagine that you're an environment artist. You show up at work. You make your little coffee. You pull the morning build. You're like, all right, I'm ready to work on my ASD mission today. It's going to be super cool. Editor crash, right? And now, oh yeah, the team's already on it. We're fixing it. But you just lost two hours. And so we had major issues with this. And so when we started the year, actually, when we did the show, just after the show, I think we went and finished this new branching strategy where all of the patches this year have come out of the same stream that we call SC Patch. And this patch receives almost no technology update. It's basically us saying, we can do this content with no new engine tech. No editor update, no tooling update, nothing. Now, that's not to say we are not touching the engine. We've fixed bugs. We've improved stability. But we bring these things in with embedded green light, full verification, tooling check before they go in. And so it's really about moving all these checks beforehand so that we don't destabilize the content teams who are under the gun all the time to produce the stuff that they have to, and we don't want to slow them down. And so that's another reason for why the stability is better. It's because we are also not introducing massively breaking changes in the engine that we then need to debug at patch time, right? That's been a big helper on stability at the cost of features, right? That was the whole point of the year so far.

I was saying the session length is calculated when you connect to a game shard, right? So sitting on the main menu doesn't count, right? It's really when you're actually actively connected to a game server. And obviously, it ends when you disconnect. So that sounds like a simple metric. And it kind of is. But it's very important because it gives you an idea of, you can have very high concurrency and a very short session length. And that means that's a terrible patch. It means people are going in... Let's talk about the patch we should not talk about. When you would get in game, if you could get in game, and then can't get out of bed, 30K, end of session. Super high concurrency, but very low session length. And so the session length is a bit of an indicator of the quality of the game, but also of the desire of the player to keep playing the game. And so the longer the session length, it typically means that, hey, the content's being engaged with, players are playing the game, blah, blah, blah. I'm sure some people go out in deep space, and they mine, and that's it. And they can mine for hours, and they pull the average up. Other players go PVP for a couple of minutes, and then they exit, and then they drive the average down. But overall, the session length is always a metric that we look at as a secondary metric to look at the quality of this. The session length has more than doubled since the 3.x era of the game, easily. And obviously, in the case of server recovery, your session continues. And so that helps quite a bit, because before it would cut the session in half. So there's other factors that contribute to it, but that's why it's an important one.

So, let's start with: we didn't have tech previews in, let's say, April of last year. We started this idea of having tech previews last year. So it's a new kind of tool that we have. And obviously, we used tech previews in the server meshing lead-up to validate key assumptions, to actually isolate the testing. If you remember when we did the major technical previews on the 3.x branch for server meshing, it was taking the actual previous build with server meshing technology. So uninfluenced by new code, uninfluenced by the current patch, something we know is kind of in a stable range that we can use to test the technology alone. It's also a way to de-risk just bringing that in the stream and then going Cortez maneuver: now we're stuck with it, we're moving forward. And so the tech preview has been invaluable for that. And especially for the server meshing project, that's been a super important tool, because we just don't always have the ability to replicate what players are going to do. And to be honest, even with bots and virtual players and whatever, it's never the same. You know, 4,000 players filling ships with hot dogs.

Yeah. So this tech preview that we're bringing in is more than that, actually, behind the scenes, right? This is literally, I told you about SC Patch. So all the patches this year have been coming from SC Patch. The tech preview contains all changes that we've brought to our mainstream. This is where the engine development happens. This is also where we bring back some technology from Squadron 42 into Star Citizen. And we are moving all of this together up to SC Patch. So this will now become, when we're happy with all the testing that we got through on this, and it checks all the green lights, the new baseline from which we build patches. Right now, this includes about eight to nine months of graphics changes, right, which have not seen the public yet. And so a lot of Vulkan changes, obviously, to the great pleasure of our great Ali Brown, who really wants us to finalize this merge so that we can bring these to players. And there's some great stuff in there. So a lot of Vulkan optimization, Vulkan changes, but also, in a way, a way for us to test all of these changes that we've been doing to the core engine rendering outside of the lab. Because we have a graphics lab internally that we test on. But there's never anything like shipping it to players and having all the different kinds of hardware and configs and settings and driver versions that we need. The majority of the stuff in this tech preview is player-facing, but there's a ton of back-end tooling that comes with that. A major update to the editor with some new material addition systems and light group systems that are in dire need of coming in, and major upgrades to our scripting system that powers missions, which the mission teams, Elliot and his team, are like, when is it going to be done?

Now, to be clear on the mission tooling there, this is part of Mission 2.0. It's a prerequisite, but it's not the Mission 2.0 system. That's a much larger endeavor that we're still on. But this time it's tooling. So if some players are tech savvy here, it's basically a visual scripting system that we're upgrading massively, which I think we demoed years ago at this point, but it was called Subsumption at the time, and it's now called StarScript. And so the StarScript system is basically taking the tooling that the teams are using now and making it much more reliable, robust, with syntax checking, error checking, validation, auto layout, stuff like that, that the teams really need to build more quality missions.

Well, so what's behind this, this enables us to bring in... I'm going to go back to streaming just to build the excitement. No, I'm joking. So off of our main technology stream is another feature that's been built there that needs all of this to work, and that's the engineering gameplay tech preview. That's right behind it. So the moment we can bring in the base tech preview changes, then we'll be able to run the next testing on engineering gameplay straight off of that patch. So that's going to be super interesting.

When this is done and gets fully integrated, goes to live and whatnot, the next thing is engineering. We desperately want this tech preview with engineering out before the end of the year. Now, this is the middle of September. There's still October, November, December. That's still a lot of time. It's not anything anybody would say is soon. And there's no guarantee that we're even going to hit it. Me saying it's our intention to get this before the end of the year is not the same as saying it will come before the end of the year, but it is everybody's desperate intention to get this thing into Tech Preview at the end of the year.

Hey, but on this, I got a little segue for you. So there's one thing that kind of proves the improvements on the live service: there's a lot less players on PTU and a lot less players on tech previews than before, which is a good thing, right? That means players want to play the real game and I'm really happy about that, but it does mean, if you want to help, join those tech previews, right? The more people we get, even if it's for a short period of time, it's really helpful because we get a larger set of people that can try it out and that gives us more data.

That's one thing I forgot to put on the list of things we've changed this year, which is that we take maintenance windows. We used to never do that. Now, why not do this every day, right? The issue that you get is that it's not just resetting the server. So rebooting the servers, reloading the game shard from persistence and continuing, you're going to get right back in the same situation. So let's imagine. We're going to talk about cleanup density a bit later, but let's go back to the 3.x days where we persist all the data in the shard and we have deep intersections. So a ship entered another ship. It's an intersection. It's costing on the server. The server FPS is now shit. Super low. Physics threads using all the resources. And then you're like, OK, cool. Let's do a daily reset. It's going to fix that, right? So you go through it. You reboot it. Cool. Now the server reloads from persistence, and those ships are still intersecting. And so you're right back where you were. So you would look at the graphs, and you'd see FPS super low, zero during the reset, 30, then 5 instantly. And so what we've done for this is, now we have way more systems to deal with this, to reclaim performance for these cases. So first of all, we detect them. But also, we do emit fresh shards periodically. So we kind of decide when that happens. When we do massive weekly maintenance, we don't take it every week, but we do it when we have to make configuration changes or things like that. We'll take a downtime. And we're probably going to emit new shards and or run the repair on some of those shards. And then that will help. So we kind of do that. We just don't do it systematically, because the reality is that's not completely how it works for the backend persistence side of this. And it's not something that we can solve easily because the conditions are kind of too great. You need the shard to be live to solve these things.

We're going towards that now. Basically, right now you can almost consider that every Tuesday we try to do a maintenance. Typically early in the morning UK time is a good time in general, just because that's where the least players are playing typically. But that said, we don't take it every week because we don't always have to do it. And so sometimes it's not very necessary. But I'm an advocate to always take it, so that sometimes we use it to accelerate hotfixes. So in case you haven't noticed, we do hotfixes all the time. So we have these new deployment systems now. And we're able to roll hotfixes way faster. But when we roll a game server hotfix, it's going to take time to... what's the word that Ahmed uses? Percolate. It's going to take a while before it percolates. And so sometimes you can have a game shard that has 10 servers, two of them have the hotfix, the others don't have it. So depending on where you're at in the shard, you have the fix or not. And it takes a while before all the game servers actually run it. Now, obviously, if it's an unstable build, it's going to be much faster, because then the server crashes, and the new instance comes back with the new build. But these maintenance windows give us the opportunity to also synchronize that with a client hotfix when it's necessary. And so we can take everything down, patch the client, patch all the game servers in one go, emit fresh shards if we have to, and then start again. And so that's why typically there's big changes that can come about in maintenance, and we want to use them more.

This is a beautiful one. That was fun. So yeah, like you mentioned, we're walking up to 4.3. We're at the point where pretty much everything has been validated and tested. There's something with the Wolf that we're going to try to do more in the future, which is that we try to respect what we call the art of surprise. Typically before, we used to put everything in a patch, go, go, go. Now for certain things, we want to try to respect some level of surprise. We find that it's cooler for you guys when you discover things when it goes live and not necessarily on PTU two weeks before, right?

Or in shows every week.

Or in shows every week, exactly. And so the Wolf is part of that. And so the Wolf was going on PTU for validation because it was part of the 4.3 release, at the very last minute. So I think 24 hours before. Just to make sure everything's fine. Well, it turns out everything was not fine. And so we look at the reports. There was one crash that we had to solve, which was solved rapidly. But then we looked at some very curious reports, and so players would basically report, I forget the exact IC, but something like: I landed with my Wolf, exited the ship, and then was walking around. Then my friend took my Wolf and flew away, and then the world disbanded around me. So initially it sounds like a fluke report, but then a bunch of people reproduced it and it was like, I have this, I have this, I have this. And so, what is going on? And so typically that's where our player relations team goes like, something's fishy here. So we started to look into it. So we assembled a quick crew to check this out. And now this is kind of a skeleton crew because we're moving to release. Typically, it's very late here. I'm in the UK right now, so it's very late here in the UK. US teams, it's night for them. So just get this crew on together. Start looking at this issue. And so basically, the issue that we're seeing is that sometimes... when a bug starts with 'sometimes', you know it's going to be a fun one. So sometimes when you exit the Wolf, you desync between the client and the server. And what actually happened is, so we have this tool we call the server renderer, which allows us to basically view what the server sees, right? And so think of it like a side camera that we have when we have access to the game server and our game client, and you can see what the server sees. And so you'd see my character get out of the Wolf with that super nice animation. And he jumps at the end, jumps off the ship. This is super cool. And then I'm walking around on my client. Everything's fine. Game server: I'm literally in EVA animation floating over the planet. And the thing is, I can't exit this condition. No matter what I do on my client now, the server will always see me attached to the ship in EVA. So we're like, OK, what's going on? Now we start looking at this. So at this point, John Crewe and I are losing our minds trying to already get a proper reproduction. But then we realize, oh, we put new animations in for this ship. So the Wolf was one of the new ships that have enter/exit animations that are brand new, just captured, imported in the game, all that stuff. We don't do this all the time. And putting animations in the game is not just, oh, put the anim in. It's timing. There's events to link them. There's linking them to the actor system, how they get triggered, inserting them into ship pipelines, stuff like that. So we start debugging this. First step was, oh yeah, they probably forgot to put some of the seat events on it. So we do a first patch. We try it out. Now the QA guys come back like, nope, still happens. So, I think the timing is wrong on these events. So let's change that. Now, I think we went through like seven or eight iterations of fixes to try to figure out what was actually happening with that exact animation and why the ship system was disconnecting it. We went into full debugging on the actor. So the actor is your character and all the states it can go through. And we could just see the two differ. But it made no sense. Why is one working and the other not working? So at this point, we're kind of running out of ideas. That's when I called upon, I used a button, called the Bug Smasher himself, Mark Abent, on the show. And so Mark came in and was as flabbergasted as we were about what the hell is causing this. And so we went into a bit more debugging sessions. At this point, we're deep engine-side physics and collisions to try to figure out what's going on. And then in the end, Mark just goes, in the typical Bug Smasher fashion: what about gravity? And so we take a look at how the ships have physics grids. These grids have sizes and mesh sizes, and the animation is based on the geometry. And there was no gravity in the ship, zero, zero, zero, right? And so basically, it was a combination of the physics grid and the fact that the actor on the server runs at a lesser resolution. So your client when you're in game is at full resolution. The server doesn't run at full resolution when it's simulating your character. And so the combination of this would make the server actually reach the EVA state. So now you're floating in EVA, right? But with no gravity and no other movement, it will never solve itself. And it's also overlapping over the grid, which is too big. That was a bit of a combination of blunders, from geometry to wrong physics grid to no gravity, and it made the bug also not 100%. It wasn't like, exit the ship and it happens; it took about 10 to 15 times, you would get it one time. All right, so we fixed the bug. We get this in there. Internal QA validates it. And then to be able to properly validate it, as you know, we need players. And so that's where we mobilized you guys on PTU to come and try this out. And this was one of the best moments I've had testing in this game, where roughly 100 people showed up with Wolves at Boondoggle on Hurston. And then 200 people are entering and exiting the Wolves to see if they get detached from the animation. It was a fantastic moment. And so, yeah, that was a fun one to troubleshoot.

Well, so obviously we launched server meshing at the end of last year and we've been running a server mesh environment for now seven months, eight months, nine months. We're nine months already. And so that's the static server mesh. And so that means we have territories. Just a quick recap for everybody. That means that we have a configuration for these territories. And we can pick different top-level object containers from the tree of the universe. And then we basically distribute them on different resource servers. So at the moment, we run a 10 game server per shard setup. And so that means that we basically distribute the key areas on these 10 game servers. And that's how, when you transition from one zone to another, it's seamless. But you're moving from one server to the other. So we've been kind of taking the year to get used to being a server mesh environment. And so that means we've discovered a bunch of gameplay bugs related to this. I think we talked about some of those in January that have been solved. For example, the quantum drive systems didn't properly handle the transition. Sometimes we'd lose power in transition, and that would cause the thing to reset, pull you out of quantum, things like that. That's one example. But there's a series of other issues that were related to that. But more importantly, we've been also learning about how to properly distribute the load of the simulation. So now we have this tool, we have this map. It's not the end of it right now. Every patch this year, we've changed the configuration. I don't think we've had a single static configuration since the beginning of the year. Every patch has been different. And so obviously, we've tried to make sure that we dedicate resources to where there is an event. So if it's ASD locations or Hathor locations or Stormageddon locations, we've kind of reconfigured the mesh at each release to basically guarantee some server power at each place, because we know that people congregate in these areas. We've also learned a couple of things, because the teams that make content don't necessarily always talk to the team that makes the tech. Where you put those locations or how many there are is important. We learned that through the Hathor moments, where they were a bit too close and we couldn't distribute them properly because of other limitations, which I'll get into in a bit. And so we've changed that config every time. And so it's been really a good learning experience for the team. We've also solved a bunch of issues in the static mesh related to performance. We talked about stability of game servers earlier, but the hybrid is at the center of this. This is the networking component that you're connected to when you're playing the game. It's doing the replication. It's doing the binding and all that stuff. For this guy, the stability score is zero. Meaning if there's one crash, we don't ship the patch. That's it. That thing needs to run perfectly at all times. And the kind of quality metrics that we expect from the hybrid are very, very intense. The speed at which they transfer packets or accumulate packets and transfer them between areas and elements has to be very, very fast at all times. Or else you get rubber banding, you get desync, things like that. And so there's been a lot of performance improvements behind the scenes. Especially, I think, just before Stormageddon, or is it after? At this point, it's a bit of a blur. But I know that we found a couple of key optimizations that have really helped the hybrid maintain stability across some of the big binds that we were seeing. Again, binding is this process that when your client needs to bind to an area, it means it's receiving updates for an area. This whole binding process is a bit costly on the hybrid, especially when there's big areas being bound. And so optimizations have gone into that. And that has helped desync quite a bit. There's always room there for improvement. And the team are still actively working on this. That's never going to be the end of it. But it's much improved since the release of 4.0. I hope the rubber banding is getting a lot better every patch. We're hoping to always cut some of those big bottlenecks as we go. So that's static server mesh. We also learned a couple of things on how to distribute those locations. So the cost of a location and the amount of players is not necessarily the same weight. Funny enough, a lot of players in the same place, let's say you have an empty room with nothing in it. And I can pack 300 players there. No problem. Server's good. No problem. If I add AI in there, we start getting into issues. If I add ships in there, bigger issues. So the cost of the location, not necessarily multiplied, but factored in by the number of players there, is kind of what the cost of an area is, and how we should distribute the computing power that we pay for to run the game. And so we've been learning a lot about this in all these experiments we've been doing, and that's informing dynamic server meshing, right? Meaning what the algorithm needs to be in order to properly distribute the load of the simulation to the different components in the game that can do the replication.

Well, I'm not going to advance the date, but I can tell you that there's multiple things that compose dynamic server meshing. Obviously, the team is working on those aspects right now. The key aspect that dynamic server meshing will bring is a kind of arbitrary subdivision, meaning that we could assign a server to anything that's a streaming group. So inside this big tree, you've seen the PES discussions where we show how the data is laid out into one big giant scene tree. Any of those nodes that are deemed a streaming group can actually have a server having authority over that entire area. And so DSM will definitely bring us the ability to kind of arbitrarily subdivide, where we're no longer tied to these territories or this territory map. That's the first big thing we're going to gain from dynamic server meshing. And we're working on that part at the moment, which is basically how we can move these streaming groups from one server to the other. How authority is transferred. Who's managing it? What component is taking care of that? So that's under development right now. There's also a lot of, I want to say, parallels or similarities between dynamic server meshing and instancing. And so we've talked about that we're working on instancing. That's also part of this work. We technically already have the mechanics of instancing in the game in the form of the instanced hangars, which basically allow you to have multiple stacked areas on top of each other. But you get to see only one, and only one of them is bound to your client and replicated to your client. Historically, if we go back to the 3.x days when we put this in, initially, the instancing was actually not working from the networking side. And so you would replicate all the hangars that were at that location, even though you couldn't see them, which was even worse. Now you actually get assigned the proper tag on your character. And so when you actually take the elevator, it's bringing you to an actual instance, which is basically yours. So the instance tech that we're building on top of that will basically leverage some of the concepts we're putting into dynamic server meshing, because we want to be able to distribute those instances across the mesh. And so right now, they're location-bound. But we want to not just have them location-bound, but have any of the servers take those off so that we can share the load. Again, optimizing the distribution of the simulation, of the computing workload that we have, across what players are doing. And so that when you enter an instance, we can guarantee a certain level of performance without needing to have a little dude at the back going like, oh, we need to change the territory map for this, right? And so we're actively working on all this stuff right now to bring us to the dynamic part of server meshing. And obviously, with that, even though this is not our focus right now, we are still planning to go to higher player counts, but this is totally not our priority right now. Our priority is really on getting the functionality and the computing workload properly distributed.

Well, so you're right. It's not like server meshing isn't already a thing, right? In the past, even Ultima Online had server transitions. Now, they were not necessarily seamless, but you still had server transitions and gridding, which is a typical approach, which is you take the world, you split it into a grid, and then you have servers between them. And then the rest of the different strategies come from how the servers communicate to each other, and how the state is distributed, and how you transition from one to the other. And I think that the areas where our approach differs is that we've extracted the simulation from the game and the replication of the game. So replication is how game server and clients stay in sync. And so when you're in an area and somebody opens the door, the door opens on your client. That's because it's been opened on the server, and it's also because somebody else had authority to send that message to open the door. So this is all replication. Well, normally, you have one game server, and it does both replication and simulation. So we've taken the Star Engine. We've extracted the networking from it, so the replication, into the hybrid. Now, we call it the hybrid because it's the first stage of server meshing, right? But the hybrid has all the networking components. It has the state in its memory. It holds all of the information for the shard. And it does the replication. So you're connected as a client to the hybrid. So you're not connected to the different servers you're actually on. The game server that's simulating your area, you're not even on it. You're not even in active connection to that server. You're actually actively connected to the hybrid. And the game servers are actually clients with authority, if you want, if we simplify. And so that's one area where it differs. The second one is how we separate space. And you alluded to it, which is that our territory mappings, our mappings of how the authority is managed, are in 3D space. So it's not a grid. It's fully made in 3D space. It's 3D space and a tree, which is the other bit. And so what that means is, I'm going to try to make that simple. Let's say I'm on the yellow server, and suddenly there's not enough players on the yellow server, and the location is a bit low cost, then we might want to despawn that server. Well, we can transfer the authority from everything the yellow server is doing to its parent. There is a parent. Whereas when you're doing gridding, that's a bit of a different model. In the past, I've seen solutions for meshing which include adjacent servers talking to each other, for example, things like that. The client actually disconnecting from one server, connecting to the next, which in all of these cases were things that created side effects that we didn't want. So in the past, we did try these things. If I go way back to when we started the server meshing project, that was the first approach, a bit more classical. But we realized, to properly offer what we've shown in, for example, the meshing demo Paul did a couple of years ago, or what you're actually playing now on live, to be able to have bullets completely traverse the transition and hit on the other side of it, we needed to have a solution that was native to how we were doing it. The bet behind it was that the hybrid was going to be fast enough. And so far, it seems like it's working out, though it is a lot of work to keep it at that speed with all the needs that we have. So that's, I would say, the differences in the approach. And every game is going to have its own approach, right, tailor-made to its gameplay. So in our case, we're trying to make the biggest universe ever. And so we need the most scalable way to approach it. Also, we've taken the most difficult route, I feel.

No, I think that's pretty much the state of it. We're advancing on the dev for it. Obviously, we don't want to bring destabilizing changes to the game. And so we're making sure that these are going to be tested first. We will go to a tech preview when we feel it's the right time. Right now, it's not the right time. And so we're still working on these aspects. But that's our objective. We're not going to do a big 'oh, dynamic server meshing is up', right? That's not going to be how we're going to roll this out. We will most likely be bringing elements of meshing into play, right? Like the ability to despawn servers in areas that are not being simulated, the ability to scale up the mesh dynamically from the territory map, and then later on possibly the instancing support that goes through it, and then after that, the final jewel of dynamic re-subdivision and reallocation.

Let me pull up my notes. Okay. So freight elevators is one of our hero squads, right? Now, to your previous interventions, that came way too late in our planning, right? And so that's one thing that I think, if we could go back in time, that squad would have started way, way earlier.

I'm sorry, to be clear, work had started earlier. We hadn't had the hero squad on it.

Correct. Exactly. Good clarification. And so let me go through some of the stuff that we've been going through on freight elevators in general. So to start with, obstruction issues. A lot of the issues that we ran into were related to how the elevator actually detects obstructions and/or contents. So, for example, blocking the doorway, items being positioned on the roofs of the planetary freight elevators. Now, a lot of this, by the way, I'm going to talk more about the planetary versions of it, which are 10 times worse for some reason. But in terms of obstructions, you could leave items on top of the elevator, and then it was detected as an obstruction. And so these obstructions, first of all, the checks were super rough. And so they would go like, there's something, I'm not going to move, which is a bit like, come on. Somebody would put a small bottle on top of the freight elevator...

And the whole damn thing would stop working.

Exactly. That would break the whole elevator until that was removed.

MedPen. It was a MedPen. It's so small.

Yeah, but see, that's even worse, because there's a problem with how the items are set up, right? And so when we create a prop or we create an item, there's a bunch of things that go into it. Physics proxy, weapon proxy, all kinds of things that go on top of it. One of them is the bounds of the item. There's multiple bounds, in case you thought it was just a sphere or a shape or a rectangle. There's more than one bounds. Long story short, a lot of items have incorrect bounds. And so what that does is that when you leave them on the freight elevator, they might clip into the elevator itself. And so couple that with the bad obstruction checks, that could cause some really crazy exploits. Like, for example, leaving, I think, a MedPen on the freight elevator. It would clip through the platform. You couldn't even see it. So you'd be like, there's nothing on it. And then it would still block it. I think that actually is not completely fixed. So the squad still has to get to all of these items. And so for the bounds issue, what we're doing is we're adding a lot more checks on how the things are submitted, again, beforehand. So we're revisiting all bounds for items, right, and how their setup is made, to make sure that none of them can be in that situation. So we're kind of doing a system-wide check on this, in order to make sure that when we get into these situations, it's going to work the way we think it should. This is all super interrelated. Items that would get destroyed but would no longer have geometry, but would still be considered obstructions. And so that was a very egregious one, I think, that we saw very early, which is there was a way for you to destroy an item. Its geometry would disappear, but it would leave the entity present. That's it. That elevator is gone. You need tools to see that.

At least with the MedPen or something, people could zoom in and try to scan the pad, try to find it and remove it. This next one, though, the geo's gone.

Yeah, that's it. This one was the worst one, right? And so the thing too with the freight elevator bugs here, some of them were the setup of the actual freight elevator geometry, right? Actually, that was a regression. So it was a bug that got fixed and was reintroduced. Which is the shape, because we have multiple types of freight elevators, right? We have the horizontal one, we've got ones that are vertical, some with doors, some without. And so some of those were badly set up; the planetary ones were especially egregious in that case. And so there's still some bounds checks to be added. But when you got all these messages about obstructions, that was all related to this. One of the other issues we had with freight elevators was actually one of the first ones we had to fix, which is that the freight elevator is set up into two things. There's the kiosk and then there's the platform. And so the kiosk hosts the manager component. It's the one that's managing the stuff. And the platform is kind of slave to the manager. So there was an issue where, depending on how an area would stream in or stream out, the platform or kiosk would stream in first or stream out last. And this caused these two components to now be disconnected. So they would not find each other anymore. And so in those cases, that entire location was dead. That location, game over. Because they would never find each other. There was no model in place or process in place to go and try to fix that. And so this is part of what we do on the hero squads. We've done the same with the transit system, which is we introduce a self-healing approach. And so in this case, the system would actually retry to connect. So if a manager doesn't have its connection to its platform, then it would get retried over time until it actually connects. And so this way, even in the case where we do run into an issue where they disconnect, they could still reconnect. And so that solved the issue. That was actually rolled out in a hotfix.

Let's continue on our list of bugs with freight elevators. The UI and platforms could get out of sync. And so this is where the platform will have content. So there's clearly a box on the thing. And the UI is showing that it's empty. And so that basically makes the whole thing inoperable, because now you can't ship your box at all. In this case, I think that the streaming reconnection actually solves this by self-healing. But also the UI will re-query multiple times to be able to make sure it's always updated. This is a side effect of the game teams trying to cache a bit more data than they should. And so now it cached an empty elevator. So that's a weird state. So a bit of background on how the platform actually works. So you've got your inventory system, your warehouse. That's where your things are stored. So you can query for items in it. You get the list of what's in it. You can unstow items from it, which means bring them into the game world and physicalize them. Or you can stow them into it. The freight elevator and the ship cargo platforms are actually using an abstraction on top of that called the inventory container. And so the inventory container can be open or closed. What that means is that when the inventory container of the platform is closed, all of the items that are physically on it are actually stowed. When it's opened, it means that all of the items that were in this inventory container are in the game world and being acted upon. So when you unstow things on your platform, that's what happens. There was a weird case where, if there was a delay in the back end... so the back-end systems for all these operations have queues of operations that run, and so they can handle the delay. But in this specific case, there was a weird state where what's actually unstowed and the state of the container would be desynchronized. And so we've added another repair flow for this that would actually detect the state of it and actually resynchronize it in the cases where that would happen. This was because we didn't properly handle failures. And so in the case where an unstow would fail, the state would now be in a weird situation. And so now we do the proper error handling, which is another thing that we've changed in our processes for development: error handling is now part of code review. So we really need to make sure every single... I mean, that sounds basic, right? But I'm just saying the way it is, right? Some cases were not handled. And so you need to handle those failures. And same thing for door states. So whether the door is open or closed after a stream in and out or a server crash would sometimes clash. And so the kiosk can think the door is closed, but it actually is open. And so we do the same now. We can actually properly set the flow back. So one of the lessons we've learned with freight elevators in general, and all the issues I just went through and the reason why we ran into them, is we need way more multiplayer testing. I think that the major key behind it is that the teams have spent time on the in-hangar elevator, right? So we ran into some issues with the in-hangar elevator, for sure, and there's still issues there. The big difference with the Resource Drive event was that we were driving players to the freight elevators, and there's more than one player interacting with it. And so that's one area where we have improved our testing and our testing approaches so that they're properly tested multiplayer, and not just when we reach PTU or in focus tests. Now it's part of the general testing for these open areas.

Well, so self-healing is basically when we add functions to a system to detect that it's in a bad state and attempt to fix it. And so you could call it a kludge in some areas. Sometimes it's not the right thing. But I think that in this case, for a lot of the systems that we have where we are working on a replacement, for example, and we're trying to keep this one running, self-healing will come in and basically rectify the situations. The way our game is made, with our streaming system, makes some trivial things in a normal game very complex. And sometimes we run into these issues where the game teams can't really identify what's specifically causing an issue. They just know it happens. In the past, we used to wait for the key engineers that can actually decipher these conditions to come in. But these guys are obviously, everybody's swamped with work. And so sometimes the delay was so long that you guys as players would have to deal with the condition for long periods of time. And so self-healing is basically an approach where we're telling the game developers: while you're investigating the proper root cause for this, provide a self-healing alternative, which keeps the system running properly. It's self-repair, basically.

Well, so one thing, I think we touched on this a bit in January, right? One of the things that we put into 4.0 was the beginning of some physics cleanup, right? Now, since the beginning of the year, we've unified a lot of the cleanup systems, right? So namely, we had an abandoned vehicle tracking system, we have the physics intersection cleanup system, we have the actual density manager. And so now all of this has been unified into one platform. And so what we're able to do with this now is we have density classes. So depending on the type of item, we can define the density class, which basically allows us to say no more than X in a volume of Y for this kind of item. And obviously, there's way more parameters. I'm just trying to keep it simple so we can understand what it does. We've recently added the ability for designers to basically vary these density classes by region. And so now they can say, in this location, make this super aggressive, make this less aggressive. And so the density manager part of this is really about items in the game world that are dynamic that we want to make sure we keep at a manageable level. And so if you start putting water bottles in one place, at some point, it'll start cleaning up because they're in the same volume, and the density class of the water bottle will inform how many we want to keep and how we want to do that. There's also lifetime cleanup. So things we want to live, but we want them to disappear after a certain amount of time. So we've added that functionality in. So that's a lot better. And then vehicles come into play. So abandoned vehicles. We've been tuning this since the beginning of the year behind the scenes, this and the physics cleanup. So abandoned vehicles is basically when you leave a vehicle completely abandoned. So no passengers, nobody's left on it. It's left on a planet. You leave it there. Already, it's put on a timer. And so at some point, we'll have to clean this up. If it gets interacted with, if passengers come into play on it, whatever, that revives it, resets the timer. I think, if I remember right, that's set to something like two hours right now for abandoned vehicles, regardless of class for now. So that is something we want to add, which is basically change the density class for larger ships so that maybe they go faster. This is the kind of thing we need to keep tuning. And so the abandoned vehicles will actually clean up those areas. Now with the new system we have coming in, we'll be able to override these values per region. And so for planetside locations where we know there's going to be traffic, we can make that a lot more aggressive. And the last part of the cleanup system is the pink slip. I call it the pink slip, which is, hey, my ship turned pink, and bam, it's gone. Which is basically areas of the game where your ship is entering into a deep intersection with something else. And this is true of all areas. So if something enters deep intersection for a number of frames, based on the size of the entities and the size of the area, we will clean it up. We're going to first turn it pink, full pink, for a certain amount of time, and then, oof, it disappears. The reason for that is because the servers are still very much physics-bound. And currently, I'm not saying it's impossible, but currently, there is no elegant way to break those collisions automatically and cleanly. It typically creates a maelstrom of physics problems when you try to fix it. And now you've backed it out into another physics element that is now colliding. And so you end up with these massive unstable systems. And so the pink slip is there for the moment. We're trying to improve this. We've been tuning these values quite a bit since the beginning of the year. Every patch, we adjust the density classes, the abandoned vehicle timers, and the physics intersection costs a lot. I feel like we're seeing a lot fewer reports now. It's getting a lot better. I think we're getting into the sweet spot, but it's unfortunately not a magic 'just turn on density tracking', right? We already have density tracking. It's really about tuning the numbers so that it feels good, right? And that's the hardest part of this.

This is one of the ones that completely infuriates me because it touches on many things. So spawning in the game is all over, right? There's multiple different types of spawners. Some of them are item port spawners, for example. I hate using the bartender as an example. Apologies. But the bartender picks up a glass from an item port that's on the counter, and then we are at some point going to respawn a glass on it so we can do the animation again, or things like that. Same thing for AA turrets. In 4.0, we've had many, many issues related to overspawning from those spawners, and we fixed a lot of them, mainly conditions where designers would set those up as infinite spawn and no limits, which is great, right? So that's one thing. Now, since we've been putting out content this year, the different locations all have a population of AI NPCs in the place. And so we've had overspawn, I think, with every content patch this year, where every patch there's at least one report that comes up. It's like, infinite NPCs at location. Is it meant to be this hard, right? And I was like, all right, what's the cause now? This is an example of: the symptom is the same, the cause has been different every time. This is something I really want to get time to solve properly. The replacement system is in the works, but it's not close to being there. So the way this works, I'll give you the most recent one, for example. I think it was in 4.3.0. The mission module is the system that actually activates the NPC population for a mission in a given area. So when you enter the action area of a specific part of the mission, then the mission code is going to go and initialize submodules. And then one of those submodules will say, hey, put some AI in there. And then it's going to activate the spawn closets in the area. And that's going to start. The issue is, there was a bug with the way the mission flow was working for these things that would cause these modules to doubly initialize in certain cases. And when that started, then it would actually do it again and again and again. And so that's the latest source of these overspawns. The proper fix for this is the population manager. This is something that our R&D team have been working on, which is a replacement... spawn closets are still going to be around, but the systems that drive the spawn closets to make these AIs appear and spawn on top of, what's the word they use for this, the base population in an area, like, let's say, walk around New Babbage, all the social NPCs. Currently, all these social NPCs are placed there, right? And some of them, like, again, the bartenders, are driven by mission modules or area modules. And those always get this overspawn. So we just need this replacement system at this point, because we keep repeating this error from different angles, right? And so the population management system, its approach is different. It's going to know what's in the game world. It's like, I need five guys. Do I have five guys? We're good, right? Instead of going, when I start, I spawn five guys. Which is a complete problem. And so yeah, overspawning will be solved by this. In the meantime, we're putting in more verifications for the cases where you stream out and you stream in and the module resets, or you abandon a mission or start a mission while you're already at the action area, so that the actual logic doesn't run twice and that there's actual limits in those areas.

So, obviously, that's a bit of a whack-a-mole thing until you get the new system online.

Yeah. One thing that I'm hopeful about, though, for this is that the teams have actually built some good additional debugging tools that allow us to view the state of this. It sounds a bit weird, but it was difficult to even know how many NPCs are there at this location. That was actually difficult information to find. So now we've got super good tools to be able to tell that. If I'm able to just open the tool and view that information, it means that game designers can do the same. And so that means they can debug those problems in advance without us having to do it post-launch, which is the big thing.

Little story behind this. So if we go back to the 3.x days, the game server has everything. It's simulating everything. All the entities are on one server. And so there's, I don't want to say shortcuts, because this was just the way you would build things at the time, where, for example, doing interdiction, the server has all the entities. So it's able to do cross-zone checks. It's able to raycast across zones. It's able to do all that stuff. Now with server meshing, you could be on a quantum path that brings you across game servers, right? And if you've ever looked at, some of our backers have made outstanding tools to calculate interdiction positions and stuff like that, if you look at this, you'll see that the edges and the ends of this are where it actually matters that you get position. That means that since we currently use the territory map around planets and moons, typically that will interrupt the quantum jump at the interdiction point that is the most effective or close to the most effective. And so what that needed is a solution that allows us to have a view of the interdiction system that's basically cross-server for the game shard. The replication layer has that information. We do have it in the hybrid. But we are very reluctant at allowing what I'll call game code here, I'm not trying to be pejorative, to query the replication layer, for multiple reasons. We want to keep them isolated. For this, we needed to build a solution. So the online team have actually built a new service that serves every shard, but is able to provide our game devs with basically a way to build a small representation, a small data representation of the game world for things that matter. So if you can picture this, you're in your ship, you start your quantum drive. What actually happens is now the game server is going to register, in this new service called the Entity Cluster Service, some piece of data that says, we have a quantum drive at this location. And it's going to keep that updated. And on the other hand, I've got someone who starts a quantum interdiction device. That will also connect to this and then inform this kind of matrix view of the quantum interdiction network. That doesn't matter which server you're on. This is all kept updated in semi-real time as you're quantuming and moving and as the interdiction devices are activated or deactivated. And so what that allows the game server to do now is, it's receiving these updates. It's got this, let's call it a symbolic view of the game world. And so now it's able to do proper raycasting math to see if the interdiction is going to hit cross-server. It doesn't matter if you're on the same server or not. You can actually trigger the interdiction across. And then that basically gives us this ability to have cross-server coordination for things without having to push all this to the replication layer or query it from it. So this is mostly a gameplay system, but we are using it for beacons. So we're going to be using it even more for the full beacon implementation. But it's going to be the same kind of thing, where now you could be in New Babbage, and someone has a beacon open in Pyro, and if the gameplay wants to allow it, you might be able to get that beacon or see that beacon or bind to it, right?

Well, so out of all of this, all the stuff you mentioned now, I know that our intent is to reactivate medical beacons, absolutely. I think that's a core part of the game we're trying to do, and so we really want those reactivated. The Entity Cluster Service I just told you about is what can enable us to do cross-star-system beacons. I don't think we're there yet, but we definitely want to get there, where we could actually pick those up. So right now, all of the technology pieces that you need to fix all the issues you mentioned about beacons are there. We have the ability to have controlled notifications, localized detection of beacons, cross-server, cross-star-system notifications, whatever. It's really about, can we put a team on it to basically add all these things, but at least all of the hurdles are kind of solved with this. The interdiction work we did and this new service, the cluster service, is the thing that we need to be able to do that, to reactivate all that. So that's already live.

Transit. So, well, we do have a hero squad on this, right? And so for everybody's sake, just to make sure we're all on the same page, the transit system is the system that drives elevators, not freight elevators, elevators, trains, and some area-based transit for these things. So the transit system has been the bane of our existence. And so at the beginning of the year, we put some significant resources in that hero squad and brought in some very, very experienced developers to the team to help push these fixes along, on top of what I discussed earlier about the methodology that changed for these issues and the introduction of self-healing techniques to try to get it moving. We are accepting the current state of transit. I feel that right now, we no longer lose carriages, or when we do, it gets auto-fixed. The destinations mostly work. Double doors rarely happen. The system mostly responds well. There's still issues. I'm still seeing in my playthroughs the train timers with insane values, which is a bit of a drag. Or the transit still has this unfortunate propensity, in the event of a server crash while you are in movement on the train, that your client will keep interpolating that. Which might get you into space, right?

We still get the occasional missing elevator. There's still a few trains that trap people.

Yes, that can still happen, right? But right now, the state at which this system is, the approach for that system is, we're kind of accepting its state, right? We're still tracking the big egregious issues that we have, like the ones that we just mentioned. Those are still being tracked and worked on. But overall, from a leadership standpoint, we're kind of in a space where now we want to talk about transport.

Correct. So transport is a complete rewrite of transit, which has a much different approach to how it manages peripherals, things you click on to call up things, gateways, which are the areas at which the carriages stop, and carriages, of course. And so the transport system is a lot more resilient in its approach to streaming problems, which has been a thing in the past for transit. Also, the way it's developed is a lot more robust. And so the state of transport, let's talk about that. So the approach that we've taken for it is, the system is kind of built at this point. And we have a single transport link that we use internally. So not yet on a player-facing build, but we have an internal build of this where the New Babbage habs are completely driven by the transport system. Which is a pretty big milestone for the team, because it means the system is basically kind of fully featured, in terms of it's able to drive one of the major hubs. We are currently working at instancing those transit links. So just to reiterate what I said a little bit back in January, the introduction of instanced hangars, which means dynamic gateways in the transit system, is really what made it a big issue. And so now the team are working at adding that functionality into transport to make sure that we properly support dynamic gateways. Which will allow you to have a transport link to and out of an instance. Now, this could be an instanced hangar. But in the future, it might be an area or location that has some stuff in it. And so that's the state of transport right now: we're working on the instanced transit link and trains. So we've done elevators. And now we're putting trains in with the full feature, and the trains have an additional peripheral, which is the timer, the added timer on top of it.

You've already got the first version. It's already working internally in New Babbage. We're adapting it for instancing. And there's some high hopes that we'll be able to get that in at some point in 2026.

So we are absolutely tech previewing that system, where our goal is going to be to basically, we want to try to do one full location with it, so that that entire location, elevators, trains, all that stuff, is completely transport-driven. And then tech preview that, and make sure that the same methodology and QA checks that we've added for transit are going to have to fully pass before we actually accept to bring it in.

Oh, man. Well, that's one of the casualties of this, right? I hate ATC. The functionality is really cool, but it's one of those systems, a bit like transit. So on ATC, we haven't done a lot of work on it, mostly focused on metrics. ATC is a system that you call when you want landing or when you want to leave. It is unfortunately responsible for hangar queues and is also responsible to assign you a landing area and coordinate your instanced hangar. So basically create the instance manager for your hangar that creates the hangar, and then manage the pad in your hangar. And so for ATC, we're completely not happy with its stability overall or the queue times, which is the one thing that we're actually looking at at the moment, to make sure that the system works the way we want it. That's why we've put all these metrics in about: at this location, the time to exit at this location is this. Do we have enough gateways? Does the system perform the way it should? So we're still in an info-gathering phase here to have the proper plan. But we do know one thing, which is that we are going to change the way the interaction between ATC and hangars actually works. Currently, ATC manages everything. Like I said, it gives you your landing area. It says, hey, I need an instance at this gateway. It spawns your hangar, instance created, all that stuff. Adds that instance tag to your player, then the doors open. Now you can get into your hangar, but now ATC sees the pad in your hangar as its landing area. It's managing this, right? And so it's taking care of all that stuff, which is a bit problematic because it's an old system. We know of its flaws. And so we do want to split these responsibilities so that there's a hangar manager. So that the moment that you're done going through those doors, that's it, done. ATC is out of the picture. Now you're completely in your hangar, and the hangar manager takes care of this. And so we've been doing tech design on this, but unfortunately, because of the different focuses we have this year, this is one casualty that I was not able to push forward on.

Yeah, there's some of these stations that have like 10, 15, 20 doors, but you'd swear the ATC only ever uses one or two of them.

Yeah, that's the one thing that we want to solve, even with the old system, before we even start hangar management.

Cool. LTP, right, is the famous system we're talking about. Well, let's go a bit into what LTP actually does, right? Because there's some misconception around it and also a bit of misconception about the amount of effort we actually put on it. So historically, if we go back to the 3.x era, we developed the first LTP system, right? It came a bit as a side request, which was, we patched four times a year. It was a bit like, oh, there's some stuff players want to be able to persist cross-patch. We're still in development, so it's OK. We're wiping all the time. And we want to wipe because blah, blah, blah. There's other considerations. So let's build something. So we built a very small system initially to persist stuff. And the way that worked is that the game server would basically submit these commands to say, this guy picked up a gun. This guy dropped a gun. This guy has this. And so that was a solution that kind of worked at the scale of a single-server kind of system. And we had a lot fewer items at the time. That solution was in no way scalable, right? But it kind of did work better overall than the current one, to be honest. Now come PES in 3.18, come 4.0, server meshing, now we persist everything else, right? So we need to change how we deal with long-term persistence. And so let me describe a bit how this works currently. So when you acquire a ship from, let's say, the web platform, that ownership record lives on the web platform. It's in there, right? And every time you log into the game, we check, oh, this guy has this thing. Does he have that ship in the game, in his entitlements, right?

It's called an entitlement. That's an important term.

That's your entitlements, right? So we make sure you have an entitlement record. When you actually open up your ASOP terminal and you view your list of ships, what you're actually viewing is your list of entitlements. These are not your ships. These are actually your entitlement records. Attached to your entitlement records is the insurance document that goes with it, which describes the loadout of the ship. We call this an entity class loadout, which describes what is in each port. And if there's no entity class loadout or insurance document with it, then it's the default loadout. So that is your entitlement records. That's what you see in ASOP. Then we also query for: is there an entity for this entitlement record at the moment? And that will look at the game database in this current moment. Let's call it the inventory database for now. So we cross-link the two. So for each row in your ASOP terminal, you'll see your entitlement record. The state of the entitlement record will allow you to see if you need to deliver it first. That means I have no entity in the game. So when you deliver it, now we create the entity in the game. And then if it's stowed in an inventory, then you can retrieve it. It starts from the entitlement record. And then if the ship is stowed, then you'll see its location is set. If it's left in space at a location, you'll see its location is set. If it's lost or destroyed, I think there's a couple of different states, but one of them is destroyed, then you can claim the ship. And then that will use the insurance document to destroy whatever matches this entitlement. So we have a way to find entities that were created from a given entitlement. It gets destroyed, and then we use the insurance document to give you a brand new ship at the location where you're at. That's in-game. That's the way the in-game part works. So let's do a clear example. I'm in New Babbage. I have a Hawk in my list, and I deliver it. It gets created at that location. I retrieve it. It gets unstowed onto the pad. I use it around, blah, blah, blah. I fly out to another place. I lose the ship. I respawn in Klescher if I couldn't do my time. Then I go out to Grim HEX or whatever, I respawn there. And now I'm in a different location. I see the Hawk in my entitlement list, but the ship is nowhere to be found. I claim it. A new ship is created. It's put in there.

The stuff that you pay actual money for, or that you get awarded through like a subscription or a gift, those sit on the web platform. Every time you log into the game, the game goes, hey, what am I entitled to? The platform says, these are the things you're entitled to, and it gives you them back. It's why you are never at any risk of losing the things that you pledge for. That stuff is, for the most part, very stable. You can always rely on getting that stuff back. That's not how the in-game stuff works.

Correct. So we went through the web platform stuff and how ASOP works, right? Let's go into, now I go to New Deal and I buy a ship with aUEC, right? That will actually give you an entitlement, right? Because you can claim those ships, right? You can transfer those ships. So up to now, these will work identically. But when you log into the game, we don't reconcile your list of purchased in-game ships from an outside source. They are just entitlements in your account. And that's it. We keep them. OK, let's stay on ships for now before we get to the items. Now, the thing that's important to know here is that the actual ship you're playing with, the amount of possible states for that ship are almost infinite. You could leave the ship in space. You could leave it in your friend's hangar. For some reason, people are able to do that. You can stow it in inventory. It can be inside an Idris that's stowed in someone else's thing. All kinds of stuff can happen. And so there's multiple paths to update that specific state of that ship. Whereas your entitlement, once you have it, you have it. That's why your ASOP terminal always has it. Now, let's talk about how LTP works. Now, this is within a game patch. When we release a new patch, inventory database, entitlement database, they're gone, right? We start from scratch, and now we're rebuilt. So you reconnect, you come in, your web purchases are being reconciled, we create new entitlements for them, and then when you get in-game, you create new ships, and then you play with those ships and everything works. What is LTP? LTP is when you acquire things in-game and you have ships that you own, right? The way LTP was designed, and maybe this is the part where there's a mistake, right? But still, I'll explain how it works. When you stow something in an inventory that you own, the systems will create what we call LTP records. These LTP records are close to what an entitlement is, right? But they say, Jared owns a Hawk, right? And those are stored in another database that is outside the game environment and lives cross-patch. And so now when we wipe the entitlement database and we wipe the game database and we start fresh from a new patch, when we reconcile your account the first time, we're going to give you your web purchases for sure, and we're going to give you back your LTP records. And so that will create those entitlements. That's the intent. Now the problem is, right, we need to make sure that we always stow the stuff back into inventories that we own properly and that those records actually save properly. And it turns out that there's a million goddamn ways in which stuff gets stowed in this game. For example, claiming a ship by insurance, then you get the ship. It gets stowed from another path. And in that path, the LTP wasn't correct. We've had cases where we tried to fix bugs related to the F7C turrets, for example, that got disconnected, that were a default loadout item that we didn't want to wipe when you claim insurance on it. And that messed up the LTP records for it. And so this whole thing basically is very fragile. And so what we realize now is that the list of LTP records and the way we update them is not always a slam dunk. In fact, there are a bunch of cases where the record's not updated right. We even had a case in 4.2.0 very recently where we realized two days after launching the patch that, the way the recording was going for the ships, there was an issue where, instead of giving you, let's say, I'm going to take your Wikelo Polaris as my example just to make it super egregious, your Wikelo Polaris that you stowed in your inventory by going to your hangar and storing it back in would actually persist as one of its components in LTP. So you would get a cooler instead of the ship. And so that was obviously a very big issue. And we ran corrective programs to solve that. And so that's why, if you notice, in 4.2.1, you didn't have many reports of people losing ships. But now we're back at 4.3.0 and we've had reports of lost ships between patches. And so we're pretty certain at this point that there's still cases where LTP records are incorrect, through certain game flows. There's also, unfortunately, a lot of signal-to-noise ratio issues, like people claiming they've lost ships. We investigate their accounts. They never had those ships. We get a lot of that, right? And so at this point, we have to be very prudent.

So you have the inventory system, where stuff goes and gets stored. And the records for these, let's say you have a ship that you store. You're at your hangar. There's a ship on the pad. That ship is unstowed. It actually lives in the game shard database and is actively being simulated on, meaning we're getting updates from the game server on the ship, its position, its location, the state of doors, thrusters, all that stuff. It's multiple messages per second on these things. So it's unstowed. Now you press the store button on your ASOP, right? Platform goes down, doors close. Now the system will stow that ship. What that means is it deletes the ship from the shard, removes it from the game world, and then moves it to the inventory system. So it gets stored into your location inventory, right? It's stored in there. At that point, if you own the inventory where the ship is getting stored, we create an LTP record for it, meaning that if we wipe the patch right at that second and you come back in the new patch, we will give you back that ship for sure, right? Through the hangar, 100%. But there's other areas where we create ships when they're already stored. There are areas where we do repair and we modify stuff like that. Those are the areas where now the LTP record might not get created, or we might interpret it as a separate thing. The other thing is, when we stow things into an inventory that you own, it's not just a ship. This is the bit that sometimes people oversimplify. It's the tree of the ship. So if the ship has a box container that has 32 SCU of boxes that have handguns in them, I'm going to give you the ship plus the handguns, anything in that tree that is flagged for long-term persistence. So this is where designers look at items in the game and they say, this we want to store cross-patch. There are things we do not want to store cross-patch: ammo, consumables, stuff like this. That's a design decision that we've taken in the past. And so there's this whitelist that comes into play that decides what items we'll create LTP records for and when those get processed. And so the process of taking something that you're storing and then creating LTP records from it is really about processing the whole tree. So when you store an Idris, we might be creating hundreds of LTP records for stuff that's in it, right? And so that gets to be an issue. You can start thinking about the edge cases now, right? You have a Constellation... I don't even know if it fits, actually. Let me take a smaller ship. You have a Gladius in the bay of an Idris, and you store the Idris in your inventory. Technically, you might get the LTP record for the Gladius in this case. Now, it depends on ownership and how that's actually trickling. And so these are all the cases we have to deal with. And so when we told players in the past, oh, you need to store your stuff, what does that mean? I'll give you a concrete example of where there would be a lost ship based on these parameters. Let's say there's no defect in the system, which we're not completely sure of yet. I'm pretty certain there is a defect, but let's start with 'there's no defect'. You play the game with your Gladius, and you leave it in space somewhere. And now you respawn at your station, and your ASOP says Gladius destroyed, or whatever. And now you have the claim button. And you're like, oh, cool, I'll do that later. And you leave the game. And then we patch. You don't have that ship. Because when you unstowed the ship to the game shard to play with it, we removed the LTP record for it. That's the way the system is designed. We're going to change that, because there's too many paths in which stuff gets stowed that could lead to problems or that don't respect the rules properly. And that's what we've come to realize. Without going into too much detail about how we do this, this is part of our item imprint epic, so a very large approach there. Because we always said item recovery tier 0 is just tier 0, so there's more item imprint work there. But part of this is a revamp of how we deal with long-term persistence. And we're going to take a very, I'm going to say, simple approach to this. We're going to persist your entitlements across patches. And so instead of trying to always reconcile all the data from the game state, which, I just gave a couple of examples, but I could go on, the turducken example of stuff in stuff in stuff, all these things can cause either the records to go to the wrong place or the wrong things being persisted. So instead of that, we're going to persist your entitlements. So that means that if you see a ship in your ASOP terminal, you're going to come back in the next patch, and you're guaranteed to have the same list. There's no ifs, ands or buts, because we're keeping that database cross-patch completely. And now, the other question people have, I've seen some comments. People are like, why don't you just persist the whole thing across patches, right? We cannot do that, for reasons related to the data format. So let's say between 4.2 and 4.3, the Gladius you play with is not the same Gladius between patches. It's completely changed. Its shape has changed. Its entity hierarchy has changed. The parameters on them have changed. The components that drive it have changed. And so we don't yet have the tools or the proper approach to version every single part of the game properly so that we could forward-migrate the data, right? So that the Gladius that you've stored in 4.2, its whole tree, its parameters, its snapshots, its variables, all that, you come back in 4.3 and we migrate it to the next one. That's something we want to do. But we're completely not there yet. And so we need this LTP layer. So your entitlement records actually serve that purpose right now. Because when you claim a ship from insurance, we will basically do the same exact thing, which is we persist the fact that you own this ship, and we allow you to obtain a recent version of that ship on the current patch. It'll be exactly the same mechanic.

Well, we're actively working on this right now. So big parts of the online team are dedicated to this. There's a few things that we need to solve in order to put this into full production, I mean, release it. Obviously, since those are back-end systems, they have to be super robust and very well tested, because you know what happens when they're not. But that's the approach. In the meantime, though, we're discussing mitigating options. So I'm not going to promise those here. But we did do a mitigation in 4.2.1, actually. It kind of worked for ships, specifically ships, not all items. And so we might actually use that again to ease up on some of the pressure before all the systems come in. Because the reality is, your entitlement is what you're actually driving from the ASOP terminal. And so we can just ensure that you have all those entitlements again. And so that's the mitigation. And so we're probably going to be putting that into play, just so that in the meantime, while we're developing the feature, there's not as much ship loss and things like that.

Very good, actually. We're coming up on the two-year anniversary of the merger of Turbulent and CIG. It's going really well, I think, overall. So we did want to do some adjustments, though, because the Turbulent teams, even though they're fully working on Star Citizen, were still under the Turbulent name, which caused us a bunch of grief, to be honest, in having to explain, like when we're hiring people and bringing them into the team, it's like, so what is there? Oh, it's Star Citizen, but it's Turbulent. But really, we're the same. That doesn't help. So we all wanted to make sure that this was one good unified message. And so for the game teams over there, basically, we kind of split the two businesses, because Turbulent still exists as a web company. And the web teams are fully inside CIG now. And so changing the name of the studio to Cloud Imperium Games Montreal made sense, to be able to leverage the outstanding talent in the city and build stable teams over there. And the team has grown quite a bit. And so this was really about clarifying stuff like this. And I think it's going well. The integration has been fantastic so far. The way the teams operate across the board at CIG is improving. I think that even though there's transatlantic time zones, you can see a lot of the fruit of the collaborations between the teams show up right in the stuff that we've produced this year. The teams travel back and forth. The key leads are exchanging. And there's a really good gel between this. And so it's been, to be honest, really positive so far.