Benoit Beausejour · Chief Technology Officer

Well, it translates into aggressively monitoring. What we call stability internally covers some of the very simple metrics, like do servers crash, or do hybrids crash, do clients crash, and together, do these cause client disconnects? And so all these metrics go into what we call basically stability, but it goes further than that. Those are just basically the indicators, like the big level indicators. What stability is, is bare-bones things, like are you able to successfully connect to the game?

Jared Huckaby · Host, Community

Can you log in?

Benoit Beausejour · Chief Technology Officer

Correct. Can you do successful interactions? Like, can you go to your ship and press F to join your cockpit and then it takes 35 seconds? Is it consistent and reliable? Meaning, is it like this all the time? Or is it there only sometimes, right? So stability is about consistent and reliable experience interactions. It's also, when we work on stability, we're talking about improving the observability. So when you guys are playing the game, you have a very immersive experience, and issues that affect you during your gameplay are part of your bubble, right? We need to be able to see those issues from the outside. So from the server side, from the cloud environment side. And so improving observability gives us that, because there are many systems in the game that have not been instrumented to be observed, right? And so sometimes some older systems, I'm not going to name ATC, are going to have some hiccups that we can't even see, but players report to us and then we have to investigate, but we need more observability. So that's working on stability as part of that. Changing systems that typically are problematic into self-healing systems so that they can just repair themselves. So when they get into a case that we know, we just modify the system to make sure it tends towards a state that's functional. And the observability comes into play to make sure that it does function. Working on stability also means establishing budgets. So budgets for errors, budgets for performance. And so that affects everybody in the company. But if we talk purely on stability, it's all about, is the content that we've produced here or the feature we've produced here using too many resources, whether it's CPU, GPU, server calls, bandwidth, storage, networking? So reinforcing those budgets, going about finding the offenders that go above these budgets and chasing them down, solving them. So that's stability work. Reducing error rates, obviously, that's part of it, that's the result. And the challenge with working on stability, especially in light of server crashes, is that it's a fog of war. So you can launch a version of the game and there's one crash that's going around. We can identify and dedupe crashes, right? So we know what they are. So we're looking into one and then we're like, oh, if we fix this one, it's going to be very stable, right? But then when you fix that one, that allows the program to continue. And then more bugs show up, more crashes show up. So it's a fog of war. It needs constant care and attention. One thing we've done this year to help that is that we have a much more robust hotfix process that, frankly, we use a bit too much. But it does give us the ability to get some fixes out when we see big problems right away. But it's to contribute to that fog of war. So that's what working on stability is for me. It's all of these issues together.