Talk to me about our internal stability scores: how we've determined them and how we're using them.
TL;DREvery build gets an auto-calculated, imprecise stability score, from optimal to critically unstable, based on disconnects, crashes, runtime and concurrency; 4.3.0 sits between optimal and excellent, while 3.23.x was around unstable.
Yeah, so stability score is, I don't think we've ever discussed this publicly. So we have this measure, which is an indication that we use, and the team that actually comes up with this would already tell you it's alchemy, right? It's not an exact science, but it does give us a pretty good idea. So we try to establish a stability score for every build, right? We actually auto-calculate it based on the data that we get.
Not patch, every build.
Every build. Yeah, every build. So we send a build to PTU or QA or staging environments or whatever, and we always have a stability score that's calculated over time. And so what the stability score is, I don't want to go into the alchemy of it, but we're trying to figure out basically the stability of the environment. And so if you're connected to a game server and you get disconnected, we're going to count that. Now, the reason for your disconnection will be used as part of that. How long the build has been running is also part of that equation. And the concurrency of it is also part of it. And so with these black magic numbers, we basically get a score out of it. That's one thing. Now we've put labels on. So we used to use the score internally as just a number like, oh, it's a 0.08 build. And then at some point, I got a bit fed up with that because I'm like, what does that mean? And so the values for this are optimal, excellent, stable, unstable, volatile, crippled, or critically unstable. So that's the range, right? So we go from optimal, which is a great stability score, to critically unstable, and all of the things in between. And this is based on all the stats that we get. And we kind of auto-calculate it. Now, we still have Jeffrey Pease, internal alchemist, always playing around with these numbers or adjusting the score, telling us how bad we're doing. But it is actually pretty accurate in terms of what the perceived stability will be from a player's perspective. And so if I compare, what was September 11 last year? Was it like 3.23 or maybe 3.24 at that point? 3.23 or 3.24, I forget which one. I would say the stability score for 3.23.x was probably in the kind of unstable range, right? That would be a normal range for the 3.2x, this kind of thing. So the stability score, server score for 4.3.0, for example, is between optimal and excellent. And so that means the rate of server crashes and disconnections overall is really good. Now, this score is not just server crashes. It's also client crashes or client disconnections. So if everybody gets disconnected because one of our cloud routers dies, then the stability score of the build will be terrible, even though the servers are up and they're running real fine. So it kind of encompasses everything. We're trying to get one number for all of it.