What happened [with the 4.8.2 release yesterday]?
TL;DRA bug-fix check-in that shouldn't have been in the stream made Idris ships incompatible with their saved copies, so game servers crashed on spawning them until Idris spawning was paused and the ships were manually fixed.
Well, it started with an attempt at a bug fix a month ago. So with this 4.8 release and the way we've been doing patches this year, we have two streams when we work off from the working set. We have a base stream that we use for the base release, so 4.8.0. And we have a hotfix stream where we bring all the continuous changes along. So there was an attempt at bug fixing something on the Idris that was actually creating some performance issue somewhere else that was checked into that stream and nobody saw it and it went through. So we've been on the hotfix stream now for 4.8.1, now going to 4.8.2. In 4.8.2, we went back to the base stream, which are supposed to be at parity at this point, except additional content because we've added additional content in the patch. That change was included. And so what's supposed to be a routine update became a massive, wide-sprawling issue because basically the Idris was no longer compatible with its persisted copy. And so what that would do is players have in their ASOP terminal a version of the Idris that's from 4.8.1. This change, which was not supposed to be in the stream at all, gets in there. And so now the game server is expecting another shape for this. The game server has a series of checks and balances for this, but he's the last banker in the chain. And so for him, if the schema is incompatible, he's like, there's data corruption risk, I'm out.
Who's he in this analogy?
Well, he's the game server, basically. So the game server is trying to spawn the Idris, wherever it is, right? So that's called the async spawn chain. And so as he's trying to resolve the object containers for this and to materialize the ship in game, he comes around an incompatibility. And then there's a hard check for this. Because if we let those through, if we disarm this check, for example, then the game server might corrupt ships or might be twisting data and then we would go wider than that. And so the game server would crash.
And so every time an Idris would be attempted to spawn, that server in that region would crash and then restart and then try to load the Idris again and then crash again. And so as players were unstowing more Idrises in more shards, more servers got affected, until you reach a point where now the ops team is in the background and they're lighting all the fires, calling the Rohan. And so we're looking at this and basically we start cordoning off shards, because at the beginning we're not sure what the issue is. And what happens in this case is the more shards are getting offline, the less capacity we have for players that are either entering the game or shard hopping. We have a lot of people shard hopping. And so what happens is they hit the dreaded 60012 error, which means, hey, there's no capacity in your zone. There is capacity, but we've had to restrict it in the meantime. And so we took a hard decision later in the night to prevent players from spawning the Idris temporarily, just to stop breaking other shards. We brought more shards online to keep people playing. And then we basically went through and manually rectified. That's what we call percussive maintenance. When you see that in the status, it means we went in and we did a drastic measure to basically fix the Idrises that were either spawned in shards or still left in inventories, to make sure that they had the proper schema. And so now we know that all the Idrises are fine. We've lifted that temporary stoppage for that ship. And so now we're back. That's what happened.
This is cabaret trays being spawned by some subcomponent in the Idris cargo room.
And it's not even anything specific. In this case, it's the Idris, but it's not because it's a capital ship or anything.
No, it could have been any ship in that case. So normally, there's a check-in discipline that we apply, especially this year, because we have a different publishing model this year. And so ships that have been distributed already in a given major-minor patch cannot be modified. Now, for some reason, this check-in passed all the validation checks and ended up in the build.