Right before we released 4.3, we discovered something with the Wolf that didn't show up in any testing. Walk me through what happened with the Wolf.

This is a beautiful one. That was fun. So yeah, like you mentioned, we're walking up to 4.3. We're at the point where pretty much everything has been validated and tested. There's something with the Wolf that we're going to try to do more in the future, which is that we try to respect what we call the art of surprise. Typically before, we used to put everything in a patch, go, go, go. Now for certain things, we want to try to respect some level of surprise. We find that it's cooler for you guys when you discover things when it goes live and not necessarily on PTU two weeks before, right?

Or in shows every week.

Or in shows every week, exactly. And so the Wolf is part of that. And so the Wolf was going on PTU for validation because it was part of the 4.3 release, at the very last minute. So I think 24 hours before. Just to make sure everything's fine. Well, it turns out everything was not fine. And so we look at the reports. There was one crash that we had to solve, which was solved rapidly. But then we looked at some very curious reports, and so players would basically report, I forget the exact IC, but something like: I landed with my Wolf, exited the ship, and then was walking around. Then my friend took my Wolf and flew away, and then the world disbanded around me. So initially it sounds like a fluke report, but then a bunch of people reproduced it and it was like, I have this, I have this, I have this. And so, what is going on? And so typically that's where our player relations team goes like, something's fishy here. So we started to look into it. So we assembled a quick crew to check this out. And now this is kind of a skeleton crew because we're moving to release. Typically, it's very late here. I'm in the UK right now, so it's very late here in the UK. US teams, it's night for them. So just get this crew on together. Start looking at this issue. And so basically, the issue that we're seeing is that sometimes... when a bug starts with 'sometimes', you know it's going to be a fun one. So sometimes when you exit the Wolf, you desync between the client and the server. And what actually happened is, so we have this tool we call the server renderer, which allows us to basically view what the server sees, right? And so think of it like a side camera that we have when we have access to the game server and our game client, and you can see what the server sees. And so you'd see my character get out of the Wolf with that super nice animation. And he jumps at the end, jumps off the ship. This is super cool. And then I'm walking around on my client. Everything's fine. Game server: I'm literally in EVA animation floating over the planet. And the thing is, I can't exit this condition. No matter what I do on my client now, the server will always see me attached to the ship in EVA. So we're like, OK, what's going on? Now we start looking at this. So at this point, John Crewe and I are losing our minds trying to already get a proper reproduction. But then we realize, oh, we put new animations in for this ship. So the Wolf was one of the new ships that have enter/exit animations that are brand new, just captured, imported in the game, all that stuff. We don't do this all the time. And putting animations in the game is not just, oh, put the anim in. It's timing. There's events to link them. There's linking them to the actor system, how they get triggered, inserting them into ship pipelines, stuff like that. So we start debugging this. First step was, oh yeah, they probably forgot to put some of the seat events on it. So we do a first patch. We try it out. Now the QA guys come back like, nope, still happens. So, I think the timing is wrong on these events. So let's change that. Now, I think we went through like seven or eight iterations of fixes to try to figure out what was actually happening with that exact animation and why the ship system was disconnecting it. We went into full debugging on the actor. So the actor is your character and all the states it can go through. And we could just see the two differ. But it made no sense. Why is one working and the other not working? So at this point, we're kind of running out of ideas. That's when I called upon, I used a button, called the Bug Smasher himself, Mark Abent, on the show. And so Mark came in and was as flabbergasted as we were about what the hell is causing this. And so we went into a bit more debugging sessions. At this point, we're deep engine-side physics and collisions to try to figure out what's going on. And then in the end, Mark just goes, in the typical Bug Smasher fashion: what about gravity? And so we take a look at how the ships have physics grids. These grids have sizes and mesh sizes, and the animation is based on the geometry. And there was no gravity in the ship, zero, zero, zero, right? And so basically, it was a combination of the physics grid and the fact that the actor on the server runs at a lesser resolution. So your client when you're in game is at full resolution. The server doesn't run at full resolution when it's simulating your character. And so the combination of this would make the server actually reach the EVA state. So now you're floating in EVA, right? But with no gravity and no other movement, it will never solve itself. And it's also overlapping over the grid, which is too big. That was a bit of a combination of blunders, from geometry to wrong physics grid to no gravity, and it made the bug also not 100%. It wasn't like, exit the ship and it happens; it took about 10 to 15 times, you would get it one time. All right, so we fixed the bug. We get this in there. Internal QA validates it. And then to be able to properly validate it, as you know, we need players. And so that's where we mobilized you guys on PTU to come and try this out. And this was one of the best moments I've had testing in this game, where roughly 100 people showed up with Wolves at Boondoggle on Hurston. And then 200 people are entering and exiting the Wolves to see if they get detached from the animation. It was a fantastic moment. And so, yeah, that was a fun one to troubleshoot.