Elevators and trams: broken or missing elevators in various locations, preventing access to hangars and other areas or simply trapping players inside, and trams that either let you fall through the floor or yeet you into low orbit. My question to you: what the fuck?

So, brief history of transit. This is a very old system that's been in the game for a long time. And transit in the context of streaming is already a non-trivial problem to solve, and I need to dive into the why to explain a bit the complexities behind it. So the transit system is composed of multiple components. You've got peripherals. So that's, for example, the button you press to call an elevator or a tram. It's the indicators that say, hey, the tram's coming in this amount of time. Those are peripherals. You have gateways, which are locations where a carriage can stop. So, for example, if you're in Lorville and you go to the train, where the train stops, it's like a station. That's called a gateway. You have carriages, which are either elevators, trains, trams, all these things that move along the path. And then they go from gateway to gateway and they respond to orders that are submitted from peripherals. All sounds nice and dandy, what's the problem, right? So in any other game that doesn't have full-on level streaming, all of these things are always loaded. So the manager, the piece of code that actually is managing, fulfilling the orders and moving the carriages in general, or has the logic of the carriages, all this network is streamed in. It's all loaded. So it's always available. It knows where the carriages are. It knows what's happening. Everything is there. In our environment, because of our SOCS technology that allows us to have more performance on the servers because we unload the rest of the zone, that means we unload those carriages. So think about your Lorville train that you see stop at you and then leave. On the server, if nobody's on the track, it doesn't have a carriage there. It's unloaded, right? And so the current iteration of the system actually destroys that carriage and will recreate carriages when it needs it, right? And there's been a ton of issues with this. So that's the first part, right, is the streaming part. That makes it a bit more complicated. The second part is that the way the system was built initially, it would fulfill these orders, send orders basically: you call the elevator from your hangar, I need to send a lift there. The system is going to tell the lift to move. The lift moves on its own, right? So it will jump and then do the actual path. We want those to be physicalized so they're actually following certain paths. Now, the issue is that, combined with streaming, these orders and state are lost in certain conditions. And so the system just needs a whole lot of changes on its communication layer to be reliable, and it needs a fundamental change in how it fulfills these orders. Instead of just sending a message to a carriage and hoping for the best, the transit manager needs to be actually always tending towards the state that it should be in, so that it can self-heal for any kind of issues, right? Whether creating the carriage failed or streaming out and losing a piece of it, the system needs to change to be able to properly do that. So that's what the new system that we're writing is going to do. It's going to solve these communication issues with streaming and it will make the system self-healing. We also need to do fixes to the current system immediately to get that working. So now we do have a crack team on this right now, where we've significantly increased the size of the group that is looking at the current elevator issues, which we have absolutely regressed on. And they're looking at adding the self-healing component I just talked about directly in the system. So I was happy to hear that we now have proper reproductions for at least a couple of cases where the gateways are basically becoming defunct. And so you call the elevator and it never shows up. That's the number one case that we see at the moment. Or the other one is a lift shows up, you press a button and it doesn't move. So that's when the transit manager has lost this carriage or the gateway is no longer usable. And so now we have those reproductions, and they're not just happening to players. Now we have an actual case where we're able to actually go down and we've seen what actually is going on. So we can put in the self-healing things I just talked about to fix it.