Freight elevators: where are we at now, and what do you want to tell us about them?

Let me pull up my notes. Okay. So freight elevators is one of our hero squads, right? Now, to your previous interventions, that came way too late in our planning, right? And so that's one thing that I think, if we could go back in time, that squad would have started way, way earlier.

I'm sorry, to be clear, work had started earlier. We hadn't had the hero squad on it.

Correct. Exactly. Good clarification. And so let me go through some of the stuff that we've been going through on freight elevators in general. So to start with, obstruction issues. A lot of the issues that we ran into were related to how the elevator actually detects obstructions and/or contents. So, for example, blocking the doorway, items being positioned on the roofs of the planetary freight elevators. Now, a lot of this, by the way, I'm going to talk more about the planetary versions of it, which are 10 times worse for some reason. But in terms of obstructions, you could leave items on top of the elevator, and then it was detected as an obstruction. And so these obstructions, first of all, the checks were super rough. And so they would go like, there's something, I'm not going to move, which is a bit like, come on. Somebody would put a small bottle on top of the freight elevator...

And the whole damn thing would stop working.

Exactly. That would break the whole elevator until that was removed.

MedPen. It was a MedPen. It's so small.

Yeah, but see, that's even worse, because there's a problem with how the items are set up, right? And so when we create a prop or we create an item, there's a bunch of things that go into it. Physics proxy, weapon proxy, all kinds of things that go on top of it. One of them is the bounds of the item. There's multiple bounds, in case you thought it was just a sphere or a shape or a rectangle. There's more than one bounds. Long story short, a lot of items have incorrect bounds. And so what that does is that when you leave them on the freight elevator, they might clip into the elevator itself. And so couple that with the bad obstruction checks, that could cause some really crazy exploits. Like, for example, leaving, I think, a MedPen on the freight elevator. It would clip through the platform. You couldn't even see it. So you'd be like, there's nothing on it. And then it would still block it. I think that actually is not completely fixed. So the squad still has to get to all of these items. And so for the bounds issue, what we're doing is we're adding a lot more checks on how the things are submitted, again, beforehand. So we're revisiting all bounds for items, right, and how their setup is made, to make sure that none of them can be in that situation. So we're kind of doing a system-wide check on this, in order to make sure that when we get into these situations, it's going to work the way we think it should. This is all super interrelated. Items that would get destroyed but would no longer have geometry, but would still be considered obstructions. And so that was a very egregious one, I think, that we saw very early, which is there was a way for you to destroy an item. Its geometry would disappear, but it would leave the entity present. That's it. That elevator is gone. You need tools to see that.

At least with the MedPen or something, people could zoom in and try to scan the pad, try to find it and remove it. This next one, though, the geo's gone.

Yeah, that's it. This one was the worst one, right? And so the thing too with the freight elevator bugs here, some of them were the setup of the actual freight elevator geometry, right? Actually, that was a regression. So it was a bug that got fixed and was reintroduced. Which is the shape, because we have multiple types of freight elevators, right? We have the horizontal one, we've got ones that are vertical, some with doors, some without. And so some of those were badly set up; the planetary ones were especially egregious in that case. And so there's still some bounds checks to be added. But when you got all these messages about obstructions, that was all related to this. One of the other issues we had with freight elevators was actually one of the first ones we had to fix, which is that the freight elevator is set up into two things. There's the kiosk and then there's the platform. And so the kiosk hosts the manager component. It's the one that's managing the stuff. And the platform is kind of slave to the manager. So there was an issue where, depending on how an area would stream in or stream out, the platform or kiosk would stream in first or stream out last. And this caused these two components to now be disconnected. So they would not find each other anymore. And so in those cases, that entire location was dead. That location, game over. Because they would never find each other. There was no model in place or process in place to go and try to fix that. And so this is part of what we do on the hero squads. We've done the same with the transit system, which is we introduce a self-healing approach. And so in this case, the system would actually retry to connect. So if a manager doesn't have its connection to its platform, then it would get retried over time until it actually connects. And so this way, even in the case where we do run into an issue where they disconnect, they could still reconnect. And so that solved the issue. That was actually rolled out in a hotfix.

Let's continue on our list of bugs with freight elevators. The UI and platforms could get out of sync. And so this is where the platform will have content. So there's clearly a box on the thing. And the UI is showing that it's empty. And so that basically makes the whole thing inoperable, because now you can't ship your box at all. In this case, I think that the streaming reconnection actually solves this by self-healing. But also the UI will re-query multiple times to be able to make sure it's always updated. This is a side effect of the game teams trying to cache a bit more data than they should. And so now it cached an empty elevator. So that's a weird state. So a bit of background on how the platform actually works. So you've got your inventory system, your warehouse. That's where your things are stored. So you can query for items in it. You get the list of what's in it. You can unstow items from it, which means bring them into the game world and physicalize them. Or you can stow them into it. The freight elevator and the ship cargo platforms are actually using an abstraction on top of that called the inventory container. And so the inventory container can be open or closed. What that means is that when the inventory container of the platform is closed, all of the items that are physically on it are actually stowed. When it's opened, it means that all of the items that were in this inventory container are in the game world and being acted upon. So when you unstow things on your platform, that's what happens. There was a weird case where, if there was a delay in the back end... so the back-end systems for all these operations have queues of operations that run, and so they can handle the delay. But in this specific case, there was a weird state where what's actually unstowed and the state of the container would be desynchronized. And so we've added another repair flow for this that would actually detect the state of it and actually resynchronize it in the cases where that would happen. This was because we didn't properly handle failures. And so in the case where an unstow would fail, the state would now be in a weird situation. And so now we do the proper error handling, which is another thing that we've changed in our processes for development: error handling is now part of code review. So we really need to make sure every single... I mean, that sounds basic, right? But I'm just saying the way it is, right? Some cases were not handled. And so you need to handle those failures. And same thing for door states. So whether the door is open or closed after a stream in and out or a server crash would sometimes clash. And so the kiosk can think the door is closed, but it actually is open. And so we do the same now. We can actually properly set the flow back. So one of the lessons we've learned with freight elevators in general, and all the issues I just went through and the reason why we ran into them, is we need way more multiplayer testing. I think that the major key behind it is that the teams have spent time on the in-hangar elevator, right? So we ran into some issues with the in-hangar elevator, for sure, and there's still issues there. The big difference with the Resource Drive event was that we were driving players to the freight elevators, and there's more than one player interacting with it. And so that's one area where we have improved our testing and our testing approaches so that they're properly tested multiplayer, and not just when we reach PTU or in focus tests. Now it's part of the general testing for these open areas.