Back to search

The most demoralizing thing is when your stuff disappears: people earn a Polaris from Wikelo, and then we release a patch, now every month, and it's a crapshoot who gets to keep it and who loses it. What is happening, and what the hell are we doing about it?

TL;DRLost ships come from the long-term persistence system being fragile, since stowing happens through many paths that don't always record correctly, and the team is still finding cases where records are wrong.

Benoit BeausejourChief Technology Officer

Cool. LTP, right, is the famous system we're talking about. Well, let's go a bit into what LTP actually does, right? Because there's some misconception around it and also a bit of misconception about the amount of effort we actually put on it. So historically, if we go back to the 3.x era, we developed the first LTP system, right? It came a bit as a side request, which was, we patched four times a year. It was a bit like, oh, there's some stuff players want to be able to persist cross-patch. We're still in development, so it's OK. We're wiping all the time. And we want to wipe because blah, blah, blah. There's other considerations. So let's build something. So we built a very small system initially to persist stuff. And the way that worked is that the game server would basically submit these commands to say, this guy picked up a gun. This guy dropped a gun. This guy has this. And so that was a solution that kind of worked at the scale of a single-server kind of system. And we had a lot fewer items at the time. That solution was in no way scalable, right? But it kind of did work better overall than the current one, to be honest. Now come PES in 3.18, come 4.0, server meshing, now we persist everything else, right? So we need to change how we deal with long-term persistence. And so let me describe a bit how this works currently. So when you acquire a ship from, let's say, the web platform, that ownership record lives on the web platform. It's in there, right? And every time you log into the game, we check, oh, this guy has this thing. Does he have that ship in the game, in his entitlements, right?

Jared HuckabyHost, Community

It's called an entitlement. That's an important term.

Benoit BeausejourChief Technology Officer

That's your entitlements, right? So we make sure you have an entitlement record. When you actually open up your ASOP terminal and you view your list of ships, what you're actually viewing is your list of entitlements. These are not your ships. These are actually your entitlement records. Attached to your entitlement records is the insurance document that goes with it, which describes the loadout of the ship. We call this an entity class loadout, which describes what is in each port. And if there's no entity class loadout or insurance document with it, then it's the default loadout. So that is your entitlement records. That's what you see in ASOP. Then we also query for: is there an entity for this entitlement record at the moment? And that will look at the game database in this current moment. Let's call it the inventory database for now. So we cross-link the two. So for each row in your ASOP terminal, you'll see your entitlement record. The state of the entitlement record will allow you to see if you need to deliver it first. That means I have no entity in the game. So when you deliver it, now we create the entity in the game. And then if it's stowed in an inventory, then you can retrieve it. It starts from the entitlement record. And then if the ship is stowed, then you'll see its location is set. If it's left in space at a location, you'll see its location is set. If it's lost or destroyed, I think there's a couple of different states, but one of them is destroyed, then you can claim the ship. And then that will use the insurance document to destroy whatever matches this entitlement. So we have a way to find entities that were created from a given entitlement. It gets destroyed, and then we use the insurance document to give you a brand new ship at the location where you're at. That's in-game. That's the way the in-game part works. So let's do a clear example. I'm in New Babbage. I have a Hawk in my list, and I deliver it. It gets created at that location. I retrieve it. It gets unstowed onto the pad. I use it around, blah, blah, blah. I fly out to another place. I lose the ship. I respawn in Klescher if I couldn't do my time. Then I go out to Grim HEX or whatever, I respawn there. And now I'm in a different location. I see the Hawk in my entitlement list, but the ship is nowhere to be found. I claim it. A new ship is created. It's put in there.

Jared HuckabyHost, Community

The stuff that you pay actual money for, or that you get awarded through like a subscription or a gift, those sit on the web platform. Every time you log into the game, the game goes, hey, what am I entitled to? The platform says, these are the things you're entitled to, and it gives you them back. It's why you are never at any risk of losing the things that you pledge for. That stuff is, for the most part, very stable. You can always rely on getting that stuff back. That's not how the in-game stuff works.

Benoit BeausejourChief Technology Officer

Correct. So we went through the web platform stuff and how ASOP works, right? Let's go into, now I go to New Deal and I buy a ship with aUEC, right? That will actually give you an entitlement, right? Because you can claim those ships, right? You can transfer those ships. So up to now, these will work identically. But when you log into the game, we don't reconcile your list of purchased in-game ships from an outside source. They are just entitlements in your account. And that's it. We keep them. OK, let's stay on ships for now before we get to the items. Now, the thing that's important to know here is that the actual ship you're playing with, the amount of possible states for that ship are almost infinite. You could leave the ship in space. You could leave it in your friend's hangar. For some reason, people are able to do that. You can stow it in inventory. It can be inside an Idris that's stowed in someone else's thing. All kinds of stuff can happen. And so there's multiple paths to update that specific state of that ship. Whereas your entitlement, once you have it, you have it. That's why your ASOP terminal always has it. Now, let's talk about how LTP works. Now, this is within a game patch. When we release a new patch, inventory database, entitlement database, they're gone, right? We start from scratch, and now we're rebuilt. So you reconnect, you come in, your web purchases are being reconciled, we create new entitlements for them, and then when you get in-game, you create new ships, and then you play with those ships and everything works. What is LTP? LTP is when you acquire things in-game and you have ships that you own, right? The way LTP was designed, and maybe this is the part where there's a mistake, right? But still, I'll explain how it works. When you stow something in an inventory that you own, the systems will create what we call LTP records. These LTP records are close to what an entitlement is, right? But they say, Jared owns a Hawk, right? And those are stored in another database that is outside the game environment and lives cross-patch. And so now when we wipe the entitlement database and we wipe the game database and we start fresh from a new patch, when we reconcile your account the first time, we're going to give you your web purchases for sure, and we're going to give you back your LTP records. And so that will create those entitlements. That's the intent. Now the problem is, right, we need to make sure that we always stow the stuff back into inventories that we own properly and that those records actually save properly. And it turns out that there's a million goddamn ways in which stuff gets stowed in this game. For example, claiming a ship by insurance, then you get the ship. It gets stowed from another path. And in that path, the LTP wasn't correct. We've had cases where we tried to fix bugs related to the F7C turrets, for example, that got disconnected, that were a default loadout item that we didn't want to wipe when you claim insurance on it. And that messed up the LTP records for it. And so this whole thing basically is very fragile. And so what we realize now is that the list of LTP records and the way we update them is not always a slam dunk. In fact, there are a bunch of cases where the record's not updated right. We even had a case in 4.2.0 very recently where we realized two days after launching the patch that, the way the recording was going for the ships, there was an issue where, instead of giving you, let's say, I'm going to take your Wikelo Polaris as my example just to make it super egregious, your Wikelo Polaris that you stowed in your inventory by going to your hangar and storing it back in would actually persist as one of its components in LTP. So you would get a cooler instead of the ship. And so that was obviously a very big issue. And we ran corrective programs to solve that. And so that's why, if you notice, in 4.2.1, you didn't have many reports of people losing ships. But now we're back at 4.3.0 and we've had reports of lost ships between patches. And so we're pretty certain at this point that there's still cases where LTP records are incorrect, through certain game flows. There's also, unfortunately, a lot of signal-to-noise ratio issues, like people claiming they've lost ships. We investigate their accounts. They never had those ships. We get a lot of that, right? And so at this point, we have to be very prudent.

Star Citizen Live›Tech Talk w/ Benoit Beausejour·
Published on this date.11 Sep 2025