TL;DRA bug-fix check-in that shouldn't have been in the stream made Idris ships incompatible with their saved copies, so game servers crashed on spawning them until Idris spawning was paused and the ships were manually fixed.
Well, it started with an attempt at a bug fix a month ago. So with this 4.8 release and the way we've been doing patches this year, we have two streams when we work off from the working set. We have a base stream that we use for the base release, so 4.8.0. And we have a hotfix stream where we bring all the continuous changes along. So there was an attempt at bug fixing something on the Idris that was actually creating some performance issue somewhere else that was checked into that stream and nobody saw it and it went through. So we've been on the hotfix stream now for 4.8.1, now going to 4.8.2. In 4.8.2, we went back to the base stream, which are supposed to be at parity at this point, except additional content because we've added additional content in the patch. That change was included. And so what's supposed to be a routine update became a massive, wide-sprawling issue because basically the Idris was no longer compatible with its persisted copy. And so what that would do is players have in their ASOP terminal a version of the Idris that's from 4.8.1. This change, which was not supposed to be in the stream at all, gets in there. And so now the game server is expecting another shape for this. The game server has a series of checks and balances for this, but he's the last banker in the chain. And so for him, if the schema is incompatible, he's like, there's data corruption risk, I'm out.
Well, he's the game server, basically. So the game server is trying to spawn the Idris, wherever it is, right? So that's called the async spawn chain. And so as he's trying to resolve the object containers for this and to materialize the ship in game, he comes around an incompatibility. And then there's a hard check for this. Because if we let those through, if we disarm this check, for example, then the game server might corrupt ships or might be twisting data and then we would go wider than that. And so the game server would crash.
And so every time an Idris would be attempted to spawn, that server in that region would crash and then restart and then try to load the Idris again and then crash again. And so as players were unstowing more Idrises in more shards, more servers got affected, until you reach a point where now the ops team is in the background and they're lighting all the fires, calling the Rohan. And so we're looking at this and basically we start cordoning off shards, because at the beginning we're not sure what the issue is. And what happens in this case is the more shards are getting offline, the less capacity we have for players that are either entering the game or shard hopping. We have a lot of people shard hopping. And so what happens is they hit the dreaded 60012 error, which means, hey, there's no capacity in your zone. There is capacity, but we've had to restrict it in the meantime. And so we took a hard decision later in the night to prevent players from spawning the Idris temporarily, just to stop breaking other shards. We brought more shards online to keep people playing. And then we basically went through and manually rectified. That's what we call percussive maintenance. When you see that in the status, it means we went in and we did a drastic measure to basically fix the Idrises that were either spawned in shards or still left in inventories, to make sure that they had the proper schema. And so now we know that all the Idrises are fine. We've lifted that temporary stoppage for that ship. And so now we're back. That's what happened.
No, it could have been any ship in that case. So normally, there's a check-in discipline that we apply, especially this year, because we have a different publishing model this year. And so ships that have been distributed already in a given major-minor patch cannot be modified. Now, for some reason, this check-in passed all the validation checks and ended up in the build.
TL;DRBeausejour says 4.8 involved constant firefighting, including two bad check-ins that broke persistence, though the overhauled entitlement system worked well, and Lind says extra protection against such check-ins is needed.
Well, for me, if I take it from the machine room side of things, we've been firefighting since we launched 4.8 from a whole slew of different issues. Some of them are to be expected when we launch a patch that has this many core changes. We've touched on systems that are really central to the traversal again. So we're talking about the entitlement system, which has not changed in years at this point. And we've completely overhauled the entitlement system in 4.8 to power in part the item imprint system, but more importantly, to pull the entitlements out of the patch cycle and persist them across patches. What we call the LTP layer is not just what you have in your inventory, it's also what you own. Your ships and all of the items that you purchase in-game now have an entitlement record that holds the insurance loadouts for it, all of the customizations that you make on it. And that's hosted outside the game. Not outside the game, but outside the game patch. Which means that when we come back from a patch, there's no risk of, oh, I'm not getting my ship back. What you see in ASOP in the first patch will be exactly the same data source in the second patch. And so that risk of losing ships across patches is completely gone. So we're very happy. In fact, that bit actually went very well in 4.8, even though up to the launch, it was always last-minute things to look at. But we have been firefighting across a multitude of different issues that were tangential to that. Some of them we can touch on a little bit, because I've made some notes to talk about gameplay systems here and why they were affected by some minor infrastructure issues that turned out to be massively contributing to the perception of the patch. So I'll talk a bit about that in a bit, but basically we've been firefighting since 4.8.0. I want to touch on one thing because we just had an off-the-cuff moment about the Idris last night. But two weeks prior to that, we had another downtime patch that needed a lot of rectification. So the teams that run the online group were hard at work on this maintenance. And unfortunately, another one of those unwanted check-ins came in during that 4.8.1 cycle, where, in an attempt to fix the VGL flight suit that was missing hands and feet, another check-in came in that removed sub-components of this flight suit, but not just removed them from the flight suit, deleted them from existence from the game. And that's not something that you can do in a minor patch. And so clearly, that's two instances in this patch where transition check-ins should have been validated and prevented and went in, and then after that became an infrastructure problem. Because same issue here: if somebody was wearing the VGL flight suit and referred to these entity classes that were actually representing, I think, the nozzle and the front pack of the suit, then the game server would crash. And so we had, again, to do percussive maintenance to remove those items and fix that. And so that highlights to me a massive issue with how we validate check-ins and how we get changes in the game across persistence, until we have full forward porting compatibility on the persistence layer, which we don't have at the moment.
Live servers, you're always running, right? We're talking about forward patching. If we'd been behind closed doors, you would have done that, right? No one would have ever seen the game until you'd done that piece of technology, but that's not who we are. We're always running, and that's what makes it really, really exciting to work here as well, right? Because we're putting ourselves out there all the time. But if I look at this specific case, you're saying, right, we have checks and we have contracts on what you can and can't do in these sub-streams that we've already released, where the data management is super, super strict. And obviously now we invalidated them twice in a row. So it's time to look at how we really add that extra layer of protection.
TL;DRBenoit Beausejour says the game is still being turned into an MMO and content and features are outpacing the guardrails and cross-system validation, so the team needs to add stronger validation.
I think there's a part of that which is that we're still transforming the game into an MMO. We're still in the process of taking a game that used to run on a single server and now obviously a server mesh, which is working, but the game systems need to expand as well. If you add the content pressure on top of that, of getting content into the game, releasing it at the same time, stabilizing it, hardening it, and features that come in and do the inverse. New features come in and they destabilize the systems. Without stronger guardrails that also need to evolve with this, that's where we ended up. And so, to be honest, I think right now we're outpacing our ability to verify everything that's being added to the game. And this is where we need to put a lot of our energy at the moment, to put in those guardrails, so that the check-in that would cause this crash issue, right? Those are identified at this point. And so it's all about adding these validation frameworks to prevent that from happening and keeping the data stable and fully validated. Now, that sounds... well, why didn't you do that before, right?
No, no. There are guardrails in place, but they're not keeping up with the pace. We create more content and more features than we have time to add guardrails and to prevent these cases. An example I could give on something unrelated to persistence: we used to make a weapon, we make a ship, that's it. Now we have a team making weapons. They balance the weapons. They design the weapons. They work in that realm. They have guardrails in there. They know what to check for, outliers, data sets, how power is distributed, et cetera. You go to the next team. The next team is making ship components. They have their guardrails. They have their approaches. They balance it out. You have another team that's making recipes and dismantling. They have their own guardrails, their own systems. You have another team that's making economy. They add those things to shops. They add pricing. They make sure that that's balanced. And then it moves on to another team that's adding this content to loot boxes. Now, we didn't have that a while ago. And then mission rewards are added in there. All of these have validation, but there's no holistic validation across. We're at the point now where all these systems communicate with each other all the time. And crafting has changed the game quite a bit in that area. And so that translates into more weapons being created, more content being created, more missions giving more rewards, and all these things need to be validated together, not in silos.
TL;DRBeausejour agrees the weapon pricing and the crafting breakdown were validated separately with no check across them, and says human validation is no longer enough, so machine validation is needed for everything.
Transversal to check all of that. And also, we're definitely now, post 4.7, in a realm where just human validation is not enough, right? And so a lot of game design data is typically managed by the team, but I think that we're way past that and we need machine validation for everything, because the rate at which we build it and the amount of technology we're putting in is just too fast for validation to keep up.
TL;DRBeausejour says the team is building a validation framework that designers can use to query across all game data and find outliers, plus more compatibility checks and the beginnings of patchable database data in 4.8, aiming for forward-patchable data.
Sure. So it's a problem we've known about and we've been trying to solve. And so more recently, there's more changes that have been coming from the engine side to give us a framework for validation. When I say framework for validation, I mean one that game designers can participate in, right? It's not like, oh, they need to hand it off to an engineer, the engineer is busy and he can't do the work and now the validation is not added, which is the mode that we've been in in the past, right? And so we've been in the process of building a validation framework for all game systems and all pieces of data stored in the game that can do queries across all data, whether it's, hey, I have a mission in this location that requires this, that pays out this. Is that an outlier compared to others? And so being able to do cross-data queries across the entire game, which is not something we've had before. Which allows us then to just express these checks, find outliers in mission payouts, find outliers in how, if we go back to the example of the gun: it's for sale at this location, at this price, it's dismantlable into these components that can be sold at this location, that exchange value is too high, right? And so creating these validations will become part of the design work with this framework, which will allow us to have all these guardrails. On top of that, we've got the more technical validation. So the two issues we got in 4.8, related to modifying ships that were already distributed in the main patch and removing entity classes, for example: both of those things already have locked checks right now. And that's the first thing we did when we discovered them. But we're going to keep adding more of those compatibility checks. 4.8 also has the beginning of database patching. So in 4.7, we didn't have the full ability to persist all the crafting properties on weapons. So if you crafted weapons, all the modifiers were not necessarily going to persist through LTP, right? In 4.8, we have introduced in the backend patchable serialized variables. We didn't talk about it much because it's kind of mixed into the crafting system, but basically now, in the schema of a component, which stores the data for something, let's say it's a weapon and I have the modifiers for it, let's say elongated nozzle, faster damage rate, whatever, we now know which one of those values is patchable. And then we're able this way to persist them and patch them in the next release. We also determine which ones we store when you stow it. We don't stow all variables, which used to be the case. And that's the foundation for us to keep working at what we really want to achieve, which is a forward-patchable data set, so that even if the ship team goes in, changes all the serialized variables for a ship and rejigs the whole object container layout for a ship, we'll be able to live migrate your ship to the new version without crashing the game servers in the meantime. So basically embracing this movable data set. But that's tricky to do. Obviously, we've been at this for a long time. But 4.8 has the beginning of that, just for crafting properties.
Well, it's pre-validation, right? This validation framework I'm talking about is pre-check-in. I'm working on the game. I'm like, I'm going to add a new gun and I'm going to price this and I'm going to make it super powerful. Well, that would flag. I could not check that in. The system would go, wait, you need some authorization before you can check this in, because of these reasons, which currently we don't have. That's cross-checking. Obviously in each silo, yes, but not within.
No, we're working on this right now. To get all these checks and balances in place, I don't know if it will be like a 4.9 kind of thing, but it's happening now. Because everybody around the table knows now, it's clear from the 4.8 experience, that machine validation is required across the board.
TL;DRJens Lind says the problem came from the 4.8.1 database combined with 4.8.2 data, an environment QA doesn't have, so hiring many more QA people wouldn't have caught it.
In this case, it took the situation of how the database on 4.8.1 was, with the new data from 4.8.2, to create the situation. And that's not something that the QA environment has. So whatever we do, we as a technology group would have to come in and at least give them that. And then if you put your 500 QA on it, then probably one of them would have spawned an Idris. But without that environment, you could put as many as you like on it. No one would ever have discovered this situation. This is just too far removed. And I think that sort of comes back to the large dataset, right? As you have a large dataset, you just can't scale in manpower alone. You need something that scales way beyond that.
TL;DRBeausejour says the team knows what to do, that building the guardrails will take a while but not months, and that he's confident such crashes and misconfigurations will stop as long as nobody trips the validation.
I think we know what we have to do. I think it's going to take a while to get all of these guardrails in place, right? But in the meantime, we just have to respect the rules that we've set forward, right? So that while we can't machine check everything right this second, and to be honest, I'm not talking about months of work here on these validation frameworks, right? But it's about putting in the ability for every team to contribute guardrails, which right now is a bit splintered, and so we need to bring this all back together. So I'm confident that, especially for the errors we just talked about, like releasing a piece of content that crashes every server on launch, twice, right, or evident misconfiguration of items or costs or recipes, I'm confident that that will solve the issue, that those issues will no longer occur as long as we respect the validation and nobody trips them off, which is the next battle. I think that we can eliminate that.
TL;DRBeausejour says the cause isn't pinned down yet, but some server meshes run below the target frame rate, and the team is tracking six to eight optimizations, including lock contention in the area component, for the hotfix channel.
Well, so what we know about right now is that there are some meshes, and when I say meshes, I mean a dedicated game server inside a shard, that are running way below the FPS spec that we want. And so we don't have a handle just yet on what is actually causing it. We do have performance captures, and we're tracking at least six to eight very specific optimizations that need to happen. One of them is within the area component, which is used all over the engine. Area and zone component, they kind of go hand in hand. Again, because we're pushing more content, the hierarchy tree of the game world is much bigger. So there's more areas, more zones, more subdivision, more, more, more. There are some areas of that where you add a lot of entities that enter or exit these areas, and the engine provides the listeners to track entities entering areas or leaving areas or zones changing, right? And obviously there's concurrency constructs around this to prevent one thread from accessing another thread, right? And so, just keeping it simple without diving too much into the low level, which we will probably do in some of our desync areas, there's lock contention in the area component at the moment in some places. There's lock contention in some areas of the game. Well, I literally mean area: if there's many entities in this spot, because of the way it's configured, we're seeing basically FPS drops in those areas. And so we're tracking, like I said, about six to eight major optimizations that we're trying to get in our hotfix channel, to try to get this FPS back up to the spec that we want, which is above 20, across the...
No, no, no. And it's important to note that when a server is running low FPS, it doesn't mean that it will create desync, but especially when you're interacting with entities that are simulated on that DGS, AI characters, NPCs, then you're going to perceive these things, because the source is slow at creating its output. The traffic goes from your game client to the replication node, replication node to game server. Now, if the game server is running slow: at the beginning of the frame, he's taking the state from the network to the game and applying it. At the end of it, he's applying the changes back to the network. If that loop is slow, then your client's going to predict something, and then that prediction will get corrected the next frame, and that will create a rubber band, right? And so that's the gist of why there's desync in general. Why does the client have to predict? Because if he doesn't predict, then you're going to be at the tick of the server, and you'll see...
TL;DRBeausejour says ship cleanup is one thing being looked at: the system seems to work but may be misconfigured in some areas, letting ships pile up, and more entities make servers do more work.
That's one of the things that we're looking at, how aggressive or not the ship cleanup is actually happening. That's one of the things we're looking at, yes.
Because you can watch a Twitch stream or look at an Issue Council report and you'll see dozens and dozens, I feel like it's even more, of bricked ships everywhere, with the insurance changes bricking these ships. Is this viably part of what it is?
In part, yeah. That for sure doesn't help, because the more entities are replicated, the more the server needs to do every frame, and so that contributes to this. So, yes.
No, a server not performing so great tends to have a slightly larger vehicle population or a very large active population. That can also happen. In general, it's just a lot of everything. We're quite a heavy game, entity-wise. We want to avoid getting to the sort of size that we were in Stanton before we went into meshing, right? Because that was consistently difficult to manage.
I know that there's been some changes related to how we perform entity cleanup. So unattended vehicle cleanup used to be a separate system that ran on its own. That's been merged into what we call the density manager, which is basically an engine functionality that will look at overpopulation in clusters of areas that designers obviously have control over. For example, we want to make sure the exit routes for hangars are cleaned up. And so these have active cleanup zones configured, right? That's designer controlled. Now the engine will then apply the logic on the volume, determine, oh, there's too many of X in this area, and will start popping stuff out, right? Basically going to do the cleanup. Now for ships, the unattended vehicle cleanup system used to be separate. Now it's been brought into the density manager, I think end of last year, something like this. So it's been a while. And this one tracks whether there's player passengers on a vehicle and will, based on a certain timer configuration, and the values are different depending on the areas, either stow the ship back into the originating inventory or destroy it, depending on if it's a player-owned ship or if it's an AI ship. It does look like maybe there's a misconfiguration in key areas that are leaving ships there. So we don't think the system is incorrectly functioning. We think it might be inaccurately configured in some areas. And so that would allow a very popular area to just accumulate ships like crazy. And so that's what we're looking at right now.
TL;DRThe speakers are unsure the density manager has a specific rule for bricked ships, but say it will eventually, and that it works on how crowded an area is more than on timers.
It will. If it doesn't, it will eventually. And it's less about timers. It's more about density. It's trying to say you have overpopulated this area. So it's less about that ship having been there for a long period of time. It's more like, there's a lot of ships here. Let's remove one. Let's figure out which one is best to remove. Unattended vehicle does have a timer as well. So it's sort of like we talked about with validation. You start with a timer, then you start adding density, then you start building up these views of the world to understand which areas are problematic. Like, oh yeah, the entrance to a hangar, we have to be super strict.
There's a place, as we're talking about this, something that pops into my mind that I think is also in our process. Not even the development process, but let's call it live operations of the game. We're very lean in terms of our live operations at CIG, because we're in development. And so even though we do run a live service, we try to run a really lean ship on that side. And so we don't have an army of site reliability engineers that are looking at every aspect of, oh, shard 25 has an overpopulation of ships at Jackson Swap between this time frame and this time frame. We don't go to that depth. And so we track overpopulation across all shards, but we can't go granular into this. But that is an area where we could improve this, detect those things and better tune these values. Typically right now we will tune the density manager values every patch or every hotfix, but that is a slow cadence compared to how you would react if you see, oh my god, on every shard this location is loaded, let's effect the density manager change right now. Right now we don't have the ability to do that live, and that's a problem. That would improve the game quite a bit.
TL;DRBeausejour suspects behavior changes tied to the Imprint and insurance changes, such as more ships left behind or players favoring loosely configured locations, while Lind can't point to any other gameplay change that explains it.
For me, it's behaviors. So there's one thing about running a game server, which is that it's kind of an organic behavioral system, right? And so you can have a game server with a version of the code and a version of the data running a certain way, and then players start doing something different, and that completely changes the performance profile. And so I feel like in 4.8, most likely, my hunch for these six to eight performance issues that we're looking at is that it's related to some Imprint changes. So the way people use ships, leaving bricked ships behind: are there more claims than before? Are there other egregious issues that showed up that are creating more claims that would affect the amount of entities on the server? Is there a malfunction on, for example, the density manager cleanup that is leaving items in place? Or are people just going to a location that's configured to be very loose on its density, and it's somehow the location that people have decided to go to in this patch? So sometimes the performance profile of the game servers changes just based on behaviors, not on, oh, we changed something.
I mean, it's hard not to just parrot what you just said. It does feel like, when I think of 4.8, there's nothing that I can point to and go, yeah, those gameplay changes we made could have caused it. The only thing I can think of is, we did performance improvements, but there is a large change in how we handle vehicles and the lifetime of vehicles.
I mean, that would be, based on what I know of what that patch put out there. I just don't recall there being any larger changes on the gameplay side that I feel I could explain it with.
TL;DRBeausejour lists several suspected causes without going deep: navmesh tile changes on planets, an entity spawner respawning, area and zone lock overhead, ship cleanup with active MFDs and flight systems, and fire and temperature physics.
I don't want to deep dive into each of them, because it's a bit intense if we go down that route. But there's navigation system dispatching of tile configuration changes on planets, which affects our AI using NavMesh on planets. So there's a performance cost on that that is costing us a chunk of frame in the servers. This is purely based on the servers we know are affected by this problem.
Yeah, exactly. This is cross-linking a lot of things. This is a pet peeve of mine that I keep pushing on, and we're going to definitely resolve that from a software point of view: there's an entity spawner in some of the areas that are affected that is coming in and out of streaming and respawning its spawn set every time. These are the NPCs that just spout out. That's one example; I have multiple others related to this. That's one. The area component we talked about. So what we call register with external systems is a phase when a component or an entity is coming online, when it's streaming in. It needs to register with external systems. That's a notorious area. That is an issue on multiple components. This one, from the area component, is a problem in that place.
Correct. This one is more engine and, let's call it, area configuration related. This one bleeds into the zone lock that we talked about earlier. So those are two, area and zone lock. So this is really tied to the amount of content in the game and the complexity of that content in some areas. Because we're also making more and more complex scenes, like space stations and the content we've put out recently, whether it's TSG or Rockcracker, there's a lot more complex layouts, and those elements contribute a lot to this. And so those are part of that. The ship cleanup is another big one that we're looking at at the moment, which is probably the source of the other two, which is the amount of registered entities that have either MFDs and/or the flight system active.
Well, we don't know specifically. I bet that if I went into that area and I looked, there would be a bunch of MFDs that are somehow visible. So these components have a visibility aspect to them where, if you look at it, it's going to replicate for you. But if you don't look at it, it doesn't. Might be a malfunction with that. We're not sure.
It could be broken ship MFDs. We've seen weird things; everybody knows about the graveyard. There's been other graveyards in other areas, right? So there might be a new one we haven't found yet where somehow MFDs end up there.
And one final one, that is fire. So some ships catch fire. And depending on the size of the voxel grid, we've seen that the area physics properties of how the temperature system interacts with fire actually need a big optimization, and it's costing us frame time. That's a combination thing, because internally we can't reproduce it, but we see the cost on some of the game servers that are running slow.
TL;DRBeausejour and Lind say optimization and crash fixing are a fog of war, where fixing the known issues may help or may just reveal new ones, and different servers can have different problems.
Now the thing with everything in optimization is that it's a fog of war. You fix that, that's what was slowing it down, then you find the next thing. You get rid of the elephant, and behind it is another one. The same properties apply to crashes, right? Typically when we do stability checks on game servers, which, by the way, other than the two big issues that we've had, the baseline stability for 4.8 outside of these two events of the missing entity classes and the Idris problem has been in a really good range overall. And so we've been happy about this, other than now the stats are terrible because of this. So we'll have to take a look. But the same properties apply to crashes. So when we look at top crashes, you're like, oh, we just have to fix this one. It's super high frequency. Then we fix it. And then, oh my god, there's three more. And so it's a fog of war. Optimization and crash stability are fog-of-war activities where, yeah, you've got seven now. You know what those are. We fix them, it might be the end of it, it might not be.
You can get clusters as well, and that's the same on the performance side. We might be looking very, very closely at the servers performing poorly, and we might be fixing things that help those, but that doesn't help at all with the other servers, because they might be running into completely different issues. There might be something that's happening specifically on those servers that's unique to them, that brings them to that low rate, whereas the rest of them don't benefit as much from that. So you always have to look closely at the details and react to those, but proactively try to look at the big picture and where we are going overall. Once we fix that cluster there, how do we get all of it to benefit from a smooth ride with zero desync?
TL;DRBeausejour says shard lock is the price of keeping shards fluid instead of a classic MMO setup, that a 4.8 bug stopped some characters from being stowed and has been fixed, and that players should be patient before using character repair.
Well, so I want to say first that shard lock is kind of a community word. We didn't come up with this, actually. But the way we've architected Star Citizen, even though it's an MMO, we operate it from an ops side of things like a matchmade game. When you come in and you pick a shard, or you shard hop to one of your friends, you're just getting one of the shards that's currently running, right? And there's the process of bringing your character and unstowing it into the shard, and then allowing you to unstow items from your inventory down into the shard, and putting it back up there and getting out of there, right? It's kind of our attempt at not applying a classic MMO server setup to Star Citizen, where you would create your character on a server which then needs to run at all times, that when it's low population we need to consolidate, that we need to do character transfers or character renames, this typical kind of MMO grid system, right? It's a company desire to not do that pattern and to instead keep it fluid like it is now. So shard lock is a little bit of a price to pay for that. If you have any movement between shards, your character will need to stay in the shard until the hybrid decides that you're properly disconnected, that you've been gone or that you're coming back or not, and then move you back up. Now, in 4.8, we did run into some issues for some players. In this case, it was caused by an infrastructure failure, a small knock-on from a change that we made as part of Imprint. We track where you are with what we call the stow context. And so for the entities that we care about, because we don't care about every entity, and your player entity is definitely one we care about, we track where it is in the game world using your stow context. So are you stowed in the global inventory? And in what inventory are you stowed? Or are you unstowed in a shard? And if so, in which shard? The problem that some characters ran into is that some players are not very patient with the system when they go back to the main menu. And so they would go back to the main menu and it says, oh, your character's still in the shard, right? Or they would kill the client. Sometimes they do that. Kill the client, come back in, and then, oh, your character's still in the shard. I'm shard locked. Character repair. Well, the character repair in that case is the one that would cause the problem, where now the player would never get stowed, because the repair would try to repair your character. Anyway, there's a bug there that made it so that your character cannot be stowed. And so these players were shard locked. Some others ran into another condition. It was a fairly small subset of players overall that got into that situation, but it took us a while to identify the root cause, and that fix has been deployed. But I just wanted to mention that shard lock in and of itself isn't a bad thing. It's what enables us to have the flexibility to bring you into another shard kind of transparently, but we need to wait for the process to finish on your current shard before we can move you up, right? Now, we are in the process of putting in place active monitoring for players that get unstowed in a shard and stay there for a long period of time, so that we can attempt self-corrective actions if we run into that condition again, so that we don't need to chase the actual source. We can actually get some self-healing in place. But I just wanted to touch on what shard lock is and why it's kind of normal, and that it's better to be a little bit patient with it than to jump to character repair right away.
TL;DRThe team is working on hangar bugs such as exterior doors staying closed, players being told to enter someone else's hangar, and repositioning getting stuck after impound, mostly from failed handshakes, missing timeouts or invalid reposition positions.
So hangars. I'm going to start with hangars. Now, I'm going to pick the ones that I think are interesting in the list, in each segment. These aren't in any particular order. This is just some of what people are working on today. So: PU, multi-locations, instanced hangars, "exterior hangar doors remain closed while interior hangar door opens." So that's when your hangar doors open, but the gateway outside is still locked.
So they belong to two different parts, right? You have the exterior and you have the interior. The interior is yours. You can control that in your instance. The exterior needs to be synchronized with your interior door. So what most likely happens there is, for some reason, your interior thinks you've been given the right to this gateway. Or you could have been given a right to a different gateway, and there's an exterior door opening somewhere to nothing, and your door is opening to a closed door. So there's a few things that could happen there, and normally it's that handshake between the exterior and the interior, via the notorious ATC.
But this one goes with its little brother, overlapping hangars: "Player reposition when landing in hangar: ATC given, but seen as trespassing in someone else's hangar, results in being repositioned." Which I think is a sibling.
Yeah, definitely, because you're looking at a situation where you think your interior has been given a certain gateway, you're heading there, but it's actually someone else's casa. So you're correctly trespassing, but you've obviously incorrectly been told that it was yours.
And then the third of the trifecta in this segment, if you put them all together, is "players can get stuck at repositioning player indefinitely after getting impounded." And so if you get impounded, there's a chance that the server will never complete the repositioning that's supposed to take you out of your hangar. Most likely in this case it's one of our charter elements, which is that there's an asynchronous call waiting for a callback that never happens. There should be a timeout so that it can be retried and complete. In this case, it gets stuck there.
It can also be validation. For some reason, the reposition positions that were created are no longer valid. Because I think one of the rules is you will not be repositioned until we know there's floor underneath you. So until there's floor.
TL;DRThey want to fix players losing their persistent hangar, which they suspect is stowed in another inventory, and are working on login issues like 64008 errors caused by location resolution being handed to dead or rebooting servers.
Area 18. The hangar is actually there, but I never spawned it, in my case. But in these cases, with most players I've discussed with, the hangar is actually stowed in another inventory somewhere else for some reason. So that's the hunch so far. We'll see what the team actually comes up with, but that's one that we definitely want to solve. Now, I've got more on the hangar side, but I figure in the interest of time I would keep going with the other categories. We've got login flow related issues. So we talked a bit about shard lock. That's one of them. So basically being a bit more communicative about shard lock, and no longer having shard lock at all for players that stay in the shard. 64008 spawn resolution error and its little brother, 64006 disconnection. Again, resolve player location. I can speak to this one a little bit. This is typically a symptom. If we fix this issue specifically, the display will change. But what actually happens is that when you connect to a game shard, the replication layer, so the hybrid that's in front of it, is going to coordinate bringing your character into the game world. But it doesn't know at the start which of the 10 servers it's running your character should be on. And so it needs one of those servers to help resolve your location, so that we can unstow you in the right area in the tree and then hand off authority to the server that's managing this region. The way this is done is that the replication layer will hand this work off to any of the game servers on the shard. Now, when we have Idrises blowing up servers left and right, there's a very high chance that the server this is handed to is dead, or is in the process of dying and is rebooting. And so what happens in this case is that the player will get to the end of the context establisher tasks, the CETs, and he will get a 64008 or 12 error. We need to make this a bit more robust on that side, so that we know which servers are completely alive, and if a server is doing a reboot, we don't assign them a resolution task. That would be one thing. But also trying to streamline that so that it's a bit more efficient. A small one I wanted to touch on here was "friends list showing 0 out of 1 instead of the player counts per shard." So we're rejigging this bit. It's going to come back. This is a temporary thing.
TL;DRDocking is being worked on, including a docking-collar fix that was resolved but reopened after failing on the network, plus collision loss that ejects players and cargo services not answering hails.
Correct. Let's talk about docking. The very first one: "unable to dock using docking collars." That is problematic, isn't it? We looked at this this morning and it was actually resolved when we were talking. It is now reopened. So we're going to have to dive into that. But basically, docking is a subject that's always coming back. And this patch has also been contributing to the overpopulation of ships, because players would just basically land on pads and leave the ships there, where there's no impound. And so that's a big issue. So the first one is unable to dock using docking collars.
So I can quickly just go on: when you say it's resolved and then reopened, what happens there is the developers, to the best of their abilities, found the issue, they've fixed the issue, they've tested the issue locally, and then gone, it's solid. I know how to cause the error, I put a fix in, I validated it, I've resolved it and solved the issue. But obviously, in this case, where it sounds like the feature is not working at all, clearly there is a situation that either they didn't predict or that doesn't happen on a local machine. Again, we talked about that.
That said, I will say that we have a very, very impressive setup for locally running what is a replica of what we play in live, which is amazing. It's an amazing system.
"After manually docking a vehicle, there is a chance for the docking bridge to lose collision, causing the actor to start to EVA." Most likely close to another issue we've seen on the command module about managing physics skip lists, which is how we skip elements from collision in certain cases from a given perspective.
You don't want the docking collar to collide with your ship and cause lots of problems there, right? There's a period in time when the two are trying to get together.
"Cargo services will not respond to hails, making players unable to request cargo loading zone," related to ATC and docking in this case. That's another one we're looking at.
TL;DRFreight elevator issues are being looked at, including missions with no cargo to collect and a generic overload error that is hard to fix without more granularity, plus a cleanup bug that can lose cargo.
Freight elevators, let's skip to that. So "hauling missions have no cargo to pick up at the collect location." So this is basically when you get to your mission location, pick up this Adonite from this location, and you get there and there's nothing to pick up.
Yes. That's a bridge between missions and freight elevators, which is interesting, and which requires specific people in this case. "Freight elevator is not working and displaying an overload error, preventing cargo pickup." Do you want to speak about this one? Because I think we talked about this today.
Well, I think we talked about the overload error. It's one of those that's a little bit too generic, that can imply many different things. Similar to the ASOP saying your ship is unknown. It becomes a bit of a catch-all.
People pull up and it looks like a small box somewhere, but they're getting an overload error. The system thinks the small box is a much bigger box kind of thing.
And like Philip said in the letter you so eloquently read out, obviously we fixed tons and tons of bugs on freight elevators as part of a long-running initiative. And we've probably fixed lots of overload issues as part of that. But because overload is a catch-all, unless you get more granularity...
Well, it's a variation of that really, because the freight elevator looks at the volume and will make sure that it's got stuff that you own in it and that there's no overload on it. So that's one of the things.
That's an example where there's another bug further down, which is "you have a chance to lose your cargo by activating the freight elevator kiosk." Which is basically a malfunction of that cleanup to prevent the overload. So that needs to be solved.