TL;DRDirect hand-offs between players are technically possible, but fidelity, the risk of others grabbing or stealing items, and loss of safety are concerns, and whether to build it is a design and prioritization decision.
So that actually brings me into my first point, which is fidelity. In Star Citizen, it's all about fidelity. We want everything to be immersive and keep you there. And two players performing together is always quite challenging. The takedown becomes a little bit easier because it's a... less peaceful.
Exactly. And body carry or body dragging tends to be more, you know, you're already out. One of the persons is in a position where we know kind of what they're going to be doing. We're obviously two players goofing around doing all these things and one is trying to give something to another. But I think the other point is obviously the security of it: if I hand something over, if I want Benoit to have it, I don't want Carl to come running in and grabbing it from me. Or a system like that opening up an avenue for people to take things unfairly from other players. Those are the main considerations, I would say: the safety and security of them, and knowing that you're not going to lose your stuff.
So I'd like to bring it back to the question: why don't we already have it? Well, it's technically possible. There's a bunch of challenges to resolve, like we've just talked about, but ultimately it's a design question. It's a prioritization question. Do we want to do this over doing something else? Do we want this in the game? Those are questions for our creative director, game director. And it'll be in their minds, I'm certain. And then when it comes up, we'll have solutions ready if we want to do it.
Well, I can say this: in general, this is part of the discussions that we've had on the UI, specifically for chat during your gameplay, that if you're downed, you should be able to chat. And so I would say that yes, if you follow that decision, then if you're on the respawn screen, you should be able to chat as well.
TL;DRGround vehicles are planned for the freight elevator, but only after decoupling it from ATC, sharing the insurance flow with the kiosk, and making freight elevators more reliable first; no date given.
Oh yeah, absolutely. What I want to do before we bring ground vehicles up from the freight elevator is I want to decouple from ATC. I shouldn't need permission from the air traffic controller to bring my ground vehicle up. I'm in my hangar. I'm in my own little safe space. I can do what I want.
They should care about what happens when I leave the hangar, not what I do in the hangar. So I definitely want to do that. And then the other part is the insurance handling flow. Ground vehicles are vehicles, and vehicles have a slightly different insurance flow than other items. So that means we need to make sure that that insurance flow is available to the freight elevator, for freight to be able to do the same thing. I wouldn't want to go to the ASOP terminal to deal with insurance on my vehicle, to then go to a different terminal to bring it back up. So those are the two technical hurdles for me that I would like to see solved; I think that will make it a more robust feature.
Yeah, this ties very directly to our first question. Ground vehicles coming up from freight elevators, I think that's appeared in our internal roadmap, but we're not going to do that until we feel comfortable that we have really, is "robustified" a word? It is now. Robustified freight elevators. We don't want to build a feature on top of something that has reliability issues. And we don't want to exacerbate that by looping it into more gameplay loops. So we want to instead invest the time fixing up freight elevators like we just described before, and then layer on top.
I think also the point about decoupling from ATC is an important point. The air traffic controller in the game has evolved into also the manager of your insurance claims or the destroyed state of ships or things like that. And we don't want to keep that logic in this area. We want to be able to use that elsewhere properly. And so we just don't want to build on top of the current ATC to support the freight elevator with ground vehicles. We really want to decouple that. So Jens, you mentioned something just earlier about how insurance is different for ground vehicles. What did you mean by that, really?
It's just purely a technical aspect. So it's not that vehicles have a special type of insurance. It's for us on the engineering side: the flow itself is different and we just need to move it out so it becomes available for the freight elevator kiosk.
TL;DRA long desync post is nearly finished, and a first Tech Preview change to rigid body syncing looked promising; more targeted changes are planned, and ships and vehicles aren't covered yet.
Well, when we reviewed the questions, we saw a lot of questions related to desync. Recently, we have done some testing on our Tech Preview for some desync changes. This is kind of a result of the work that we've been doing in writing the desync post that I got coerced into writing. And so that is still ongoing. I know this is the underlying question. There were also some notes about this. And so we are about done with writing this desync post, which has a lot of information trying to explain where desync comes from. It's big. It's 45 pages.
And so one of the things with this is that we didn't want to just come out with some additional post or information without some level of action as well. And so we've been looking at improving some of the netsync elements. And so what we've been testing recently was a targeted change to how we simulate rigid body entities with the physics engine.
Well, as a first drop, it was great. That's why we wanted to move it to Tech Preview rapidly, because we wanted to gauge the actual impact of this change, which seems to be positive. Now, there's a couple of fallouts from it that we're going to fix, but that's an example of how we start to look at more of the desync from different angles. Obviously, our post will go into details about all the sources. Because unfortunately, it's a bit like the flu: there's a symptom and then there's the cause. And so there's many causes for desync.
Yes, correct. And so we'll talk about that in the deeper post. Plus, I'm hoping that some of these changes are going to yield good results, which I think this first Tech Preview at least showed us some promise of. We're going to do more of those targeted changes so that we can measure that and see the impact on players. That's really what we wanted to navigate: when we see improvements locally, we see improvements with our development setup, internal testing, there's nothing like bringing it to you guys to see if there is an actual effect when you're trying to do something with it, whether it's a Jenga tower or, in this case, since we're doing rigid bodies, that was really the test. And so that change was really targeted at how we synchronize these objects, and especially revisiting a decade-old system that's been in the engine forever. And so I'm happy with the first pass result. I think this is really promising. And then if that scheme does hold up, then we're going to roll it out to more elements in the game. Currently, for example, ships and vehicles are not targeted by that. There's also a bunch of different types of entities that are not targeted by that. So there's a lot more things to roll out there. But I think the approach is looking sound, so I'm happy about it. That's what's coming down the pipe: I think the desync post will answer a lot of questions, I'm hoping. And then we're going to do more Tech Previews for these tech changes.
I mean, it's really cool to see these changes coming in. It's really exciting. I think it's the start of some really, really great changes. So the team's done awesome on that.
TL;DRSelling or deleting earned ships is technically possible with the new entitlement and economy systems, but whether to do it is ultimately a design and creative question.
It's ultimately a design question. There's a big design and creative element to this as a feature. But we did dig out some interesting information to talk about from the tech perspective.
The answer is yes, we can do it. And I can add some color to some of the elements here. So in 4.8, there was a major change to the game, the back end specifically, where we introduced a new entitlement system. Now, for those who don't know, the entitlement system tracks what you own. This used to only track your actual platform entitlements, which are either elements you've received through Subscriber flair or actual real-money purchases or things like that. Now, since 4.8, the system tracks everything that you own, whether you bought it in an in-game shop or through any other acquisition method except stealing. That's the only one that's not covered by this.
You're essentially the legal owner of this. You've been identified as having legal rights to this item. So when we go back to the give, when you give it to another player, you would legally give them the rights. So we'd have to tell the entitlement service about that.
Correct. And the entitlement system also tracks your insurance loadouts. And so through this, the ability to sell a deed like this would involve the entitlement system, and that system can do it. Obviously, we already have wallets for transactions and other things like that. Maybe some color commentary here: there's also active work in consolidating a lot of the economy-adjacent systems together. So we have the wallet system, which is called the ledger service, that actually drives all of your monetary currencies. And there's all different systems that attach to that. So we're bringing all those into a single economy system, which will include, at some point, marketplace features, reactive scenario features and other elements of the economy. If we did build a system where you could sell an existing ship, then those systems would be involved in making the transaction. We already have the ability to do that right now.
Yeah, so deleting a ship could actually be a zero-cost transaction and maybe even be recoverable in that case, in case you've done it in a moment of weakness.
Yeah, exactly. And now the advantage of the new system too is that it persists cross-patch. So your entitlements are completely persistent cross-patch. We don't recover them from the previous patch. They are just there. And so that's why we no longer have issues like, oh, I lost my ships between patches, or things like that. Those are completely persisted by the entitlement system now.
TL;DRServer Meshing improved server frame rate, but NPCs still react slowly when servers underperform; upgraded AI behaviors finished for Squadron 42 can't come to Star Citizen right away, and more server work is needed.
So Server Meshing has been amazing. I think it's definitely delivered on what we needed from the technology. And I think it has lifted server frame rate, which I would hope means people in general have a better experience. But there's still a lot more we want to do to get server frame rate up, which will help NPCs, because in general NPCs can be perceived as slow to react, because they need a little bit of time to cook. They need a few frames from the server to get all the decisions made. So if you are unfortunately on a server that isn't performing at the rate we want, then it could be a bit slower.
There's also a condition we know of that we need to solve, which is related to large areas streaming in at once, which can cause server stalls, like we've seen with Siege, and that's something that we're tracing. Solutions are on the way to resolve that. It's a lot of deep engine work here, because we need to parallelize a phase of the initialization of entities.
We should be cautious of attributing every AI issue to server performance. Sometimes we do just see genuine issues or glitches or bugs and things like that. We just work through them, we fix them. There's a bunch of good work been done recently, right?
Well, that's what I was going to point out: we've been developing upgraded behaviors for our AI. Obviously, we've been tuning this work in Squadron 42. At this point, that work is completely signed off, and we're very happy with those new behaviors. This touches on many elements and many systems, which is why we can't bring it back right away. But we're talking about balancing the perception system and vision system so that the AIs, the NPCs, are a lot smarter about picking their targets. The ticket system tuning: this is something that in Siege of Orison we've actually seen, because this is one of the instances or areas that we've had the most active AIs in. And you could see that there are load balancing systems in place when many AIs are in the same area, where the server is going to give a certain amount of budget to each AI. And that's tuned for the current game. Siege of Orison had 10 times more AIs in general. So you could see how the load balancing wasn't perfect. This we're tuning on the SC side, but there's been massive work on this. We have a lot of changes related to how the tactical queries actually operate, which is how AIs actually plan their work. So there's quite a bit of work that has been done and completed there that really changes the behaviors in general. SC still uses an older set of roles for AIs in general: pusher, defender and strafer are the major three archetypes that we use on SC. We have a lot more roles now that are available in this new version of the AI subsystems that we need to bring back, including things like using cover, relocating between cover properly.
And so that's better tuned for a large population of NPCs. There's more roles available to pick. So if there's only five pushers, then obviously if you have a population of 70 trying to engage, then you need a few more roles.
It's like a double dip of improvement as well. The more roles you have, the more varied those roles will be, the less intense some of them may be in terms of performance requirements and things like that, which helps with performance, obviously. And then, like you said, greater variety of AI behaviors and archetypes means maybe a need for fewer AI, which again will help with performance.
TL;DRBeausejour says 4.8 involved constant firefighting, including two bad check-ins that broke persistence, though the overhauled entitlement system worked well, and Lind says extra protection against such check-ins is needed.
Well, for me, if I take it from the machine room side of things, we've been firefighting since we launched 4.8 from a whole slew of different issues. Some of them are to be expected when we launch a patch that has this many core changes. We've touched on systems that are really central to the traversal again. So we're talking about the entitlement system, which has not changed in years at this point. And we've completely overhauled the entitlement system in 4.8 to power in part the item imprint system, but more importantly, to pull the entitlements out of the patch cycle and persist them across patches. What we call the LTP layer is not just what you have in your inventory, it's also what you own. Your ships and all of the items that you purchase in-game now have an entitlement record that holds the insurance loadouts for it, all of the customizations that you make on it. And that's hosted outside the game. Not outside the game, but outside the game patch. Which means that when we come back from a patch, there's no risk of, oh, I'm not getting my ship back. What you see in ASOP in the first patch will be exactly the same data source in the second patch. And so that risk of losing ships across patches is completely gone. So we're very happy. In fact, that bit actually went very well in 4.8, even though up to the launch, it was always last-minute things to look at. But we have been firefighting across a multitude of different issues that were tangential to that. Some of them we can touch on a little bit, because I've made some notes to talk about gameplay systems here and why they were affected by some minor infrastructure issues that turned out to be massively contributing to the perception of the patch. So I'll talk a bit about that in a bit, but basically we've been firefighting since 4.8.0. I want to touch on one thing because we just had an off-the-cuff moment about the Idris last night. But two weeks prior to that, we had another downtime patch that needed a lot of rectification. So the teams that run the online group were hard at work on this maintenance. And unfortunately, another one of those unwanted check-ins came in during that 4.8.1 cycle, where, in an attempt to fix the VGL flight suit that was missing hands and feet, another check-in came in that removed sub-components of this flight suit, but not just removed them from the flight suit, deleted them from existence from the game. And that's not something that you can do in a minor patch. And so clearly, that's two instances in this patch where transition check-ins should have been validated and prevented and went in, and then after that became an infrastructure problem. Because same issue here: if somebody was wearing the VGL flight suit and referred to these entity classes that were actually representing, I think, the nozzle and the front pack of the suit, then the game server would crash. And so we had, again, to do percussive maintenance to remove those items and fix that. And so that highlights to me a massive issue with how we validate check-ins and how we get changes in the game across persistence, until we have full forward porting compatibility on the persistence layer, which we don't have at the moment.
Live servers, you're always running, right? We're talking about forward patching. If we'd been behind closed doors, you would have done that, right? No one would have ever seen the game until you'd done that piece of technology, but that's not who we are. We're always running, and that's what makes it really, really exciting to work here as well, right? Because we're putting ourselves out there all the time. But if I look at this specific case, you're saying, right, we have checks and we have contracts on what you can and can't do in these sub-streams that we've already released, where the data management is super, super strict. And obviously now we invalidated them twice in a row. So it's time to look at how we really add that extra layer of protection.
TL;DRBeausejour agrees the weapon pricing and the crafting breakdown were validated separately with no check across them, and says human validation is no longer enough, so machine validation is needed for everything.
Transversal to check all of that. And also, we're definitely now, post 4.7, in a realm where just human validation is not enough, right? And so a lot of game design data is typically managed by the team, but I think that we're way past that and we need machine validation for everything, because the rate at which we build it and the amount of technology we're putting in is just too fast for validation to keep up.
TL;DRJens Lind says the problem came from the 4.8.1 database combined with 4.8.2 data, an environment QA doesn't have, so hiring many more QA people wouldn't have caught it.
In this case, it took the situation of how the database on 4.8.1 was, with the new data from 4.8.2, to create the situation. And that's not something that the QA environment has. So whatever we do, we as a technology group would have to come in and at least give them that. And then if you put your 500 QA on it, then probably one of them would have spawned an Idris. But without that environment, you could put as many as you like on it. No one would ever have discovered this situation. This is just too far removed. And I think that sort of comes back to the large dataset, right? As you have a large dataset, you just can't scale in manpower alone. You need something that scales way beyond that.
TL;DRBeausejour says the cause isn't pinned down yet, but some server meshes run below the target frame rate, and the team is tracking six to eight optimizations, including lock contention in the area component, for the hotfix channel.
Well, so what we know about right now is that there are some meshes, and when I say meshes, I mean a dedicated game server inside a shard, that are running way below the FPS spec that we want. And so we don't have a handle just yet on what is actually causing it. We do have performance captures, and we're tracking at least six to eight very specific optimizations that need to happen. One of them is within the area component, which is used all over the engine. Area and zone component, they kind of go hand in hand. Again, because we're pushing more content, the hierarchy tree of the game world is much bigger. So there's more areas, more zones, more subdivision, more, more, more. There are some areas of that where you add a lot of entities that enter or exit these areas, and the engine provides the listeners to track entities entering areas or leaving areas or zones changing, right? And obviously there's concurrency constructs around this to prevent one thread from accessing another thread, right? And so, just keeping it simple without diving too much into the low level, which we will probably do in some of our desync areas, there's lock contention in the area component at the moment in some places. There's lock contention in some areas of the game. Well, I literally mean area: if there's many entities in this spot, because of the way it's configured, we're seeing basically FPS drops in those areas. And so we're tracking, like I said, about six to eight major optimizations that we're trying to get in our hotfix channel, to try to get this FPS back up to the spec that we want, which is above 20, across the...
No, no, no. And it's important to note that when a server is running low FPS, it doesn't mean that it will create desync, but especially when you're interacting with entities that are simulated on that DGS, AI characters, NPCs, then you're going to perceive these things, because the source is slow at creating its output. The traffic goes from your game client to the replication node, replication node to game server. Now, if the game server is running slow: at the beginning of the frame, he's taking the state from the network to the game and applying it. At the end of it, he's applying the changes back to the network. If that loop is slow, then your client's going to predict something, and then that prediction will get corrected the next frame, and that will create a rubber band, right? And so that's the gist of why there's desync in general. Why does the client have to predict? Because if he doesn't predict, then you're going to be at the tick of the server, and you'll see...
TL;DRBeausejour says ship cleanup is one thing being looked at: the system seems to work but may be misconfigured in some areas, letting ships pile up, and more entities make servers do more work.
That's one of the things that we're looking at, how aggressive or not the ship cleanup is actually happening. That's one of the things we're looking at, yes.
Because you can watch a Twitch stream or look at an Issue Council report and you'll see dozens and dozens, I feel like it's even more, of bricked ships everywhere, with the insurance changes bricking these ships. Is this viably part of what it is?
In part, yeah. That for sure doesn't help, because the more entities are replicated, the more the server needs to do every frame, and so that contributes to this. So, yes.
No, a server not performing so great tends to have a slightly larger vehicle population or a very large active population. That can also happen. In general, it's just a lot of everything. We're quite a heavy game, entity-wise. We want to avoid getting to the sort of size that we were in Stanton before we went into meshing, right? Because that was consistently difficult to manage.
I know that there's been some changes related to how we perform entity cleanup. So unattended vehicle cleanup used to be a separate system that ran on its own. That's been merged into what we call the density manager, which is basically an engine functionality that will look at overpopulation in clusters of areas that designers obviously have control over. For example, we want to make sure the exit routes for hangars are cleaned up. And so these have active cleanup zones configured, right? That's designer controlled. Now the engine will then apply the logic on the volume, determine, oh, there's too many of X in this area, and will start popping stuff out, right? Basically going to do the cleanup. Now for ships, the unattended vehicle cleanup system used to be separate. Now it's been brought into the density manager, I think end of last year, something like this. So it's been a while. And this one tracks whether there's player passengers on a vehicle and will, based on a certain timer configuration, and the values are different depending on the areas, either stow the ship back into the originating inventory or destroy it, depending on if it's a player-owned ship or if it's an AI ship. It does look like maybe there's a misconfiguration in key areas that are leaving ships there. So we don't think the system is incorrectly functioning. We think it might be inaccurately configured in some areas. And so that would allow a very popular area to just accumulate ships like crazy. And so that's what we're looking at right now.
TL;DRThe speakers are unsure the density manager has a specific rule for bricked ships, but say it will eventually, and that it works on how crowded an area is more than on timers.
It will. If it doesn't, it will eventually. And it's less about timers. It's more about density. It's trying to say you have overpopulated this area. So it's less about that ship having been there for a long period of time. It's more like, there's a lot of ships here. Let's remove one. Let's figure out which one is best to remove. Unattended vehicle does have a timer as well. So it's sort of like we talked about with validation. You start with a timer, then you start adding density, then you start building up these views of the world to understand which areas are problematic. Like, oh yeah, the entrance to a hangar, we have to be super strict.
There's a place, as we're talking about this, something that pops into my mind that I think is also in our process. Not even the development process, but let's call it live operations of the game. We're very lean in terms of our live operations at CIG, because we're in development. And so even though we do run a live service, we try to run a really lean ship on that side. And so we don't have an army of site reliability engineers that are looking at every aspect of, oh, shard 25 has an overpopulation of ships at Jackson Swap between this time frame and this time frame. We don't go to that depth. And so we track overpopulation across all shards, but we can't go granular into this. But that is an area where we could improve this, detect those things and better tune these values. Typically right now we will tune the density manager values every patch or every hotfix, but that is a slow cadence compared to how you would react if you see, oh my god, on every shard this location is loaded, let's effect the density manager change right now. Right now we don't have the ability to do that live, and that's a problem. That would improve the game quite a bit.
I always say, if it was easy, it would have been done. So by that, you have to assume there are reasons that make it not easy. And it's usually the complexity in the interactions, like Benoit talked about: you sell a gun and then you deconstruct it, and then the material can be sold somewhere else for something else. These kinds of complex interactions between lots and lots of different systems are usually what stops us from doing something that, from a tech point of view, is literally a flag in the database that says that's bricked, I can treat it as garbage.
TL;DRBeausejour suspects behavior changes tied to the Imprint and insurance changes, such as more ships left behind or players favoring loosely configured locations, while Lind can't point to any other gameplay change that explains it.
For me, it's behaviors. So there's one thing about running a game server, which is that it's kind of an organic behavioral system, right? And so you can have a game server with a version of the code and a version of the data running a certain way, and then players start doing something different, and that completely changes the performance profile. And so I feel like in 4.8, most likely, my hunch for these six to eight performance issues that we're looking at is that it's related to some Imprint changes. So the way people use ships, leaving bricked ships behind: are there more claims than before? Are there other egregious issues that showed up that are creating more claims that would affect the amount of entities on the server? Is there a malfunction on, for example, the density manager cleanup that is leaving items in place? Or are people just going to a location that's configured to be very loose on its density, and it's somehow the location that people have decided to go to in this patch? So sometimes the performance profile of the game servers changes just based on behaviors, not on, oh, we changed something.
I mean, it's hard not to just parrot what you just said. It does feel like, when I think of 4.8, there's nothing that I can point to and go, yeah, those gameplay changes we made could have caused it. The only thing I can think of is, we did performance improvements, but there is a large change in how we handle vehicles and the lifetime of vehicles.
I mean, that would be, based on what I know of what that patch put out there. I just don't recall there being any larger changes on the gameplay side that I feel I could explain it with.
TL;DRBeausejour lists several suspected causes without going deep: navmesh tile changes on planets, an entity spawner respawning, area and zone lock overhead, ship cleanup with active MFDs and flight systems, and fire and temperature physics.
I don't want to deep dive into each of them, because it's a bit intense if we go down that route. But there's navigation system dispatching of tile configuration changes on planets, which affects our AI using NavMesh on planets. So there's a performance cost on that that is costing us a chunk of frame in the servers. This is purely based on the servers we know are affected by this problem.
Yeah, exactly. This is cross-linking a lot of things. This is a pet peeve of mine that I keep pushing on, and we're going to definitely resolve that from a software point of view: there's an entity spawner in some of the areas that are affected that is coming in and out of streaming and respawning its spawn set every time. These are the NPCs that just spout out. That's one example; I have multiple others related to this. That's one. The area component we talked about. So what we call register with external systems is a phase when a component or an entity is coming online, when it's streaming in. It needs to register with external systems. That's a notorious area. That is an issue on multiple components. This one, from the area component, is a problem in that place.
Correct. This one is more engine and, let's call it, area configuration related. This one bleeds into the zone lock that we talked about earlier. So those are two, area and zone lock. So this is really tied to the amount of content in the game and the complexity of that content in some areas. Because we're also making more and more complex scenes, like space stations and the content we've put out recently, whether it's TSG or Rockcracker, there's a lot more complex layouts, and those elements contribute a lot to this. And so those are part of that. The ship cleanup is another big one that we're looking at at the moment, which is probably the source of the other two, which is the amount of registered entities that have either MFDs and/or the flight system active.
Well, we don't know specifically. I bet that if I went into that area and I looked, there would be a bunch of MFDs that are somehow visible. So these components have a visibility aspect to them where, if you look at it, it's going to replicate for you. But if you don't look at it, it doesn't. Might be a malfunction with that. We're not sure.
It could be broken ship MFDs. We've seen weird things; everybody knows about the graveyard. There's been other graveyards in other areas, right? So there might be a new one we haven't found yet where somehow MFDs end up there.
And one final one, that is fire. So some ships catch fire. And depending on the size of the voxel grid, we've seen that the area physics properties of how the temperature system interacts with fire actually need a big optimization, and it's costing us frame time. That's a combination thing, because internally we can't reproduce it, but we see the cost on some of the game servers that are running slow.
TL;DRBeausejour and Lind say optimization and crash fixing are a fog of war, where fixing the known issues may help or may just reveal new ones, and different servers can have different problems.
Now the thing with everything in optimization is that it's a fog of war. You fix that, that's what was slowing it down, then you find the next thing. You get rid of the elephant, and behind it is another one. The same properties apply to crashes, right? Typically when we do stability checks on game servers, which, by the way, other than the two big issues that we've had, the baseline stability for 4.8 outside of these two events of the missing entity classes and the Idris problem has been in a really good range overall. And so we've been happy about this, other than now the stats are terrible because of this. So we'll have to take a look. But the same properties apply to crashes. So when we look at top crashes, you're like, oh, we just have to fix this one. It's super high frequency. Then we fix it. And then, oh my god, there's three more. And so it's a fog of war. Optimization and crash stability are fog-of-war activities where, yeah, you've got seven now. You know what those are. We fix them, it might be the end of it, it might not be.
You can get clusters as well, and that's the same on the performance side. We might be looking very, very closely at the servers performing poorly, and we might be fixing things that help those, but that doesn't help at all with the other servers, because they might be running into completely different issues. There might be something that's happening specifically on those servers that's unique to them, that brings them to that low rate, whereas the rest of them don't benefit as much from that. So you always have to look closely at the details and react to those, but proactively try to look at the big picture and where we are going overall. Once we fix that cluster there, how do we get all of it to benefit from a smooth ride with zero desync?
TL;DRThe team is working on hangar bugs such as exterior doors staying closed, players being told to enter someone else's hangar, and repositioning getting stuck after impound, mostly from failed handshakes, missing timeouts or invalid reposition positions.
So hangars. I'm going to start with hangars. Now, I'm going to pick the ones that I think are interesting in the list, in each segment. These aren't in any particular order. This is just some of what people are working on today. So: PU, multi-locations, instanced hangars, "exterior hangar doors remain closed while interior hangar door opens." So that's when your hangar doors open, but the gateway outside is still locked.
So they belong to two different parts, right? You have the exterior and you have the interior. The interior is yours. You can control that in your instance. The exterior needs to be synchronized with your interior door. So what most likely happens there is, for some reason, your interior thinks you've been given the right to this gateway. Or you could have been given a right to a different gateway, and there's an exterior door opening somewhere to nothing, and your door is opening to a closed door. So there's a few things that could happen there, and normally it's that handshake between the exterior and the interior, via the notorious ATC.
But this one goes with its little brother, overlapping hangars: "Player reposition when landing in hangar: ATC given, but seen as trespassing in someone else's hangar, results in being repositioned." Which I think is a sibling.
Yeah, definitely, because you're looking at a situation where you think your interior has been given a certain gateway, you're heading there, but it's actually someone else's casa. So you're correctly trespassing, but you've obviously incorrectly been told that it was yours.
And then the third of the trifecta in this segment, if you put them all together, is "players can get stuck at repositioning player indefinitely after getting impounded." And so if you get impounded, there's a chance that the server will never complete the repositioning that's supposed to take you out of your hangar. Most likely in this case it's one of our charter elements, which is that there's an asynchronous call waiting for a callback that never happens. There should be a timeout so that it can be retried and complete. In this case, it gets stuck there.
It can also be validation. For some reason, the reposition positions that were created are no longer valid. Because I think one of the rules is you will not be repositioned until we know there's floor underneath you. So until there's floor.
That is definitely related to this. It sounds like his repositioning position wasn't amazing in that case. It also does disprove my floor, because he clearly did not have floor.
TL;DRDocking is being worked on, including a docking-collar fix that was resolved but reopened after failing on the network, plus collision loss that ejects players and cargo services not answering hails.
Correct. Let's talk about docking. The very first one: "unable to dock using docking collars." That is problematic, isn't it? We looked at this this morning and it was actually resolved when we were talking. It is now reopened. So we're going to have to dive into that. But basically, docking is a subject that's always coming back. And this patch has also been contributing to the overpopulation of ships, because players would just basically land on pads and leave the ships there, where there's no impound. And so that's a big issue. So the first one is unable to dock using docking collars.
So I can quickly just go on: when you say it's resolved and then reopened, what happens there is the developers, to the best of their abilities, found the issue, they've fixed the issue, they've tested the issue locally, and then gone, it's solid. I know how to cause the error, I put a fix in, I validated it, I've resolved it and solved the issue. But obviously, in this case, where it sounds like the feature is not working at all, clearly there is a situation that either they didn't predict or that doesn't happen on a local machine. Again, we talked about that.
That said, I will say that we have a very, very impressive setup for locally running what is a replica of what we play in live, which is amazing. It's an amazing system.
"After manually docking a vehicle, there is a chance for the docking bridge to lose collision, causing the actor to start to EVA." Most likely close to another issue we've seen on the command module about managing physics skip lists, which is how we skip elements from collision in certain cases from a given perspective.
You don't want the docking collar to collide with your ship and cause lots of problems there, right? There's a period in time when the two are trying to get together.
"Cargo services will not respond to hails, making players unable to request cargo loading zone," related to ATC and docking in this case. That's another one we're looking at.
TL;DRFreight elevator issues are being looked at, including missions with no cargo to collect and a generic overload error that is hard to fix without more granularity, plus a cleanup bug that can lose cargo.
Freight elevators, let's skip to that. So "hauling missions have no cargo to pick up at the collect location." So this is basically when you get to your mission location, pick up this Adonite from this location, and you get there and there's nothing to pick up.
Yes. That's a bridge between missions and freight elevators, which is interesting, and which requires specific people in this case. "Freight elevator is not working and displaying an overload error, preventing cargo pickup." Do you want to speak about this one? Because I think we talked about this today.
Well, I think we talked about the overload error. It's one of those that's a little bit too generic, that can imply many different things. Similar to the ASOP saying your ship is unknown. It becomes a bit of a catch-all.
People pull up and it looks like a small box somewhere, but they're getting an overload error. The system thinks the small box is a much bigger box kind of thing.
And like Philip said in the letter you so eloquently read out, obviously we fixed tons and tons of bugs on freight elevators as part of a long-running initiative. And we've probably fixed lots of overload issues as part of that. But because overload is a catch-all, unless you get more granularity...
Well, it's a variation of that really, because the freight elevator looks at the volume and will make sure that it's got stuff that you own in it and that there's no overload on it. So that's one of the things.
That's an example where there's another bug further down, which is "you have a chance to lose your cargo by activating the freight elevator kiosk." Which is basically a malfunction of that cleanup to prevent the overload. So that needs to be solved.