TL;DRRegressions are being tackled through deeper root cause analysis, larger test plans run before submitting changes rather than after, and scaled-up validation tools and automated testing, which they hope will raise stability.
For me, there's three main ways that we're tackling this. Number one, root cause analysis. What is that? So when you're faced with a bug, you can diagnose it sort of surface level, find out what's going wrong and fix it. But unless you follow that through and you go down the rabbit hole and find out what the ultimate root cause of what is essentially a symptom is, there's a very good chance that that is going to come back. So some good examples would be freight elevators, hangars, ASOP, ATC. Freight elevators especially, we've had a decent amount of time of people on our teams really digging into this at a core level, and hopefully people in the community are starting to see the improvements from that.
I mean, that's the thing: root cause analysis changes how we assign our resources when they're debugging things, so that they stay longer on the issue. They go a lot deeper and they document the process before taking action. Then we can identify in that process regressions that have caused the problems to begin with. And then we can also build a comprehensive fix plan, which then feeds into the second point.
Validate the fix in multiple configurations. We typically used to do these tests, but deferred. Now we're trying to do them pre-submit. And so what that means is that we're doing multiplayer testing with more players pre-submit. We're doing server recovery testing pre-submit. We're covering the happy and the destructive path for each feature pre-submit. We used to do that post-submit.
And that leads into the third point that we mentioned, which is just massively scaling up our suite of tools and frameworks and functionalities for validation. So whether that's in content creation tools or scaling up our use of our automated testing framework.
It's about expanding this testing so that we can simulate live environments basically as much as we can. And so validating the content allows us to make sure that our content and code that are linked actually work properly before submit. But then after that, it's about validating that when you put a lot of players on it, it actually functions the way you would expect it to function.
And I think all of that combined can really give us the confidence that this approach is going to work, and the stability and the robustness of our game, its features, its systems, its content, the waterline of that is just going to continue to rise.
TL;DRMost of the UI already uses Building Blocks, but legacy visor elements plus the Comlink and vehicle manager apps still need replacing before Scaleform can go, and font rendering will move at some point.
Well, I can speak to that. Star Citizen currently uses a majority of Building Blocks, actually. There are major elements that we still need to be able to completely remove Scaleform from the game. There's still some legacy elements tied to the visor, which need to be ported over and transformed into Building Blocks. And there's two major applications that are left: the Comlink app, and the next one is the VMA, which is the vehicle manager, which is being replaced by the vehicle loadout system. So both of these are actively being worked on to replace, and this will allow us to completely remove Scaleform from the game. Removing one of the render passes on the UI on the visor will really help performance. And more importantly, it standardizes our UI framework in one, which helps the UI team go a bit faster in some elements. And so out of this, currently Star Citizen still uses Scaleform, even in the Building Blocks elements, for font rendering, but that's been completely replaced and will roll out to Star Citizen at some point.
TL;DRSocial tools are progressing, though slower than wanted because resources were diverted; back-end work is on track, while in-game implementation (chat revamp, Social App) is still about to start.
Yes, social tools are progressing. While it's not going as fast as we would like, because we did have to divert resources to the efforts we talked about at the beginning of the show, there's still a lot of work actually being done on social tools in general. So back-end work has been moving along quite a bit, because there's a lot of back-end changes to be able to provide that. Social tools are inherently real time, and so there's a lot of communication messages, asynchronous messages that need to go out, and so that back end needs to support that really well. That's actually on track and is going really well. What we are still planning to do here, and where we're slowed down, is implementation in the game itself. Our first drop for social tools that we have planned is a revamp of the chat systems, which includes regional chat or location-based chat, meaning that we split the main chat into multiple depending on the location that you are at. This is in preparation for larger player count support per shard, so that we don't have a massive chat at the top of the list there. More presence, rich presence updates, the chat widget, a complete revamp of the chat widget, and a new application called the Social App that replaces the Comlink app, which, as we discussed, is an old Scaleform element that we want to get rid of.
Absolutely. The chat widget is a very complicated piece of UI. Even though it's simple in its appearance, the functionality behind it is very in-depth. We want you to be able to have different channels that you can manage, manage tabs the way you want, maybe have a bit more flexibility on where the UI is positioned, how it works, can you expand it? So we have a lot of functionality for that that's being done. The design is completed, implementation on the game side is about to start, UI is already on it, so it's progressing really well. That said, there are still features being worked on in the current version of the game. So there are improvements to VoIP and FoIP that are coming into every patch.
And your face, in our case, yes. So far, I think since 4.8, 4.9 and 4.10, there have been incremental changes to the VoIP back end, which transmits all of this. In 4.10 too, actually, there are changes to the synchronization code that synchronizes the face across the network, which is a lot more robust. And so unfortunately, for this feature, one of the big things we think is holding it up is supporting the UI in the game itself. Receiving calls right now is a bit hard to find; you don't get proper notifications for it. And so obviously, to be able to really feel these improvements, you need the UI changes. It's actually moving forward and we're trying to bring any improvements that we have into the game right now.
Well, I can say this: in general, this is part of the discussions that we've had on the UI, specifically for chat during your gameplay, that if you're downed, you should be able to chat. And so I would say that yes, if you follow that decision, then if you're on the respawn screen, you should be able to chat as well.
TL;DRLAMP was limited to fighters because static canopies have a linkage issue; it's solvable and wanted on every vehicle, but needs a way to avoid reauthoring many ships and still has to be planned.
So I did inquire about this, because I did not know the answer to that. So I went to the source. When we implemented LAMP, it was a limited feature that was meant to be only on fighter ships initially. And there was a reason for that. It's because in order to get the feature in the game in a timely fashion, we knew that we could get this working with the exterior canopies really easily. For example, you take a Gladius canopy; the canopy is part of the exterior of the ship. And the way LAMP actually works is there's a submaterial on the inside of the canopy itself, so that if you look at it from that direction and LAMP is turned on, then the visibility is changed and the effects come into play. Now, the problem is for static canopies, which are part of the structure of the ship, there is a linkage issue at the moment that prevents us from properly activating or finding that submaterial from code. Now, it's solvable and we will solve it. But we're trying to find a way to do that without having to reauthor 100-plus ships to be able to activate LAMP. And so this is what's keeping us from doing it now. It's a priority call that we need to make to get it in there. We do want to do it. We do want to support it. And so we need to find a way to do it without having to reauthor all the materials in the static pieces. And so most likely, we already have all the theories set up. We just need the work to be planned.
I actually checked with John, because again, there could be a creative consideration here. Do you want to have LAMP on every vehicle? Should it be only on certain archetypes? He said, nope, we want it on every vehicle, including ground vehicles. So John loves LAMP.
TL;DRGround vehicles are planned for the freight elevator, but only after decoupling it from ATC, sharing the insurance flow with the kiosk, and making freight elevators more reliable first; no date given.
Oh yeah, absolutely. What I want to do before we bring ground vehicles up from the freight elevator is I want to decouple from ATC. I shouldn't need permission from the air traffic controller to bring my ground vehicle up. I'm in my hangar. I'm in my own little safe space. I can do what I want.
They should care about what happens when I leave the hangar, not what I do in the hangar. So I definitely want to do that. And then the other part is the insurance handling flow. Ground vehicles are vehicles, and vehicles have a slightly different insurance flow than other items. So that means we need to make sure that that insurance flow is available to the freight elevator, for freight to be able to do the same thing. I wouldn't want to go to the ASOP terminal to deal with insurance on my vehicle, to then go to a different terminal to bring it back up. So those are the two technical hurdles for me that I would like to see solved; I think that will make it a more robust feature.
Yeah, this ties very directly to our first question. Ground vehicles coming up from freight elevators, I think that's appeared in our internal roadmap, but we're not going to do that until we feel comfortable that we have really, is "robustified" a word? It is now. Robustified freight elevators. We don't want to build a feature on top of something that has reliability issues. And we don't want to exacerbate that by looping it into more gameplay loops. So we want to instead invest the time fixing up freight elevators like we just described before, and then layer on top.
I think also the point about decoupling from ATC is an important point. The air traffic controller in the game has evolved into also the manager of your insurance claims or the destroyed state of ships or things like that. And we don't want to keep that logic in this area. We want to be able to use that elsewhere properly. And so we just don't want to build on top of the current ATC to support the freight elevator with ground vehicles. We really want to decouple that. So Jens, you mentioned something just earlier about how insurance is different for ground vehicles. What did you mean by that, really?
It's just purely a technical aspect. So it's not that vehicles have a special type of insurance. It's for us on the engineering side: the flow itself is different and we just need to move it out so it becomes available for the freight elevator kiosk.
TL;DRA long desync post is nearly finished, and a first Tech Preview change to rigid body syncing looked promising; more targeted changes are planned, and ships and vehicles aren't covered yet.
Well, when we reviewed the questions, we saw a lot of questions related to desync. Recently, we have done some testing on our Tech Preview for some desync changes. This is kind of a result of the work that we've been doing in writing the desync post that I got coerced into writing. And so that is still ongoing. I know this is the underlying question. There were also some notes about this. And so we are about done with writing this desync post, which has a lot of information trying to explain where desync comes from. It's big. It's 45 pages.
And so one of the things with this is that we didn't want to just come out with some additional post or information without some level of action as well. And so we've been looking at improving some of the netsync elements. And so what we've been testing recently was a targeted change to how we simulate rigid body entities with the physics engine.
Well, as a first drop, it was great. That's why we wanted to move it to Tech Preview rapidly, because we wanted to gauge the actual impact of this change, which seems to be positive. Now, there's a couple of fallouts from it that we're going to fix, but that's an example of how we start to look at more of the desync from different angles. Obviously, our post will go into details about all the sources. Because unfortunately, it's a bit like the flu: there's a symptom and then there's the cause. And so there's many causes for desync.
Yes, correct. And so we'll talk about that in the deeper post. Plus, I'm hoping that some of these changes are going to yield good results, which I think this first Tech Preview at least showed us some promise of. We're going to do more of those targeted changes so that we can measure that and see the impact on players. That's really what we wanted to navigate: when we see improvements locally, we see improvements with our development setup, internal testing, there's nothing like bringing it to you guys to see if there is an actual effect when you're trying to do something with it, whether it's a Jenga tower or, in this case, since we're doing rigid bodies, that was really the test. And so that change was really targeted at how we synchronize these objects, and especially revisiting a decade-old system that's been in the engine forever. And so I'm happy with the first pass result. I think this is really promising. And then if that scheme does hold up, then we're going to roll it out to more elements in the game. Currently, for example, ships and vehicles are not targeted by that. There's also a bunch of different types of entities that are not targeted by that. So there's a lot more things to roll out there. But I think the approach is looking sound, so I'm happy about it. That's what's coming down the pipe: I think the desync post will answer a lot of questions, I'm hoping. And then we're going to do more Tech Previews for these tech changes.
I mean, it's really cool to see these changes coming in. It's really exciting. I think it's the start of some really, really great changes. So the team's done awesome on that.
TL;DRSelling or deleting earned ships is technically possible with the new entitlement and economy systems, but whether to do it is ultimately a design and creative question.
It's ultimately a design question. There's a big design and creative element to this as a feature. But we did dig out some interesting information to talk about from the tech perspective.
The answer is yes, we can do it. And I can add some color to some of the elements here. So in 4.8, there was a major change to the game, the back end specifically, where we introduced a new entitlement system. Now, for those who don't know, the entitlement system tracks what you own. This used to only track your actual platform entitlements, which are either elements you've received through Subscriber flair or actual real-money purchases or things like that. Now, since 4.8, the system tracks everything that you own, whether you bought it in an in-game shop or through any other acquisition method except stealing. That's the only one that's not covered by this.
You're essentially the legal owner of this. You've been identified as having legal rights to this item. So when we go back to the give, when you give it to another player, you would legally give them the rights. So we'd have to tell the entitlement service about that.
Correct. And the entitlement system also tracks your insurance loadouts. And so through this, the ability to sell a deed like this would involve the entitlement system, and that system can do it. Obviously, we already have wallets for transactions and other things like that. Maybe some color commentary here: there's also active work in consolidating a lot of the economy-adjacent systems together. So we have the wallet system, which is called the ledger service, that actually drives all of your monetary currencies. And there's all different systems that attach to that. So we're bringing all those into a single economy system, which will include, at some point, marketplace features, reactive scenario features and other elements of the economy. If we did build a system where you could sell an existing ship, then those systems would be involved in making the transaction. We already have the ability to do that right now.
Yeah, so deleting a ship could actually be a zero-cost transaction and maybe even be recoverable in that case, in case you've done it in a moment of weakness.
Yeah, exactly. And now the advantage of the new system too is that it persists cross-patch. So your entitlements are completely persistent cross-patch. We don't recover them from the previous patch. They are just there. And so that's why we no longer have issues like, oh, I lost my ships between patches, or things like that. Those are completely persisted by the entitlement system now.
TL;DRServer Meshing improved server frame rate, but NPCs still react slowly when servers underperform; upgraded AI behaviors finished for Squadron 42 can't come to Star Citizen right away, and more server work is needed.
So Server Meshing has been amazing. I think it's definitely delivered on what we needed from the technology. And I think it has lifted server frame rate, which I would hope means people in general have a better experience. But there's still a lot more we want to do to get server frame rate up, which will help NPCs, because in general NPCs can be perceived as slow to react, because they need a little bit of time to cook. They need a few frames from the server to get all the decisions made. So if you are unfortunately on a server that isn't performing at the rate we want, then it could be a bit slower.
There's also a condition we know of that we need to solve, which is related to large areas streaming in at once, which can cause server stalls, like we've seen with Siege, and that's something that we're tracing. Solutions are on the way to resolve that. It's a lot of deep engine work here, because we need to parallelize a phase of the initialization of entities.
We should be cautious of attributing every AI issue to server performance. Sometimes we do just see genuine issues or glitches or bugs and things like that. We just work through them, we fix them. There's a bunch of good work been done recently, right?
Well, that's what I was going to point out: we've been developing upgraded behaviors for our AI. Obviously, we've been tuning this work in Squadron 42. At this point, that work is completely signed off, and we're very happy with those new behaviors. This touches on many elements and many systems, which is why we can't bring it back right away. But we're talking about balancing the perception system and vision system so that the AIs, the NPCs, are a lot smarter about picking their targets. The ticket system tuning: this is something that in Siege of Orison we've actually seen, because this is one of the instances or areas that we've had the most active AIs in. And you could see that there are load balancing systems in place when many AIs are in the same area, where the server is going to give a certain amount of budget to each AI. And that's tuned for the current game. Siege of Orison had 10 times more AIs in general. So you could see how the load balancing wasn't perfect. This we're tuning on the SC side, but there's been massive work on this. We have a lot of changes related to how the tactical queries actually operate, which is how AIs actually plan their work. So there's quite a bit of work that has been done and completed there that really changes the behaviors in general. SC still uses an older set of roles for AIs in general: pusher, defender and strafer are the major three archetypes that we use on SC. We have a lot more roles now that are available in this new version of the AI subsystems that we need to bring back, including things like using cover, relocating between cover properly.
And so that's better tuned for a large population of NPCs. There's more roles available to pick. So if there's only five pushers, then obviously if you have a population of 70 trying to engage, then you need a few more roles.
It's like a double dip of improvement as well. The more roles you have, the more varied those roles will be, the less intense some of them may be in terms of performance requirements and things like that, which helps with performance, obviously. And then, like you said, greater variety of AI behaviors and archetypes means maybe a need for fewer AI, which again will help with performance.
TL;DRGround vehicles are getting active and planned work: unifying the three physics setups first, then improving handling with better drivetrain, braking, engine and suspension modeling; no timing given.
So I did ask the team about this to see what the current plans are. And yes, there's a lot of planned and actually active work being done on ground vehicles in general. So to give a bit of context, there are currently three, let's call them setup rigs, for ground vehicles in the game. There's the really old Crytek vehicle system rig setup. Obviously, it's been tweaked a little bit, but that was the original one. I think the Mule is using this, for example. It's the most simple setup of all. The second one is a hybrid version, so the tanks, like the Nova, take this old physics rig but then add a little bit of control over it. And then the final one is a fully simulated version of it, which all the other vehicles are using currently. So we do have plans to unify this so that everybody uses the best version. That's the first stage. And then we want to evolve that quite a bit to give it better handling. So like tuning up the physics for rotational clamping, collision damping, things to make the experience smoother in general for gameplay inside ground vehicles when they collide with terrain, especially with V5 coming out, where this will actually help.
Yeah, Planet Tech V5 coming out in Genesis. Greater fidelity of physics proxy generation for terrain, things like that. So that is only going to be an improvement.
So that even helps with the old rigs. So you get a bit of a better feel on the ground. But there's a lot of physics upgrades that are needed for ground vehicles, and we are planning to do them. We're talking about modeling the drivetrain, braking model, engine and suspensions. We already have suspensions, but we want to improve that quite a bit. And so from a systems and design point of view, yes, the plan is to unify all ground vehicles under one setup and then improve that setup across the game.
I mean, I worked in racing games for 13 years. I know that this stuff is challenging, but we do have an excellent team, a comparable size team and composition makeup to what I would have had in something where the entire game is about racing and about driving. So with that unification and those good people, I think we'll be able to get where we need to go.
TL;DRA bug-fix check-in that shouldn't have been in the stream made Idris ships incompatible with their saved copies, so game servers crashed on spawning them until Idris spawning was paused and the ships were manually fixed.
Well, it started with an attempt at a bug fix a month ago. So with this 4.8 release and the way we've been doing patches this year, we have two streams when we work off from the working set. We have a base stream that we use for the base release, so 4.8.0. And we have a hotfix stream where we bring all the continuous changes along. So there was an attempt at bug fixing something on the Idris that was actually creating some performance issue somewhere else that was checked into that stream and nobody saw it and it went through. So we've been on the hotfix stream now for 4.8.1, now going to 4.8.2. In 4.8.2, we went back to the base stream, which are supposed to be at parity at this point, except additional content because we've added additional content in the patch. That change was included. And so what's supposed to be a routine update became a massive, wide-sprawling issue because basically the Idris was no longer compatible with its persisted copy. And so what that would do is players have in their ASOP terminal a version of the Idris that's from 4.8.1. This change, which was not supposed to be in the stream at all, gets in there. And so now the game server is expecting another shape for this. The game server has a series of checks and balances for this, but he's the last banker in the chain. And so for him, if the schema is incompatible, he's like, there's data corruption risk, I'm out.
Well, he's the game server, basically. So the game server is trying to spawn the Idris, wherever it is, right? So that's called the async spawn chain. And so as he's trying to resolve the object containers for this and to materialize the ship in game, he comes around an incompatibility. And then there's a hard check for this. Because if we let those through, if we disarm this check, for example, then the game server might corrupt ships or might be twisting data and then we would go wider than that. And so the game server would crash.
And so every time an Idris would be attempted to spawn, that server in that region would crash and then restart and then try to load the Idris again and then crash again. And so as players were unstowing more Idrises in more shards, more servers got affected, until you reach a point where now the ops team is in the background and they're lighting all the fires, calling the Rohan. And so we're looking at this and basically we start cordoning off shards, because at the beginning we're not sure what the issue is. And what happens in this case is the more shards are getting offline, the less capacity we have for players that are either entering the game or shard hopping. We have a lot of people shard hopping. And so what happens is they hit the dreaded 60012 error, which means, hey, there's no capacity in your zone. There is capacity, but we've had to restrict it in the meantime. And so we took a hard decision later in the night to prevent players from spawning the Idris temporarily, just to stop breaking other shards. We brought more shards online to keep people playing. And then we basically went through and manually rectified. That's what we call percussive maintenance. When you see that in the status, it means we went in and we did a drastic measure to basically fix the Idrises that were either spawned in shards or still left in inventories, to make sure that they had the proper schema. And so now we know that all the Idrises are fine. We've lifted that temporary stoppage for that ship. And so now we're back. That's what happened.
No, it could have been any ship in that case. So normally, there's a check-in discipline that we apply, especially this year, because we have a different publishing model this year. And so ships that have been distributed already in a given major-minor patch cannot be modified. Now, for some reason, this check-in passed all the validation checks and ended up in the build.
TL;DRBeausejour says 4.8 involved constant firefighting, including two bad check-ins that broke persistence, though the overhauled entitlement system worked well, and Lind says extra protection against such check-ins is needed.
Well, for me, if I take it from the machine room side of things, we've been firefighting since we launched 4.8 from a whole slew of different issues. Some of them are to be expected when we launch a patch that has this many core changes. We've touched on systems that are really central to the traversal again. So we're talking about the entitlement system, which has not changed in years at this point. And we've completely overhauled the entitlement system in 4.8 to power in part the item imprint system, but more importantly, to pull the entitlements out of the patch cycle and persist them across patches. What we call the LTP layer is not just what you have in your inventory, it's also what you own. Your ships and all of the items that you purchase in-game now have an entitlement record that holds the insurance loadouts for it, all of the customizations that you make on it. And that's hosted outside the game. Not outside the game, but outside the game patch. Which means that when we come back from a patch, there's no risk of, oh, I'm not getting my ship back. What you see in ASOP in the first patch will be exactly the same data source in the second patch. And so that risk of losing ships across patches is completely gone. So we're very happy. In fact, that bit actually went very well in 4.8, even though up to the launch, it was always last-minute things to look at. But we have been firefighting across a multitude of different issues that were tangential to that. Some of them we can touch on a little bit, because I've made some notes to talk about gameplay systems here and why they were affected by some minor infrastructure issues that turned out to be massively contributing to the perception of the patch. So I'll talk a bit about that in a bit, but basically we've been firefighting since 4.8.0. I want to touch on one thing because we just had an off-the-cuff moment about the Idris last night. But two weeks prior to that, we had another downtime patch that needed a lot of rectification. So the teams that run the online group were hard at work on this maintenance. And unfortunately, another one of those unwanted check-ins came in during that 4.8.1 cycle, where, in an attempt to fix the VGL flight suit that was missing hands and feet, another check-in came in that removed sub-components of this flight suit, but not just removed them from the flight suit, deleted them from existence from the game. And that's not something that you can do in a minor patch. And so clearly, that's two instances in this patch where transition check-ins should have been validated and prevented and went in, and then after that became an infrastructure problem. Because same issue here: if somebody was wearing the VGL flight suit and referred to these entity classes that were actually representing, I think, the nozzle and the front pack of the suit, then the game server would crash. And so we had, again, to do percussive maintenance to remove those items and fix that. And so that highlights to me a massive issue with how we validate check-ins and how we get changes in the game across persistence, until we have full forward porting compatibility on the persistence layer, which we don't have at the moment.
Live servers, you're always running, right? We're talking about forward patching. If we'd been behind closed doors, you would have done that, right? No one would have ever seen the game until you'd done that piece of technology, but that's not who we are. We're always running, and that's what makes it really, really exciting to work here as well, right? Because we're putting ourselves out there all the time. But if I look at this specific case, you're saying, right, we have checks and we have contracts on what you can and can't do in these sub-streams that we've already released, where the data management is super, super strict. And obviously now we invalidated them twice in a row. So it's time to look at how we really add that extra layer of protection.
TL;DRBenoit Beausejour says the game is still being turned into an MMO and content and features are outpacing the guardrails and cross-system validation, so the team needs to add stronger validation.
I think there's a part of that which is that we're still transforming the game into an MMO. We're still in the process of taking a game that used to run on a single server and now obviously a server mesh, which is working, but the game systems need to expand as well. If you add the content pressure on top of that, of getting content into the game, releasing it at the same time, stabilizing it, hardening it, and features that come in and do the inverse. New features come in and they destabilize the systems. Without stronger guardrails that also need to evolve with this, that's where we ended up. And so, to be honest, I think right now we're outpacing our ability to verify everything that's being added to the game. And this is where we need to put a lot of our energy at the moment, to put in those guardrails, so that the check-in that would cause this crash issue, right? Those are identified at this point. And so it's all about adding these validation frameworks to prevent that from happening and keeping the data stable and fully validated. Now, that sounds... well, why didn't you do that before, right?
No, no. There are guardrails in place, but they're not keeping up with the pace. We create more content and more features than we have time to add guardrails and to prevent these cases. An example I could give on something unrelated to persistence: we used to make a weapon, we make a ship, that's it. Now we have a team making weapons. They balance the weapons. They design the weapons. They work in that realm. They have guardrails in there. They know what to check for, outliers, data sets, how power is distributed, et cetera. You go to the next team. The next team is making ship components. They have their guardrails. They have their approaches. They balance it out. You have another team that's making recipes and dismantling. They have their own guardrails, their own systems. You have another team that's making economy. They add those things to shops. They add pricing. They make sure that that's balanced. And then it moves on to another team that's adding this content to loot boxes. Now, we didn't have that a while ago. And then mission rewards are added in there. All of these have validation, but there's no holistic validation across. We're at the point now where all these systems communicate with each other all the time. And crafting has changed the game quite a bit in that area. And so that translates into more weapons being created, more content being created, more missions giving more rewards, and all these things need to be validated together, not in silos.
TL;DRBeausejour agrees the weapon pricing and the crafting breakdown were validated separately with no check across them, and says human validation is no longer enough, so machine validation is needed for everything.
Transversal to check all of that. And also, we're definitely now, post 4.7, in a realm where just human validation is not enough, right? And so a lot of game design data is typically managed by the team, but I think that we're way past that and we need machine validation for everything, because the rate at which we build it and the amount of technology we're putting in is just too fast for validation to keep up.
TL;DRBeausejour says the team is building a validation framework that designers can use to query across all game data and find outliers, plus more compatibility checks and the beginnings of patchable database data in 4.8, aiming for forward-patchable data.
Sure. So it's a problem we've known about and we've been trying to solve. And so more recently, there's more changes that have been coming from the engine side to give us a framework for validation. When I say framework for validation, I mean one that game designers can participate in, right? It's not like, oh, they need to hand it off to an engineer, the engineer is busy and he can't do the work and now the validation is not added, which is the mode that we've been in in the past, right? And so we've been in the process of building a validation framework for all game systems and all pieces of data stored in the game that can do queries across all data, whether it's, hey, I have a mission in this location that requires this, that pays out this. Is that an outlier compared to others? And so being able to do cross-data queries across the entire game, which is not something we've had before. Which allows us then to just express these checks, find outliers in mission payouts, find outliers in how, if we go back to the example of the gun: it's for sale at this location, at this price, it's dismantlable into these components that can be sold at this location, that exchange value is too high, right? And so creating these validations will become part of the design work with this framework, which will allow us to have all these guardrails. On top of that, we've got the more technical validation. So the two issues we got in 4.8, related to modifying ships that were already distributed in the main patch and removing entity classes, for example: both of those things already have locked checks right now. And that's the first thing we did when we discovered them. But we're going to keep adding more of those compatibility checks. 4.8 also has the beginning of database patching. So in 4.7, we didn't have the full ability to persist all the crafting properties on weapons. So if you crafted weapons, all the modifiers were not necessarily going to persist through LTP, right? In 4.8, we have introduced in the backend patchable serialized variables. We didn't talk about it much because it's kind of mixed into the crafting system, but basically now, in the schema of a component, which stores the data for something, let's say it's a weapon and I have the modifiers for it, let's say elongated nozzle, faster damage rate, whatever, we now know which one of those values is patchable. And then we're able this way to persist them and patch them in the next release. We also determine which ones we store when you stow it. We don't stow all variables, which used to be the case. And that's the foundation for us to keep working at what we really want to achieve, which is a forward-patchable data set, so that even if the ship team goes in, changes all the serialized variables for a ship and rejigs the whole object container layout for a ship, we'll be able to live migrate your ship to the new version without crashing the game servers in the meantime. So basically embracing this movable data set. But that's tricky to do. Obviously, we've been at this for a long time. But 4.8 has the beginning of that, just for crafting properties.
Well, it's pre-validation, right? This validation framework I'm talking about is pre-check-in. I'm working on the game. I'm like, I'm going to add a new gun and I'm going to price this and I'm going to make it super powerful. Well, that would flag. I could not check that in. The system would go, wait, you need some authorization before you can check this in, because of these reasons, which currently we don't have. That's cross-checking. Obviously in each silo, yes, but not within.
No, we're working on this right now. To get all these checks and balances in place, I don't know if it will be like a 4.9 kind of thing, but it's happening now. Because everybody around the table knows now, it's clear from the 4.8 experience, that machine validation is required across the board.
TL;DRJens Lind says the problem came from the 4.8.1 database combined with 4.8.2 data, an environment QA doesn't have, so hiring many more QA people wouldn't have caught it.
In this case, it took the situation of how the database on 4.8.1 was, with the new data from 4.8.2, to create the situation. And that's not something that the QA environment has. So whatever we do, we as a technology group would have to come in and at least give them that. And then if you put your 500 QA on it, then probably one of them would have spawned an Idris. But without that environment, you could put as many as you like on it. No one would ever have discovered this situation. This is just too far removed. And I think that sort of comes back to the large dataset, right? As you have a large dataset, you just can't scale in manpower alone. You need something that scales way beyond that.
TL;DRBeausejour says the team knows what to do, that building the guardrails will take a while but not months, and that he's confident such crashes and misconfigurations will stop as long as nobody trips the validation.
I think we know what we have to do. I think it's going to take a while to get all of these guardrails in place, right? But in the meantime, we just have to respect the rules that we've set forward, right? So that while we can't machine check everything right this second, and to be honest, I'm not talking about months of work here on these validation frameworks, right? But it's about putting in the ability for every team to contribute guardrails, which right now is a bit splintered, and so we need to bring this all back together. So I'm confident that, especially for the errors we just talked about, like releasing a piece of content that crashes every server on launch, twice, right, or evident misconfiguration of items or costs or recipes, I'm confident that that will solve the issue, that those issues will no longer occur as long as we respect the validation and nobody trips them off, which is the next battle. I think that we can eliminate that.
TL;DRBeausejour says the cause isn't pinned down yet, but some server meshes run below the target frame rate, and the team is tracking six to eight optimizations, including lock contention in the area component, for the hotfix channel.
Well, so what we know about right now is that there are some meshes, and when I say meshes, I mean a dedicated game server inside a shard, that are running way below the FPS spec that we want. And so we don't have a handle just yet on what is actually causing it. We do have performance captures, and we're tracking at least six to eight very specific optimizations that need to happen. One of them is within the area component, which is used all over the engine. Area and zone component, they kind of go hand in hand. Again, because we're pushing more content, the hierarchy tree of the game world is much bigger. So there's more areas, more zones, more subdivision, more, more, more. There are some areas of that where you add a lot of entities that enter or exit these areas, and the engine provides the listeners to track entities entering areas or leaving areas or zones changing, right? And obviously there's concurrency constructs around this to prevent one thread from accessing another thread, right? And so, just keeping it simple without diving too much into the low level, which we will probably do in some of our desync areas, there's lock contention in the area component at the moment in some places. There's lock contention in some areas of the game. Well, I literally mean area: if there's many entities in this spot, because of the way it's configured, we're seeing basically FPS drops in those areas. And so we're tracking, like I said, about six to eight major optimizations that we're trying to get in our hotfix channel, to try to get this FPS back up to the spec that we want, which is above 20, across the...
No, no, no. And it's important to note that when a server is running low FPS, it doesn't mean that it will create desync, but especially when you're interacting with entities that are simulated on that DGS, AI characters, NPCs, then you're going to perceive these things, because the source is slow at creating its output. The traffic goes from your game client to the replication node, replication node to game server. Now, if the game server is running slow: at the beginning of the frame, he's taking the state from the network to the game and applying it. At the end of it, he's applying the changes back to the network. If that loop is slow, then your client's going to predict something, and then that prediction will get corrected the next frame, and that will create a rubber band, right? And so that's the gist of why there's desync in general. Why does the client have to predict? Because if he doesn't predict, then you're going to be at the tick of the server, and you'll see...