Star Citizen | 10 From the Community - Technology in Focus

For me, there's three main ways that we're tackling this. Number one, root cause analysis. What is that? So when you're faced with a bug, you can diagnose it sort of surface level, find out what's going wrong and fix it. But unless you follow that through and you go down the rabbit hole and find out what the ultimate root cause of what is essentially a symptom is, there's a very good chance that that is going to come back. So some good examples would be freight elevators, hangars, ASOP, ATC. Freight elevators especially, we've had a decent amount of time of people on our teams really digging into this at a core level, and hopefully people in the community are starting to see the improvements from that.

I mean, that's the thing: root cause analysis changes how we assign our resources when they're debugging things, so that they stay longer on the issue. They go a lot deeper and they document the process before taking action. Then we can identify in that process regressions that have caused the problems to begin with. And then we can also build a comprehensive fix plan, which then feeds into the second point.

So the second point then was expanding our test plan. And that's the thing that you've worked quite a lot on.

Yeah, so basically this is the idea of going from a given change to a larger test plan for regressions.

And what is a test plan?

Well, it's basically a list of steps, a matrix of steps or tests that QA needs to perform, which are much larger now with these expanded test plans.

To validate the fix or to validate the change.

Validate the fix in multiple configurations. We typically used to do these tests, but deferred. Now we're trying to do them pre-submit. And so what that means is that we're doing multiplayer testing with more players pre-submit. We're doing server recovery testing pre-submit. We're covering the happy and the destructive path for each feature pre-submit. We used to do that post-submit.

And that leads into the third point that we mentioned, which is just massively scaling up our suite of tools and frameworks and functionalities for validation. So whether that's in content creation tools or scaling up our use of our automated testing framework.

It's about expanding this testing so that we can simulate live environments basically as much as we can. And so validating the content allows us to make sure that our content and code that are linked actually work properly before submit. But then after that, it's about validating that when you put a lot of players on it, it actually functions the way you would expect it to function.

And that's ultimately what this question is kind of speaking to. It's like feature roulette. You don't want to be rolling a dice.

I agree, 100%.

And I think all of that combined can really give us the confidence that this approach is going to work, and the stability and the robustness of our game, its features, its systems, its content, the waterline of that is just going to continue to rise.

Well, I can speak to that. Star Citizen currently uses a majority of Building Blocks, actually. There are major elements that we still need to be able to completely remove Scaleform from the game. There's still some legacy elements tied to the visor, which need to be ported over and transformed into Building Blocks. And there's two major applications that are left: the Comlink app, and the next one is the VMA, which is the vehicle manager, which is being replaced by the vehicle loadout system. So both of these are actively being worked on to replace, and this will allow us to completely remove Scaleform from the game. Removing one of the render passes on the UI on the visor will really help performance. And more importantly, it standardizes our UI framework in one, which helps the UI team go a bit faster in some elements. And so out of this, currently Star Citizen still uses Scaleform, even in the Building Blocks elements, for font rendering, but that's been completely replaced and will roll out to Star Citizen at some point.

So that actually brings me into my first point, which is fidelity. In Star Citizen, it's all about fidelity. We want everything to be immersive and keep you there. And two players performing together is always quite challenging. The takedown becomes a little bit easier because it's a... less peaceful.

Well, it always has one final destination.

Exactly. And body carry or body dragging tends to be more, you know, you're already out. One of the persons is in a position where we know kind of what they're going to be doing. We're obviously two players goofing around doing all these things and one is trying to give something to another. But I think the other point is obviously the security of it: if I hand something over, if I want Benoit to have it, I don't want Carl to come running in and grabbing it from me. Or a system like that opening up an avenue for people to take things unfairly from other players. Those are the main considerations, I would say: the safety and security of them, and knowing that you're not going to lose your stuff.

So I'd like to bring it back to the question: why don't we already have it? Well, it's technically possible. There's a bunch of challenges to resolve, like we've just talked about, but ultimately it's a design question. It's a prioritization question. Do we want to do this over doing something else? Do we want this in the game? Those are questions for our creative director, game director. And it'll be in their minds, I'm certain. And then when it comes up, we'll have solutions ready if we want to do it.

Yes, social tools are progressing. While it's not going as fast as we would like, because we did have to divert resources to the efforts we talked about at the beginning of the show, there's still a lot of work actually being done on social tools in general. So back-end work has been moving along quite a bit, because there's a lot of back-end changes to be able to provide that. Social tools are inherently real time, and so there's a lot of communication messages, asynchronous messages that need to go out, and so that back end needs to support that really well. That's actually on track and is going really well. What we are still planning to do here, and where we're slowed down, is implementation in the game itself. Our first drop for social tools that we have planned is a revamp of the chat systems, which includes regional chat or location-based chat, meaning that we split the main chat into multiple depending on the location that you are at. This is in preparation for larger player count support per shard, so that we don't have a massive chat at the top of the list there. More presence, rich presence updates, the chat widget, a complete revamp of the chat widget, and a new application called the Social App that replaces the Comlink app, which, as we discussed, is an old Scaleform element that we want to get rid of.

I guess that's the other big part of this. There's a lot of UI work involved in screens like this.

Absolutely. The chat widget is a very complicated piece of UI. Even though it's simple in its appearance, the functionality behind it is very in-depth. We want you to be able to have different channels that you can manage, manage tabs the way you want, maybe have a bit more flexibility on where the UI is positioned, how it works, can you expand it? So we have a lot of functionality for that that's being done. The design is completed, implementation on the game side is about to start, UI is already on it, so it's progressing really well. That said, there are still features being worked on in the current version of the game. So there are improvements to VoIP and FoIP that are coming into every patch.

What are VoIP and FoIP, for those who don't know?

Voice over IP and face over IP. It's basically being able to transmit your voice over the network to other players.

And your face.

And your face, in our case, yes. So far, I think since 4.8, 4.9 and 4.10, there have been incremental changes to the VoIP back end, which transmits all of this. In 4.10 too, actually, there are changes to the synchronization code that synchronizes the face across the network, which is a lot more robust. And so unfortunately, for this feature, one of the big things we think is holding it up is supporting the UI in the game itself. Receiving calls right now is a bit hard to find; you don't get proper notifications for it. And so obviously, to be able to really feel these improvements, you need the UI changes. It's actually moving forward and we're trying to bring any improvements that we have into the game right now.

The question was, is the development of social tools progressing? The answer is yes, very much. It's coming to a start as a new year.

So I did inquire about this, because I did not know the answer to that. So I went to the source. When we implemented LAMP, it was a limited feature that was meant to be only on fighter ships initially. And there was a reason for that. It's because in order to get the feature in the game in a timely fashion, we knew that we could get this working with the exterior canopies really easily. For example, you take a Gladius canopy; the canopy is part of the exterior of the ship. And the way LAMP actually works is there's a submaterial on the inside of the canopy itself, so that if you look at it from that direction and LAMP is turned on, then the visibility is changed and the effects come into play. Now, the problem is for static canopies, which are part of the structure of the ship, there is a linkage issue at the moment that prevents us from properly activating or finding that submaterial from code. Now, it's solvable and we will solve it. But we're trying to find a way to do that without having to reauthor 100-plus ships to be able to activate LAMP. And so this is what's keeping us from doing it now. It's a priority call that we need to make to get it in there. We do want to do it. We do want to support it. And so we need to find a way to do it without having to reauthor all the materials in the static pieces. And so most likely, we already have all the theories set up. We just need the work to be planned.

I actually checked with John, because again, there could be a creative consideration here. Do you want to have LAMP on every vehicle? Should it be only on certain archetypes? He said, nope, we want it on every vehicle, including ground vehicles. So John loves LAMP.

Oh yeah, absolutely. What I want to do before we bring ground vehicles up from the freight elevator is I want to decouple from ATC. I shouldn't need permission from the air traffic controller to bring my ground vehicle up. I'm in my hangar. I'm in my own little safe space. I can do what I want.

I mean, it's air traffic control. What do they care about?

They should care about what happens when I leave the hangar, not what I do in the hangar. So I definitely want to do that. And then the other part is the insurance handling flow. Ground vehicles are vehicles, and vehicles have a slightly different insurance flow than other items. So that means we need to make sure that that insurance flow is available to the freight elevator, for freight to be able to do the same thing. I wouldn't want to go to the ASOP terminal to deal with insurance on my vehicle, to then go to a different terminal to bring it back up. So those are the two technical hurdles for me that I would like to see solved; I think that will make it a more robust feature.

Yeah, this ties very directly to our first question. Ground vehicles coming up from freight elevators, I think that's appeared in our internal roadmap, but we're not going to do that until we feel comfortable that we have really, is "robustified" a word? It is now. Robustified freight elevators. We don't want to build a feature on top of something that has reliability issues. And we don't want to exacerbate that by looping it into more gameplay loops. So we want to instead invest the time fixing up freight elevators like we just described before, and then layer on top.

I think also the point about decoupling from ATC is an important point. The air traffic controller in the game has evolved into also the manager of your insurance claims or the destroyed state of ships or things like that. And we don't want to keep that logic in this area. We want to be able to use that elsewhere properly. And so we just don't want to build on top of the current ATC to support the freight elevator with ground vehicles. We really want to decouple that. So Jens, you mentioned something just earlier about how insurance is different for ground vehicles. What did you mean by that, really?

It's just purely a technical aspect. So it's not that vehicles have a special type of insurance. It's for us on the engineering side: the flow itself is different and we just need to move it out so it becomes available for the freight elevator kiosk.

And other kiosks.

Yeah, exactly.

Well, when we reviewed the questions, we saw a lot of questions related to desync. Recently, we have done some testing on our Tech Preview for some desync changes. This is kind of a result of the work that we've been doing in writing the desync post that I got coerced into writing. And so that is still ongoing. I know this is the underlying question. There were also some notes about this. And so we are about done with writing this desync post, which has a lot of information trying to explain where desync comes from. It's big. It's 45 pages.

It's big, but it's good. It's very good. A lot of detail from a lot of different angles.

And so one of the things with this is that we didn't want to just come out with some additional post or information without some level of action as well. And so we've been looking at improving some of the netsync elements. And so what we've been testing recently was a targeted change to how we simulate rigid body entities with the physics engine.

That went on Tech Preview. It was very well received.

Well, as a first drop, it was great. That's why we wanted to move it to Tech Preview rapidly, because we wanted to gauge the actual impact of this change, which seems to be positive. Now, there's a couple of fallouts from it that we're going to fix, but that's an example of how we start to look at more of the desync from different angles. Obviously, our post will go into details about all the sources. Because unfortunately, it's a bit like the flu: there's a symptom and then there's the cause. And so there's many causes for desync.

It's also an umbrella term, right?

Yes, correct. And so we'll talk about that in the deeper post. Plus, I'm hoping that some of these changes are going to yield good results, which I think this first Tech Preview at least showed us some promise of. We're going to do more of those targeted changes so that we can measure that and see the impact on players. That's really what we wanted to navigate: when we see improvements locally, we see improvements with our development setup, internal testing, there's nothing like bringing it to you guys to see if there is an actual effect when you're trying to do something with it, whether it's a Jenga tower or, in this case, since we're doing rigid bodies, that was really the test. And so that change was really targeted at how we synchronize these objects, and especially revisiting a decade-old system that's been in the engine forever. And so I'm happy with the first pass result. I think this is really promising. And then if that scheme does hold up, then we're going to roll it out to more elements in the game. Currently, for example, ships and vehicles are not targeted by that. There's also a bunch of different types of entities that are not targeted by that. So there's a lot more things to roll out there. But I think the approach is looking sound, so I'm happy about it. That's what's coming down the pipe: I think the desync post will answer a lot of questions, I'm hoping. And then we're going to do more Tech Previews for these tech changes.

I mean, it's really cool to see these changes coming in. It's really exciting. I think it's the start of some really, really great changes. So the team's done awesome on that.

It's ultimately a design question. There's a big design and creative element to this as a feature. But we did dig out some interesting information to talk about from the tech perspective.

I asked Benoit the question: can we do it, technically? If someone came to us and said this needs to happen, could we do this?

The answer is yes, we can do it. And I can add some color to some of the elements here. So in 4.8, there was a major change to the game, the back end specifically, where we introduced a new entitlement system. Now, for those who don't know, the entitlement system tracks what you own. This used to only track your actual platform entitlements, which are either elements you've received through Subscriber flair or actual real-money purchases or things like that. Now, since 4.8, the system tracks everything that you own, whether you bought it in an in-game shop or through any other acquisition method except stealing. That's the only one that's not covered by this.

You're essentially the legal owner of this. You've been identified as having legal rights to this item. So when we go back to the give, when you give it to another player, you would legally give them the rights. So we'd have to tell the entitlement service about that.

Correct. And the entitlement system also tracks your insurance loadouts. And so through this, the ability to sell a deed like this would involve the entitlement system, and that system can do it. Obviously, we already have wallets for transactions and other things like that. Maybe some color commentary here: there's also active work in consolidating a lot of the economy-adjacent systems together. So we have the wallet system, which is called the ledger service, that actually drives all of your monetary currencies. And there's all different systems that attach to that. So we're bringing all those into a single economy system, which will include, at some point, marketplace features, reactive scenario features and other elements of the economy. If we did build a system where you could sell an existing ship, then those systems would be involved in making the transaction. We already have the ability to do that right now.

Yeah, so deleting a ship could actually be a zero-cost transaction and maybe even be recoverable in that case, in case you've done it in a moment of weakness.

Deleter's remorse.

Yeah, exactly. And now the advantage of the new system too is that it persists cross-patch. So your entitlements are completely persistent cross-patch. We don't recover them from the previous patch. They are just there. And so that's why we no longer have issues like, oh, I lost my ships between patches, or things like that. Those are completely persisted by the entitlement system now.

So Server Meshing has been amazing. I think it's definitely delivered on what we needed from the technology. And I think it has lifted server frame rate, which I would hope means people in general have a better experience. But there's still a lot more we want to do to get server frame rate up, which will help NPCs, because in general NPCs can be perceived as slow to react, because they need a little bit of time to cook. They need a few frames from the server to get all the decisions made. So if you are unfortunately on a server that isn't performing at the rate we want, then it could be a bit slower.

There's also a condition we know of that we need to solve, which is related to large areas streaming in at once, which can cause server stalls, like we've seen with Siege, and that's something that we're tracing. Solutions are on the way to resolve that. It's a lot of deep engine work here, because we need to parallelize a phase of the initialization of entities.

We should be cautious of attributing every AI issue to server performance. Sometimes we do just see genuine issues or glitches or bugs and things like that. We just work through them, we fix them. There's a bunch of good work been done recently, right?

Well, that's what I was going to point out: we've been developing upgraded behaviors for our AI. Obviously, we've been tuning this work in Squadron 42. At this point, that work is completely signed off, and we're very happy with those new behaviors. This touches on many elements and many systems, which is why we can't bring it back right away. But we're talking about balancing the perception system and vision system so that the AIs, the NPCs, are a lot smarter about picking their targets. The ticket system tuning: this is something that in Siege of Orison we've actually seen, because this is one of the instances or areas that we've had the most active AIs in. And you could see that there are load balancing systems in place when many AIs are in the same area, where the server is going to give a certain amount of budget to each AI. And that's tuned for the current game. Siege of Orison had 10 times more AIs in general. So you could see how the load balancing wasn't perfect. This we're tuning on the SC side, but there's been massive work on this. We have a lot of changes related to how the tactical queries actually operate, which is how AIs actually plan their work. So there's quite a bit of work that has been done and completed there that really changes the behaviors in general. SC still uses an older set of roles for AIs in general: pusher, defender and strafer are the major three archetypes that we use on SC. We have a lot more roles now that are available in this new version of the AI subsystems that we need to bring back, including things like using cover, relocating between cover properly.

And so that's better tuned for a large population of NPCs. There's more roles available to pick. So if there's only five pushers, then obviously if you have a population of 70 trying to engage, then you need a few more roles.

Much richer behaviors will also allow things like not having to put as many AI agents in one area, because they can all be more effective.

It's like a double dip of improvement as well. The more roles you have, the more varied those roles will be, the less intense some of them may be in terms of performance requirements and things like that, which helps with performance, obviously. And then, like you said, greater variety of AI behaviors and archetypes means maybe a need for fewer AI, which again will help with performance.

Also potentially easier for the client to predict what an AI is going to do when they know what the role is, if the role is more defined.

So I did ask the team about this to see what the current plans are. And yes, there's a lot of planned and actually active work being done on ground vehicles in general. So to give a bit of context, there are currently three, let's call them setup rigs, for ground vehicles in the game. There's the really old Crytek vehicle system rig setup. Obviously, it's been tweaked a little bit, but that was the original one. I think the Mule is using this, for example. It's the most simple setup of all. The second one is a hybrid version, so the tanks, like the Nova, take this old physics rig but then add a little bit of control over it. And then the final one is a fully simulated version of it, which all the other vehicles are using currently. So we do have plans to unify this so that everybody uses the best version. That's the first stage. And then we want to evolve that quite a bit to give it better handling. So like tuning up the physics for rotational clamping, collision damping, things to make the experience smoother in general for gameplay inside ground vehicles when they collide with terrain, especially with V5 coming out, where this will actually help.

Yeah, Planet Tech V5 coming out in Genesis. Greater fidelity of physics proxy generation for terrain, things like that. So that is only going to be an improvement.

So that even helps with the old rigs. So you get a bit of a better feel on the ground. But there's a lot of physics upgrades that are needed for ground vehicles, and we are planning to do them. We're talking about modeling the drivetrain, braking model, engine and suspensions. We already have suspensions, but we want to improve that quite a bit. And so from a systems and design point of view, yes, the plan is to unify all ground vehicles under one setup and then improve that setup across the game.

I mean, I worked in racing games for 13 years. I know that this stuff is challenging, but we do have an excellent team, a comparable size team and composition makeup to what I would have had in something where the entire game is about racing and about driving. So with that unification and those good people, I think we'll be able to get where we need to go.