Star Citizen: Reverse the Verse LIVE - Performance and Optimization

I think for me, interference may be a bit of a strong word there. I'd say in a lot of game development, you've got times at which you might need a short-term solution, medium-term solution, long-term solution. Obviously, with a game like this, we're always aiming to try and get the best long-term solutions in, work towards doing things properly. We're not a game where we can really take shortcuts. We have to look at what is going to work beyond the next patch. But with us releasing every few months, then to get builds release-ready, we are at times going to have to look at addressing issues in certain ways that may not be relevant in a year or two's time. So for me, it's kind of just part and parcel of what we do. It's not new to us as engineers. And even if we weren't doing the patches every few months and releasing them to the public, we still need short-term solutions at times to, for example, allow design and other departments to actually get on with their work. You don't just start with very close to finished product in a cycle of game development. There's a lot of prototyping and trying stuff out and changing things around a little bit until you come up with a good mix of things that actually works well. So yeah, for me, it's just part and parcel of what we have to do.

Yeah, I think that affects just about every aspect of Star Citizen's development, not just performance optimization. We are always building the first iteration of something and then wrapping it up in a form that can actually go out to people, and then building the next iteration and building the next iteration. Like with 3.1, we're putting out the first version of our character customizer. It is not the fully featured version that we want with the DNA morphing and everything, but it is part and parcel of how we've chosen to develop Star Citizen, which is live and out in front of everybody. You're working towards a long-term goal, but you're also working towards these short-term goals and having to package everything up in as nice a bow as you can to give to people every couple months.

The first time we started discussing network optimization was the day I joined the company, about three years ago, and we've been working on it steadily ever since, trying to unravel the pieces bit by bit so that we can make these changes. We're getting there, aren't we?

Yeah. There's a lot of the work involved, say, for example, Clive putting in the new network backend and then game code taking Clive's API and using that in the game code to replace some of the older stuff that we originally took when we got the engine. So a lot of the time it can just take several months sometimes to convert over what may be literally hundreds of pieces of code that we've got to make sure that they still work, and make sure that they then use the new code, which can then be more optimal because Clive's done such a good job on it.

Absolutely livid.

That's why he's got no hair.

Yes. I had long hair when I joined the company. Not anymore.

What can we say to help dispel that? Obviously, we all know that there are performance issues. We see the same frame rate that our backers see when they play. What can we say now to educate folks that what you're seeing is not just a blanket netcode issue?

Okay, so saying that there's performance issues in multiplayer is fine. To then pin it on a particular part of code and say, well, because it doesn't happen in single player, all the code that you can see running in single player must be fine, is a bit of a leap, because basically you have different quantities of entities that we're dealing with between single player and multiplayer. So you'll have, I'm not entirely sure how many entities we've got per player, but say 20-odd entities per player for each client that joins, and then they'll spawn a ship that could be somewhere between 100 to, I think some of them get up to like 5,000 entities, and then they'll go and fill those with cargo, which is more entities, and they'll fly off and they'll interact with pirates and task givers, which are more entities that get spawned, and there'll be debris pieces flying off. So the number of entities grows very rapidly with the number of clients, and that's what causes the performance load. So if people want to call that netcode, then that's fine, but it's not the networking specifically, it's the networking and the game code and the physics.

Graphics and everything, yeah. It's really just every module and every engineering department really has to take a responsibility for their own performance that they achieve in game. When we've been profiling this stuff, you tend to find that almost no engineering department can ignore performance. We get the UI guys, the AI guys as well, the audio guys. It really is shared across the board. And it might be that, say, UI do really great optimization, and then all of a sudden the next biggest hitter on the list is the audio guys, and then it might be the graphics guys. From time to time, unfortunately, things can unexpectedly crop up, and all of a sudden we have to jump on those, fix that, and then get back to bringing the average frame time down, improving things.

I think as well, there's been a bit of miscommunication in that for a while we've admitted that the bottleneck on the server was the networking code. And I think that's been misunderstood a little bit, that it was the bottleneck everywhere. So maybe that's partly responsible.

Yeah, it's one of the reasons why we had wanted to have you on Happy Hour at the beginning of this year. But as I delved deeper into everything that was going on, we realized that this is far more than just netcode and networking. We wanted to do a much more robust feature, and that's what folks have hopefully seen on ATV yesterday. Performance and optimization is a task that affects almost every discipline in Star Citizen. So it will be all teams working in conjunction to bring Star Citizen to the 60-frame heaven that we're all aspiring to.

Yeah, and the other thing to add as well, of course, it's not just us engineers. If a designer wants to drop 2 billion entities in a level, then performance isn't going to be great. If an artist wants to model an apple out of a trillion polys and then place it on a table in an outpost, then again, it's shared company responsibility, really.

Somebody once told me that the original Freelancer model, the one that had all those crazy rivets, because the rivets were created incredibly poorly, to be honest, had a larger memory footprint than the Idris. And that's definitely not network related. It's one of the reasons we go back and we revisit these ships, not just because we want to change them for gameplay purposes, but sometimes we just need to seriously optimize the way they were built. We've gotten much better at building these ships as time has gone by.

So beyond 3.4, so 3.4, we've got on the roadmap server migration. So for two systems, they're always going to work, or the plan is for them to work, like separate instances. So the servers for one system and the servers for another system will be completely disjoint and won't communicate at all. We're thinking, maybe, it's not decided yet, that we could use the server migration technology to help provide a seamless transition via the jump points from system to system. So potentially what we could do is someone wants to jump from one system to another. Their ship gets put into a server in its own little bubble. And that server then breaks off from the mesh in one system and then is moved over to a mesh in the other system and is connected in. One of the difficulties that we'll have to try and solve if we do that is, the reason we won't mix systems together, won't connect servers from different systems together, is that each system has its own coordinate system, with the star being at the center, 0, 0, 0. So we have to move from wherever you jumped from, say, the Stanton system, to whichever destination system you're going to. Move the ship, move that server ball across space without colliding with anything else that's in either of the systems. So it's a little tricky, but we're thinking about it. We have to build the server migration tech, server meshing tech, first, and then we'll see if we can utilize it that way.

For me, I think one of the biggest things this year for us is object container streaming. That's something we're working towards in 3.3. It kind of goes hand in hand, really, with some of the new entity component work that we've been doing, so the entity component update scheduler. Really, with object container streaming, you should get optimizations on load times, for example, because you'll be coming into the PU only needing to load in the immediate area around you to begin with. So from our point of view, as devs, we're very much looking forward to being able to get in the PU and fix some bugs in an absolute fraction of the time it takes us to load into the PU right now. So it's not just going to be a benefit for you guys playing the game. It's going to be something that is great for us every day, work-wise. But to do that, there's a lot of talented guys on the engine side working on the back end for that. But also, game code-wise, we're going to have to go through lots of old game objects and convert over that code to our new entity component system. We have to make the various classes thread safe, avoid crashes. Those new classes then should in theory be ready to go on the batch update, spread the load across multiple cores, allow the game to scale a lot better. So really it's kind of hard to pin down one specific optimization when we have several all being worked towards at the same time. We have things batch updating now. We have a lot of these classes that are now thread safe. We have a lot of stuff that's been converted over. There's just a couple more releases, probably, until we're quite there. So yeah, that stuff is something I'm very much looking forward to, because a lot of my job for the last three or four years has been getting rid of a load of old code and putting in this nice new shiny stuff. So we dream of the day where we just have this nice shiny code base that's our own, and we're not fixing crashes in something that got added years ago, and we can just enjoy the new code. That's where we want to get to.

It's kind of like being asked to pick a favorite child.

Okay. Well, an object container, in its simplest form, you could maybe just think of it as actually kind of like a level without a few bits of stuff that's specific to it actually being a level, if you like. Because most of what a level is, is it's just a list of objects. So in CryEngine or Lumberyard, you're looking mostly at entities and brushes.

So it'd be large-scale pieces like ships and space stations and planets. They're all object containers.

So the data is like an XML file format, which just is containing a list of things. And then if that's an object container, the idea is you load into the map. You load one set of objects that's around you. You spawn a ship. The ship itself can be an object container. It's just a set of objects. Hop in that ship, fly across the galaxy. You no longer need the object container that you spawned in in the first place, say, Port Olisar, so you can remove those entities. So really, when we say object containers, it is kind of just that. It's a container with some objects in, and the way we've got the game set up, we can spawn those in dynamically, remove them dynamically, and that's really the streaming process, put as simply as I can.

Gotcha, so it's like when you load into Port Olisar, it doesn't have to load up the Starfarer that's sitting on the surface of Levski into your memory. You can't see it, you can't interact with it right now, there's no reason for it to send that data to you and for it to load into your memory. A Starfarer, a Reclaimer, an entire base, they're all various object containers of different sizes and scopes. And this is just the system that determines, all right, we do need to send this to you, and we don't need to send this to you, and when and where that happens.

Yes. Yes, I think so.

Do you have any elaboration?

Yeah, okay. One of the things you've got to bear in mind is that for 3.0, the servers were running on 16 cores. So we run one thread per core. So we had 16 threads available to the server. The servers that Amazon have available to us have up to 72 cores. So obviously there's a lot more cores, there's a lot more scope for us to expand and run more things in parallel. With the changes that we made to make the network code run more in parallel, it's starting to look like, yeah, we could pretty much scale up the numbers quite well. It relies a lot not just on the network code, though. It relies on, as Rob was saying earlier, the mix of everyone, of the physics code and the gameplay code and everything working together, so that we can fill out those cores as optimally as possible. I do believe that we've got the systems coming online that will allow us to do that, such as the entity, I always get stuck on this name, the entity component update scheduler. So it's just a question of leveraging those to the best of our ability. And then, yes, I think massive space battles will be a reality.

Yeah, you can see what we're trying to do all the time. We're really just trying to push things as much as we possibly can, keep raising the bar. Like Clive says, we've got the plans in place. It's really just now we've just got to continue implementing things the way we've been doing it. We'll keep on raising the player counts, allowing more ships into the PU. And so hopefully we'll reach a point where people consider the amount we've got to be massive. So yeah, we think it's feasible. We're going to keep going towards it.

Yeah, it is very similar. The only difference being that with serialized variable culling, the entities themselves are kept around on the clients. With bind culling, the binding part is effectively creating the entity on the client, connecting it to the network, that's the bind part, and letting the server know that this entity is available to talk to on the client. [Keeping] the entity around, we solve that problem. We don't have to get rid of the entity itself, which means that then the data for that entity is loaded in on the client. So what you could have is a situation where you've got a ship, you're at Port Olisar, you're watching a ship quantum travel away. So if we're bind culling, that ship would be unloaded. You'd potentially lose all the data files. And then if it quantum travels back, you'd have to load those data files in again and respawn that entity on your client. And that will cause a loading stall at the moment. So the reason that we haven't gone with bind culling for 3.1 is because we don't want players to experience additional loading stalls beyond what they already do. The reason it was on the 3.1 roadmap was because we thought, what the hey, we'll give it a try. We'll see how bad it actually is. But in the process of working towards 3.1, priorities have changed a bit, and it was always a gamble, as I say, whether bind culling was actually going to improve things or make things worse. So we decided that we'd shift focus and make sure that serialized variable culling was in and working, and then assist with the different optimizations that are being done.

Yeah, from the game code perspective as well, what you always tend to have in engineering is you can't help but have certain assumptions as to the way things work. And if all of a sudden we've got entities that were once there, albeit the other side of the universe, and now they're not, then by the nature of the fact that we have so many tens of thousands, sorry, hundreds of thousands, millions of lines of game code, there's going to be places where people have assumed once an entity exists, it's there, it's not being pulled in and out depending on whereabouts you are in the server. There's going to be fix-up time that we're going to have to work our way through.

Well, ping's definitely an unbreakable barrier. We are planning on attempting a single shard. But ping's definitely going to be the big obstacle that we're going to have to face. So a number that I've kind of got in mind is, I think it's Australia to parts of the US is something that typically, like a 200 millisecond ping; from EU to US, something around 100 milliseconds or so. So they're not insignificant latencies. So potentially what we could do is try and find a server that's geographically in the middle of the players who are interacting at that point, and try and locate all the players who are interacting. So all the players who are in Port Olisar would be on the same server, and the same server would be in the geographic middle somewhere of all the players who are at Port Olisar at that point in time, and try and optimise the ping that way for as many people as possible. What we could do as well is we could kind of layer the servers, in a sense. So everyone connects to their regional server, or servers closest to them, in a regional data center. And then those servers then communicate with one that's geographically centered around the others that are connecting to it. So you can try and minimize the ping that way. Some of it's going to come down to trying to find techniques to hide the latency as much as possible. It really depends on the gameplay situation. Obviously, it's mostly going to affect fast combat. Yeah, it's going to be challenging. I don't have any definite answers about how we're going to tackle all that, but we've got to build the tech first so that we can attempt to try and find solutions to those problems. The other thing to say is that probably Arena Commander and Star Marine, it's not fully decided yet, but they'll probably always be region-based. So you'll always be connecting to your local server for those games. So they should always have nice, sharp, low pings.

Yes, yes. It's going to be a bit awkward. So the way we cull things is not just on a distance basis. It's kind of an observed size. So if you imagine holding your arm out in front of you, your thumb up, anything that appears bigger than your thumbnail to you is something that we'd want to keep in. Anything that's smaller than the size of your thumbnail would be something that would get culled. So adding a telescope obviously means that your field of view gets narrower when you're looking through a telescope. So everything appears much more magnified. So that means that you can see much smaller objects, so we have to keep in things that are much further away. So yes, it's going to extend the ranges a lot. It could be a bit of a problem. I'm sure it'll be fine, but it's something that I'd like the game designers to rein in, please.

Yeah, I think there's similar kinds of issues with comms as well, because obviously if you're talking to multiple people in multiple locations, all of a sudden the challenge is to figure out how we can accommodate that situation without taking the load of every location that we want to show and show some detail in. That is probably one of the most challenging gameplay aspects we have to deal with.

Yes. We'd like to make it clear that all the optimizations we're doing code-wise, they're not specific to any particular machine specs. The code doesn't really care whether it's a high-end or low-end machine that it's running on. If we're putting optimizations in where we found something we don't need to update, this object in this situation, or we cut down amounts of expensive code that we call, those kinds of fixes will apply to both low end and high end. So I think it's probably hard to judge exactly how much on a high end versus a low end you save, but the code is fixed for both, so you should see a saving in both. The aim is to optimize across the board.

It kind of applies for servers as well, because obviously they're super high-end PCs, if you want to think of them that way, so any optimizations we do will benefit the servers as well as clients. Triple win.

Well, I think the thing we need to do is diagnose exactly why those spikes are occurring. Without that information, you can't really tell what the optimisation is going to be.

We need to find the problems and fix them. We've got plans in place to do that, though. There's been new telemetry systems added recently. It was talked about on Around the Verse by Bolte in Frankfurt. He's been putting stuff in that will do things like automatically capture performance telemetry when the frame rate dips below a certain threshold. So whether it's in our internal playtests, or potentially we have plans to take telemetry from backers playing the game on live as well. Basically, we should be able to get telemetry for these drops in performance, and then we can analyze it and say, it might be, oh, it's the game code, oh, it's the physics, oh, it's network. And so it's a bit like catching fish. We just need to cast the line out and hopefully, if there are any whoppers out there, we're just hoping to reel them in sooner rather than later, get them fixed up. But we've got much better systems in place for 3.1 to catch those things. And also those new telemetry frameworks, the idea with them as well is to constantly pull in data. So it's not just about the frame rate drops. It's about the baseline performance as well. So we should have much more accessible ways for your average engineer in the company to look at the performance of the game and go, these 10 functions are the slowest in the game. Oh, look, one of them is something that I know about. I can fix that. So it'll be a lot more transparent as to where the issues are, and then that should lead to them being fixed up a lot faster. We're kind of hopeful that the tools we're getting in place are much better. It is a gradual process. Some days you can get some good wins. Other days you're scratching around, and you feel like you go to places where you'd love to be able to optimize it, but it's much more tricky. Going off on a bit of a tangent there, but we've got plans to get these frame rate drops and get them sorted as soon as possible.

Well, the way we build the game, Star Citizen and Squadron 42, we're sharing the code base. So really a lot of the optimizations we're applying already will benefit Squadron 42. Anything that's maybe range-based optimizations, visibility-based optimizations, i.e. what can you turn off at range or when you can't see it. Also putting in systems to make it easier to time-slice updates. So you might only need to update certain objects once every second as opposed to 60 times a second, or in some cases you might only need to update things once every five minutes. Really, those kinds of systems that we're putting in will apply to both Squadron 42 and Star Citizen. The goal that Chris wants is to move towards more systems that, what we call LOD updates, where LOD is level of detail. So the idea might be you can have an AI character loaded in memory very far away, but for the client, they don't need to run such a detailed update function for him in that situation as opposed to when they get closer. There will be certain optimizations that are specific to Squadron 42 and Star Citizen. For example, the network optimizations will just apply to Star Citizen. There'll likely be a bunch of stuff we can optimize in Squadron 42 that we maybe can't optimize in Star Citizen, because it's a single-player title. So yeah, we're thinking about it. We're getting optimizations in already that will benefit it. So it's in our minds, and we've got guys upstairs all working on it now. So when the performance gets not so good in certain areas up there, we know about it down here on the PU floor, and we're there trying to work with them and help improve things.

I'm not too familiar. I think feature-wise, there's issues with the sheer size of the ships, and there aren't landing pads big enough for them. So there's some docking collars or docking technology that's required. Performance-wise, we've had the Idris in. We demoed it at, I believe it was Gamescom last year. So the performance technology is there, it just needs improving. I don't think it's a million miles away. On the network side of things, I think we're probably okay, because we don't really treat these big ships any differently than we do a player or anything else. It's all the same. It's all just entities to us. I don't think there's any big blockers on the gameplay side.

No, for us, game code-wise, it's really just a push towards object container streaming, and also the ability to stream ships on background threads, which should come in as part of the object container streaming work. That will eliminate the stalls that you see when you spawn a few ships at a time. We're aware of those issues. QA here tell us about those issues. As engineers, we just have to go, it's going to be fixed in 3.3. So obviously, from a gameplay perspective, the bigger the ship, the bigger the stall when it loads in, because there's more entities that make up the ship. So whilst it's maybe not a blocker in the sense of, we can spawn big ships at the moment, to get them to a point where it's good enough to get them actually in the game for the releases, we just need to get that spawning on the background and a few more of these optimizations in. We're just building up to it. We've already got ships in the game like the Starfarer, which is pretty massive.

And the Reclaimer in 3.1.

Yeah. So we're going pretty big already. But yeah, the Idris is pretty massive.