The main concern has always been trying to fit Star Citizen onto a player's computer. Obviously servers tend to be a lot more powerful than the domestic user's computer, so having all this expansive solar system, all the planets, all the locations, all the ships loaded in all the time is just not possible. Not without masses and masses of memory, which is much more than your typical user is going to have. So the primary concern is just making the game fit, breaking it up into smaller chunks that can fit on each player's PC.
TL;DRYes, servers will also need to stream locations in and out on demand because the denser universe won't fit in memory, but that is not being worked on for the current release and comes later.
That's right. That's not something we're working on for 3.3. That'll come further down the line. But because the solar system is going to get even more densely populated, with more planets and so on, even the servers, with the massive amount of memory that they have, won't be able to keep everything in place at the same time. So they'll need to stream in and out locations based on demand. So when a player goes to a location, or as he's heading towards a location rather, the server will detect that and go, okay, I need to stream in the entities that are there, the planet and derelicts and whatever are around. So by the time the player reaches that location, it's all in memory on the server. The server has to coordinate that with the client itself and tell the client what it needs to stream in, so the entities are present on the client as well. And then when the player leaves that location, the server's free to unload that information and free the memory up for anything else.
TL;DRMap streaming is more common in single-player games because most multiplayer games keep maps small enough to hold in memory, whereas Star Citizen's maps are vastly bigger so streaming must work online without stalls.
Well, map streaming is generally more common in single player games. A lot, not all, but a lot of multiplayer games will confine the maps to smaller sizes so that it can all fit in memory all at once. So once you've loaded in, all the data is resident on your client, no streaming's required, and that cuts down the chance of any stalls or any frame rate hits and makes for a smoother online experience. So the challenge is to do streaming during a multiplayer game and not have any stalls or performance hits. And it's also, as I say, about the scale that we're doing this on. Obviously, our maps are millions of times bigger than the maps of virtually any other game. So it's a question of scale and trying to do it while we're online as well.
Yeah, we're probably one of the bigger games ever envisioned at this point. We've done quite a lot of stuff over the last couple of years where we've had snippets of gameplay, Star Marine, where it's been a small contained bubble of the game, and we're now going, right, it's now that, but it's in this big massive universe that's making everything bigger. And by going, well, if we only need to stream in this one bit where the FPS is happening, the performance should be the same as Star Marine, probably, on that point.
TL;DRThey can't guarantee performance gains since hardware varies widely, but streaming should reduce client memory and entity overhead by loading only nearby things and loading in or out others as they arrive or leave.
It's not a silver bullet, as you mentioned before, so we can't make any guarantees on how performance is going to improve, partly because there are so many variations of CPUs and motherboards and RAM and everything else and GPUs that you can't predict how it's going to affect all the different combinations. So it's a best guess of what we think is actually going to help and what it's going to look like. So effectively, the difference to what you have now is: you start off in your bed at Port Olisar and Yela is loaded on your client, and there are entities on your client in memory, hopefully idle and not doing anything if the in-range update is working correctly. The difference will be that with OCS you'll load into Port Olisar and basically Port Olisar and a few entities for the star map will be loaded, and that's it. Any players and ships that are around your immediate vicinity will be loaded in. Anyone who quantum travels to your current location will dynamically be loaded in. Anyone who quantum travels away will be loaded out and your client won't have any awareness of them anymore. So it cuts down the amount of memory that your client needs. It should cut down some of the performance overhead for maintaining all these entities in memory as well. And as you move around the solar system, you'll unload areas that you don't need anymore and load in the new ones. So you head off to Levski, stream Levski in for the first time, and that's the first time your client is aware of those entities in that location.
TL;DRStreaming isn't a fixed-distance bubble but depends on how large an object would appear, so big things like a city stream out far away and small ships much closer, all decided by the servers.
Pretty much, yeah. It's not exactly a hard bubble. It's not a defined distance. It's actually the size the object will appear to you if you were to look at it. So an object that's twice as big, you will see it the same size as an object that's half as big, twice as far away. See what I mean? So a planet will stream out... Well, no, that's a bad example. Levski will stream out very far away, but a small spaceship like the Gladius will stream out much closer.
So the system's going to know that if you're approaching Levski, to bring Levski in at this far away, but if it's a Starfarer, to bring the Starfarer in at this far away, and if it's a Hornet, to bring the Hornet in at this far away. And at the end, when all is said and done, is that something handled by the servers or by the clients?
TL;DRBind culling unbinds far-away entities from the client, which breaks things like markers, entity pointers and scanning, so lightweight streamable markers were added and much gameplay code now asks the server for results.
So this is the act of unbinding the entities on the client when they go out of range. So if you're too far away from stuff, it basically streams it out, is the idea. There's quite a lot of implications of that for gameplay code, which I'm obviously slightly concerned with, being a gameplay programmer. So it's the type of thing of, if I'm on one side of the universe and I've got to go and kill someone on the other side of the universe, in the game just now you'd normally see a little objective marker, but that's not there because he doesn't exist on my client, because he has been unbound. Or simply if you have a comms call with someone who's too far away, he'll be unbound. Or similar things. And from a more technical standpoint, all our entity pointers themselves will, as soon as something goes out of range, become invalid. You can't talk to that entity anymore because it doesn't exist for you. You can't get his status, you can't get the amount of health he's got, because it no longer exists on your client. And you've got to be able to handle him leaving and then coming back again later on, so if you've got to fix stuff up, and if you've then got to rework your gameplay system to make it work with someone that's just come back. So what are the answers to those things? So for the likes of the markers, they were solved by having an independently streamable object. That's basically saying we've got this little marker, which is a really small lightweight thing. It's really far away from you, but the server's going to tell you that it has to exist on your client because you need to know some stuff about it. It's removed from the normal hierarchy. If you've got a marker, for instance, on a person that's inside a ship, normally you would need to have all these zones all the way down to get him. In this case it's just that marker, and it just gives you the zone position of the biggest zone that's still streamed in, so the planet, or the root zone if he's just in space somewhere.
I'm picturing like when you're playing golf and you pick up your ball and you put a quarter down in its place, so that you know where that thing's supposed to be. The ball isn't actually there anymore, but you know where it's going to be. And when it comes time for you to actually need to interact with it, it's going to know where.
Kind of. And it's also got the advantage that we can do things like, because it's so far away, the position doesn't need to update constantly, because you can't tell if it's 10 centimeters out. So it can update every 30 frames or every 60 frames or whatever, which also obviously increases performance ever so slightly.
Yeah, and we don't need to get the messages from the server as often, so it's not affected. And then once you get in range, once it gets fully bound, everything else has been unbound, so the network performance and stuff. And then obviously there's stuff I'm not as involved in, so the radar scanning stuff has got to be able to work when the stuff it's scanning doesn't exist on your client. So a lot of the game code was changed from everything happens on the client to the client has to ask the server for some results, or it has to push requests to the server for things that happen.
Well, even the likes of when Clive was mentioning how we decide if something streams out is based on how big it is on your screen. So if there's a Gladius that's 20 kilometers away, it's going to be one pixel on your screen. It can probably stream out, or if it's going to be less than a pixel it can stream out. There's some adjustments made to that for the likes of the radar, so the Gladius will be made to think it's bigger, because it needs to stay around, because although you can't see it, your radar can see it. So there needs to be some kind of adjustment there, so you're still able to get the up-to-date and the high-frequency updates to see where it is.
TL;DRSerialized variables are one of several interconnected technologies OCS needs, carrying the data the server tells the client it needs for things like markers so gameplay can use it almost seamlessly.
Yeah, I think one of the complexities of OCS is there's a whole interconnected bunch of technologies that need to work. So there's serialized variables that need to be there, there's the actual bind culling itself, there's... I've lost my train of thought now where I was going with this. Other technologies as well that only exist for it to work. The serialized variables are kind of the thing that almost, I don't want to say glues it all together because it's a bit of a clichéd phrase, but for the likes of the marker, you'll have serialized variables that have all the data you need in it. The server will be able to tell you the data you need to have in those variables, how often you need to get it, and it will do it almost seamlessly from a gameplay point of view. So we've just got the data we need to use.
So serialized variables are the mechanism that we expose to the gameplay programmers to allow them to communicate over the network. The game is built out of entities, entities are built out of components. Components have these serialized variables, which are values that can change over time as the gameplay programmers assign new values to them. Those changed values will get replicated over the network to the equivalent entity on your clients. It will change the value of the serialized variable there as though it's been changed exactly by the server. So it's a remote control over the state of entities in the game.
Yeah, it's almost exactly like that. We use serialized variables not just for networking, but because it's serialized... it doesn't necessarily need to be a network. You've got this serialization process that converts from, I suppose, the native format that the program's running with into a normal format, and it could be the network, it could be a save game file, it could be the persistent database. So we use serialized variables for those three things.
Not so much on the data itself. The way serialized variables save us... One way of updating the state of entities in a game over the network is to periodically, every so often, just send the state over to all the clients. And that can be inefficient because you're sending the same values over and over again and nothing's changed. So serialized variables track when something has changed and go, okay, this one now needs to be sent. These ones that haven't changed don't need to be sent. So that saves us a lot of bandwidth.
TL;DRFull client streaming means the server decides which entities a client knows about, combining distance-based streaming and asynchronous entity spawning, with bind culling replicating those decisions to clients.
It's kind of what we said already. I'm not sure what else we could say about it. So it's the server dictating what your client can see, what entities it's aware of, and as you move around, controlling and changing which entities are present on your machine.
Yes. So the difference between the two and how they fit together is: Object Container Streaming is something that we need for Squadron 42 as well, so a single player game. So it's very much the kind of traditional level streaming that we see in many games. So that controls figuring out the distances of entities, what needs to be present and what doesn't. At the minute on the server, everything's present all the time. So we've just got this distance calculation based on entities that exist, and then it can do the streaming in and out. The other part of OCS is the asynchronous entity loading. So up until now, anytime we spawn an entity it happens on the main thread, and the main thread will block for however long it takes to spawn that entity, and that includes the time it takes to load any files off the disk, so textures and physics geometry and anything else. Part of OCS is making that spawning process asynchronous, and by that we mean we tell it to start, and then when it finishes on a background thread we get the results back and the entity is finished and we can put it into the game. So those are kind of the two parts of OCS: determining what a client needs, and being able to asynchronously spawn entities and get rid of them. Bind culling comes in by replicating those server-side decisions over the network to clients, so that the streaming happens for them as well. So bind culling is server-side control of OCS on your clients.
TL;DRAsync loading, range decisions and bind culling each work, but networked spawning must be reworked so the client loads entities in the background and then tells the server when it's ready, instead of blocking.
So yeah, we've got the async entity loading working. We've got the range-based decision-making working. We've got bind culling working. It's now fitting those together, because what we currently have without asynchronous networked spawning is: the streaming manager will say, these entities are out of range for this client, you don't need them anymore. Bind culling will relay those messages, the client will drop them, and that's fine. But when you come into range of some entities and the client needs to spawn them, it will do a blocking spawn. It puts them all on the main thread. Your game grinds to a halt. So the part that we're left with now is changing the network messaging that handles spawning so that it can work with this asynchronous process, because the network messages as they currently stand assume this blocking synchronous spawn. As soon as I've said spawn, it will be spawned and I can start sending messages to that entity. But because it's an asynchronous process, we've got to load the files from the disk. It can take, depending on the number of entities we're trying to spawn...
Say a whole planet. It's going to take a few seconds to do. So we need the client to start spawning, do all its work, and when it's finished, go back to the server and say, I'm done now, can you send me some updates? And then the server can start communicating information to those new entities on the clients, and put them in their final state, because things have probably changed during the time it's taken. And then when all that process is finished, the client can finally go, right, I've got everything, I can put these entities into my local simulation and they can start interacting.
So we're a bit involved in entities dropping in and out. It's a lot of traveling around, but it's also finding all the different entities, how we can spawn them, move them, change the state for one player without changing it for another, bringing the players back together, and checking the states of everything match up. And we're looking for all the edge cases as well at the moment. Because everything's fine apart from stalling when you start loading up, say, Levski. There's a stall as it brings Levski in. Sorry, I'm losing my train of thought.
TL;DRQA checks that a player's client shows the correct state after leaving and returning, and is testing server load using headless "zombie" clients spread around the solar system, given the 50-player limit.
We'll be looking for things like, as you're dogfighting, you've got a mate in your cargo bay throwing a cargo crate out the back and you're blowing up Gordon at the same time. Another player, who, when he left, you were just dogfighting happily, comes back and everything's different. Does his client show everything as it should be? Is the cargo crate there in space? Is your cargo bay open? Is Gordon alive or dead? A lot of different states. And one of the things that I've been looking at myself this week is we need not just two people doing it. Two people seem to be quite easy to keep track of. What happens with the server when there are people or a group of people, not just out in space dogfighting, but you've got someone out in space, people off in Hurston? So we've been using headless clients, which are just low-weight clients that don't actually render the game, but we can give them basic commands. So we're filling the server...
Yeah, putting zombies in. They do basic things. They'll get out of their bed at Olisar. They'll use console commands to zip around the solar system. So we're seeing what's happening with the server now. What's the load like when there's someone everywhere, or nearly everywhere, because the player limit is 50 and it's a big solar system. So you can't get everywhere with 50 people right now.
TL;DRYes, it is daunting: it touches many teams, and making the roughly hundred components on the actor thread safe creates a deep dependency graph across audio, AI and other teams.
Yeah, I think the amount of teams it touches, even... One of the changes we've been having to make for the async spawning stuff: all the components of the game have had to be made thread safe so they can be spawned off the main thread. For the actor, that means all the actor's components first need to be made thread safe. So you've got a bunch of components that need to be done by the audio team, a bunch that need to be done by AI, a bunch that need to be done by all these different people, and some components depend on other ones being done first. So I can't convert, for instance, the local player component until the particle stuff was converted. So we've got this big dependency graph that sometimes goes several levels deep before you can actually get to it. And there's just an awful lot of stuff. There's a hundred-odd components on the actor now.
Yeah, so our game runs off an entity component system. So the entity is like an actor. It has an actor component, it has a local player component, it's got an audio component, it's got a variety of different things. There'll be ones for animation and ones for effects and all this plethora of different things, which will be shared between different entities as well, so there'll be other entities that also have animation components and stuff. But yeah, everything had to be able to be spawned.
I mean, this has been an ongoing thing for most of my last quarter, so Q2 was spent doing this for the actor. A big, big chunk of that was we had a scripting language called Lua that we use extensively in the game, and it basically had to go because it's not particularly thread safe. There was no good way to make it safe for spawning things asynchronously, so it was a case of going through... for the actor there were an awful lot of Lua files, and there was an awful lot of stuff that had just been around since this engine was created. It has evolved over the course of the engine. And yeah, it was just a lot of tidying up and cleaning up and moving things around. And again, it was by multiple departments as well.
I overheard someone today use the phrase 'truly amazing'. I don't want to oversell it. I know the backers have complaints about performance, quite legitimately. It's not at the state anyone wants it to be. But we're seeing really raw builds each day. Typically our performance can be pretty terrible. Everyone's reporting improvement. Some people are reporting drastic improvements. It's just a lot nicer to test when you're hitting a really nice smooth frame rate.
What we're seeing now is when you're in Port Olisar, you're not getting updates from Levski, Grim HEX, anything like that. So performance is great. You go to Lorville, your frame rate is still dropping very low because you're still loading Lorville and everything that's in it.
No, that's not. The performance benefits, the memory savings that you're going to get, if any, hedge bets, your mileage may vary, are going to be purely client side. Server side, all those entities still have to exist on the server. They need to be updated when anyone's near to them. So the server load is going to be heavier than it has been in the past, because we're increasing the scale of the solar system massively. Massive amounts of new content. I don't think it's likely we're going to increase the player counts.
Server-side streaming requires some extra technologies that we haven't got yet. So by working purely on client-side streaming, we're getting some of the benefit onto our clients, and we're not having to do all the technology at once. The kind of extra pieces that we need for server-side streaming: once a server decides to unload some entities, what happens to them? They're going to have to be persisted out in their current state, and then we're going to have to reload that state back in. It's not clear if we need full persistence to work, but it's going to have to be something pretty close to full persistence to get server-side streaming to work.
It shouldn't do. As Gordon said as well, the range at which entities get streamed out and streamed in should be pixel size or thereabouts, so you shouldn't notice anything. But because of the time it takes to load things in, there may be a little bit of popping, but it shouldn't affect the draw distance really.
TL;DRSo far it speeds up loading the game for developers, makes the much bigger map loadable on machines with less memory, and lets more game clients run on a single test machine.
Current improvements that we've already seen? From a development point of view, it takes me less time to load the game, which is nice. So if I'm wanting to test very simple stuff, the PU takes less time for me to load because it doesn't have to load all the stuff that's out of range.
It actually makes it possible to load the game as well, because the map is so much bigger now. Not everyone here has quite large memory machines, so some of them can't actually load the full map because it's that big. So they need bind culling on. Well, we've got two maps. We've got a cut-down map and the full map. They either have to work on the cut-down map, or if they want to load the full map, they have to turn bind culling on to be able to actually get it in, working on their mission. It's that much bigger.
The reduced memory is helping us as well. I mentioned earlier about filling a server up, which is effectively using one tester's time across multiple machines. The less memory the game is using, the more game clients we can put on a single machine.
There's no limit to how many containers can be nested inside each other. The solar system is built out of a root container, so we've got the Stanton system container. And then inside that, there's a container for each of the planets, I believe. And the planets themselves contain object containers for the moons and the space stations and the planet itself. And then on those, like for Grim HEX, there'll be the Grim HEX object container, and then inside Grim HEX there'll be object containers for the different shops. So I think we're probably getting on to about seven levels deep. But in theory, there's no limit.
I think you're going to get a different answer depending on who you ask. I'm extremely tempted to say networking, but I'm not. I'm going to say that it's probably the stuff Gordon was talking about, the conversion of entity spawning over to be thread safe and asynchronous. It's been a massive effort. And also the production side of things, because it's touched so many teams.
I think the fact that it touches everything means we need to test everything. And it's not just testing everything with bind culling, because when you find an issue on, say, the bind culling server, is it a bind culling issue? Sometimes you can't tell just from the symptoms of the issue alone. You go back to a regular server, you need to reproduce it there. So you're not just testing everything, you're testing everything twice, with it affecting things coming and going. These aren't tests you can usually do by yourself. You need two testers, preferably as many testers as you can get together. And we only have a finite amount, regardless of how many zombies I can... It's been a massive strain on our resources, getting everybody we need to be where we need to be, covering as many things as we can.
See, now you've said funny, all I can think of is crashes. There's been fun ones where you're trying to catch an AI who's decided to run away from you. I think this was earlier on. So he's in range, flying away, and you had an issue where he would reach outside of your streaming window and he'd stop. So then you'd catch up, and eventually you'd get the updates from this AI and he'd fly a bit further, because actually he's even further. So you're constantly having this little stop-start race against the guy, trying to catch up, and you don't even know where he really is.
I think mostly it's going to affect RAM, that's where you're going to get the biggest saving. It should, I'm being a bit more cautious than Chris, it should improve things on CPU; it probably won't have much effect on GPU.
I think from my experience over the last week or two, there will be CPU improvements probably, but we're also ramming a lot more stuff in. So while there may be some gains, we've also got a significantly larger amount of AI.
TL;DRQA says the only drawback is testing time, while Gordon adds that programmers can no longer assume every entity exists, so features must be designed with streaming in and out in mind.
You mentioned earlier some of the challenges involved, like the scanning, having to leave the markers when something is culled so that people can still see it.
Yeah, I don't want to generalise, but most gameplay programmers are quite happy to just assume everything exists all the time. I'd be very happy if I just had every single entity on the client at the same time, because it'd make my job a lot easier, whereas now, I've got to make sure every entity I'm accessing exists, and if it stops existing while I'm talking to it, what do I do? Lots and lots of more technical challenges. It just requires you to think about things, not that you don't think about things, but you've got to be very aware of the implications that a lot of stuff might stream out. If I'm designing this feature, will it work when not everyone's here? Do I need to move it to exist on the server rather than the client? And tweak things to make them work.
I'd say any bug that you spot, report it to the Issue Council anyway and allow us to sort it out. Probably the kind of things you're going to see are perhaps missing entities, or stalls as you approach planets or derelicts or whatever. What else might you see?
Some kind of popping as things appear. The likes of teleporting actors, that's one that's kind of floating around a little bit, so just seeing guys jump around the place. Occasionally we see issues with that.
Yeah, the background part of that says it's off the main thread. You can do asynchronous spawning on the main thread, but it's a different way of doing things. For us, it's a very precise way of describing the process.
TL;DRThe speakers stress that it took about three years, is a company-wide effort touching every part of the game, will keep being improved, and will cause edge-case breakage, so testers should submit crashes.
It's been, as you said earlier, three years since we started work on various parts of it. Although not everyone's been working on it flat out for those three years, it's taken a massive amount of time, a lot of changes, tons of people. It's not down to just us in this room or our teams in particular. It's a company-wide effort. It's been a long road and it's hopefully worth it when we finally get there.
Yeah, I think just to reiterate that it's touched every single element of the game, and it is still currently touching every single element of the game. Like I said, it's a collection of features, but there's never been a feature where there have been as many emails that have come out to absolutely everyone going, is this going to break anything for you? It's needed everyone to pitch in. I see a whole lot more 'BCC everyone' on OCS threads than on just about anything else. And I think as well, as Clive said, it's not been a short road. Before even Object Container Streaming, there had to be object containers themselves, and that started as I was starting, so that's three and a half years ago. So that was kind of the very beginning of it, and this has obviously been on the plan since then, and it's just taken three and a half years of tech and improvements.
No, this is the kind of thing that's going to be ongoing forever, probably. There's always going to be things to improve and more stuff that is added to it. The server streaming is obviously the next major step, and then there's stuff beyond that as well.
Just to reiterate that it touches everything, and from the QA standpoint... everywhere. Yeah, it touches everything. There's going to be a lot of edge cases that, with our department, as cool as everyone is, there's only so many of us. There's only so many Evocati, there's only so many PTU testers as well, and there are a lot of things that are going to break, I think. Please submit to the Issue Council and always submit your crashes.