Clive Johnson

Well, currently, when backers play the game, they connect to a server, and it's a single server holding up to 50 clients running the entire simulation on that one server. That's going to be a severe limitation because obviously there's a finite amount of processing available on one computer. So there's a finite number of clients that it can support. So as we want to extend the number of concurrent users, the number of active users in the same simulation at the same time, we need to split that simulation up over multiple servers. So the original idea is to have multiple servers running different parts of the simulation. So you might have a server running the game at Port Olisar, you might have a different server running the game at say Levski. And that's great, they're completely independent. But the problem then becomes how do you get from Port Olisar to Levski or back again if the simulation is running on different servers. So we need to connect the servers together. And what that basically means is we'll have all these servers running different parts of the simulation and they will selectively choose which other servers to connect to so that players can transition back and forth or that they can share parts of the simulation together. And then they'll drop that connection when it's no longer needed and they can make new connections with other servers.

Well, right now, we're working on a couple of things. We're working on getting the time synchronization correct, which probably doesn't sound very server meshing related. But we've been aware for a while that there's been lag, rubber banding, kind of desync issues present in the main game. Obviously, with all these servers connected to each other in the server mesh, that's going to make the problem worse. That's the thinking. So we kind of need to address it a bit now. We can't promise that we're going to completely solve it first go, but we're investigating the situation and trying to make as many improvements as we can. And a lot of the issues seem to come back to time synchronization between the clients and the server. And in server meshing, just to remind everyone, servers will be connected to each other. So one server will be acting as a client of another and receiving updates from it. They could, too, experience rubber banding and desyncs and so on. So we've got to try and make sure that that clock synchronization is right. So it's not just about getting the clocks actually in sync. It's more fundamental than that. Some of the problems we've found involve changing how the engine actually handles the concept of time, which sounds quite grand, maybe. A load of bugs that have crept in over the years. It's quite tricky. It's quite detailed work, but we're getting through it. We're seeing some good results now. Hopefully that will be done soon.

And then we're also looking at getting two servers talking to each other, which may surprise some people that that's not the first thing that we do. But it kind of was, because obviously the first thing you try when you try and build a server mesh, you just try and connect two servers together and you see what happens. And we weren't really expecting it to work and we weren't too surprised when it didn't. But it was quite a useful exercise and it showed kind of what didn't work and what we would have to change to make it work. So we kind of built those tasks out and people are working through them right now. And we're getting pretty close to where we can give it another go and have a reasonable shot at the two servers actually being able to communicate. So the fundamental problem there is that the engine's based on the assumption that a server thinks it owns the game session, whereas with server meshing, obviously, they're sharing the game session between them. So we've kind of got to make one server set up and it kind of has the game session and then another server wants to join it and it goes, okay, yeah, you can share into it. You now have as much authority and ownership of this game session as I do. It's a bit of a shift for how the game code and the network code thinks about game sessions and how everything relates to each other. So throughout the game code, well, throughout the engine, there's plenty of checks going, if this is a server, do something like this, if it's a client, do something like this, which doesn't fit with the server meshing model where you've got an authoritative server, you've got a non-authoritative server, and you've got regular game clients. So all these kind of if statements were based on the assumption that you could only ever have one authoritative server. So they all needed to change. We're talking about thousands of locations in the code. So we kind of did an audit. We split that up into hundreds of tasks for people to look at. That got spread out across basically every team that has an engineer in it. And they've been working through it. We're nearly at the end, there's a few stragglers left to do. But nearly at the end, we'll have changed it over to this new API that basically says, am I the machine that has authority over this entity? If I am, I can do it this way. If not, I'm supposed to do this instead. So that was a massive change. And that in itself is getting put out into the new builds and you can't see any changes. And that's a good thing because it shows that they're not breaking stuff by making these changes. So it's an early test. I think we're doing okay.

So the idea was to try and get as many of the tasks, as much of the work done early, where it might have a knock-on to the team. So like changing how other people write code to decide whether something should be handled authoritatively or not, how they might handle time, any of those kind of changes you want to do as early as possible. So from this point on, people are writing code the correct way and you've got less to go back and kind of revisit. And because there's such a huge amount of code that's already been written, we were kind of expecting it to take a while for those changes to get worked through. So while that's going on, the network team are kind of working on getting the servers connecting to each other. We've also done some other changes to headless clients and other things to try and allow them to be used for automated testing so that when we get these server meshes up and running, we can automatically test them rather than having QA try and fill these things up. So there's a lot of different tracks all kind of progressing in parallel. And we're making progress, but it's not easy.

That's right. That's not something we're working on for 3.3. That'll come further down the line. But because the solar system is going to get even more densely populated, with more planets and so on, even the servers, with the massive amount of memory that they have, won't be able to keep everything in place at the same time. So they'll need to stream in and out locations based on demand. So when a player goes to a location, or as he's heading towards a location rather, the server will detect that and go, okay, I need to stream in the entities that are there, the planet and derelicts and whatever are around. So by the time the player reaches that location, it's all in memory on the server. The server has to coordinate that with the client itself and tell the client what it needs to stream in, so the entities are present on the client as well. And then when the player leaves that location, the server's free to unload that information and free the memory up for anything else.

Well, map streaming is generally more common in single player games. A lot, not all, but a lot of multiplayer games will confine the maps to smaller sizes so that it can all fit in memory all at once. So once you've loaded in, all the data is resident on your client, no streaming's required, and that cuts down the chance of any stalls or any frame rate hits and makes for a smoother online experience. So the challenge is to do streaming during a multiplayer game and not have any stalls or performance hits. And it's also, as I say, about the scale that we're doing this on. Obviously, our maps are millions of times bigger than the maps of virtually any other game. So it's a question of scale and trying to do it while we're online as well.

So it's not that we're necessarily reinventing the wheel here, it's that we're adapting the wheel to a scale that's previously unheard of.

Yeah, we're probably one of the bigger games ever envisioned at this point. We've done quite a lot of stuff over the last couple of years where we've had snippets of gameplay, Star Marine, where it's been a small contained bubble of the game, and we're now going, right, it's now that, but it's in this big massive universe that's making everything bigger. And by going, well, if we only need to stream in this one bit where the FPS is happening, the performance should be the same as Star Marine, probably, on that point.

It's not a silver bullet, as you mentioned before, so we can't make any guarantees on how performance is going to improve, partly because there are so many variations of CPUs and motherboards and RAM and everything else and GPUs that you can't predict how it's going to affect all the different combinations. So it's a best guess of what we think is actually going to help and what it's going to look like. So effectively, the difference to what you have now is: you start off in your bed at Port Olisar and Yela is loaded on your client, and there are entities on your client in memory, hopefully idle and not doing anything if the in-range update is working correctly. The difference will be that with OCS you'll load into Port Olisar and basically Port Olisar and a few entities for the star map will be loaded, and that's it. Any players and ships that are around your immediate vicinity will be loaded in. Anyone who quantum travels to your current location will dynamically be loaded in. Anyone who quantum travels away will be loaded out and your client won't have any awareness of them anymore. So it cuts down the amount of memory that your client needs. It should cut down some of the performance overhead for maintaining all these entities in memory as well. And as you move around the solar system, you'll unload areas that you don't need anymore and load in the new ones. So you head off to Levski, stream Levski in for the first time, and that's the first time your client is aware of those entities in that location.

Pretty much, yeah. It's not exactly a hard bubble. It's not a defined distance. It's actually the size the object will appear to you if you were to look at it. So an object that's twice as big, you will see it the same size as an object that's half as big, twice as far away. See what I mean? So a planet will stream out... Well, no, that's a bad example. Levski will stream out very far away, but a small spaceship like the Gladius will stream out much closer.

So the system's going to know that if you're approaching Levski, to bring Levski in at this far away, but if it's a Starfarer, to bring the Starfarer in at this far away, and if it's a Hornet, to bring the Hornet in at this far away. And at the end, when all is said and done, is that something handled by the servers or by the clients?

That's handled entirely by the servers.

It's kind of what we said already. I'm not sure what else we could say about it. So it's the server dictating what your client can see, what entities it's aware of, and as you move around, controlling and changing which entities are present on your machine.

All right, so bind culling is the tech of dropping things out and everything.

Yes. So the difference between the two and how they fit together is: Object Container Streaming is something that we need for Squadron 42 as well, so a single player game. So it's very much the kind of traditional level streaming that we see in many games. So that controls figuring out the distances of entities, what needs to be present and what doesn't. At the minute on the server, everything's present all the time. So we've just got this distance calculation based on entities that exist, and then it can do the streaming in and out. The other part of OCS is the asynchronous entity loading. So up until now, anytime we spawn an entity it happens on the main thread, and the main thread will block for however long it takes to spawn that entity, and that includes the time it takes to load any files off the disk, so textures and physics geometry and anything else. Part of OCS is making that spawning process asynchronous, and by that we mean we tell it to start, and then when it finishes on a background thread we get the results back and the entity is finished and we can put it into the game. So those are kind of the two parts of OCS: determining what a client needs, and being able to asynchronously spawn entities and get rid of them. Bind culling comes in by replicating those server-side decisions over the network to clients, so that the streaming happens for them as well. So bind culling is server-side control of OCS on your clients.

So yeah, we've got the async entity loading working. We've got the range-based decision-making working. We've got bind culling working. It's now fitting those together, because what we currently have without asynchronous networked spawning is: the streaming manager will say, these entities are out of range for this client, you don't need them anymore. Bind culling will relay those messages, the client will drop them, and that's fine. But when you come into range of some entities and the client needs to spawn them, it will do a blocking spawn. It puts them all on the main thread. Your game grinds to a halt. So the part that we're left with now is changing the network messaging that handles spawning so that it can work with this asynchronous process, because the network messages as they currently stand assume this blocking synchronous spawn. As soon as I've said spawn, it will be spawned and I can start sending messages to that entity. But because it's an asynchronous process, we've got to load the files from the disk. It can take, depending on the number of entities we're trying to spawn...

Say a whole planet.

Say a whole planet. It's going to take a few seconds to do. So we need the client to start spawning, do all its work, and when it's finished, go back to the server and say, I'm done now, can you send me some updates? And then the server can start communicating information to those new entities on the clients, and put them in their final state, because things have probably changed during the time it's taken. And then when all that process is finished, the client can finally go, right, I've got everything, I can put these entities into my local simulation and they can start interacting.

Yes, definitely. I personally haven't worked on that side of things.

Yeah, I think the amount of teams it touches, even... One of the changes we've been having to make for the async spawning stuff: all the components of the game have had to be made thread safe so they can be spawned off the main thread. For the actor, that means all the actor's components first need to be made thread safe. So you've got a bunch of components that need to be done by the audio team, a bunch that need to be done by AI, a bunch that need to be done by all these different people, and some components depend on other ones being done first. So I can't convert, for instance, the local player component until the particle stuff was converted. So we've got this big dependency graph that sometimes goes several levels deep before you can actually get to it. And there's just an awful lot of stuff. There's a hundred-odd components on the actor now.

Current improvements that we've already seen? From a development point of view, it takes me less time to load the game, which is nice. So if I'm wanting to test very simple stuff, the PU takes less time for me to load because it doesn't have to load all the stuff that's out of range.

So you can iterate between builds faster.

It actually makes it possible to load the game as well, because the map is so much bigger now. Not everyone here has quite large memory machines, so some of them can't actually load the full map because it's that big. So they need bind culling on. Well, we've got two maps. We've got a cut-down map and the full map. They either have to work on the cut-down map, or if they want to load the full map, they have to turn bind culling on to be able to actually get it in, working on their mission. It's that much bigger.

The reduced memory is helping us as well. I mentioned earlier about filling a server up, which is effectively using one tester's time across multiple machines. The less memory the game is using, the more game clients we can put on a single machine.