Chad McKinney · Core Tech DeveloperiCache is effectively shorthand for a collection of services that are interfacing with the game servers and backend databases to allow us to both write out and query data efficiently and in a fault tolerant way. So this is really important because right now the way the servers are working, they're using an in-memory kind of database for this global data. Ultimately, that's not feasible. We need to build it out so that things can work in a mesh. And in order for that to happen, we need to actually write this out into a place that all the servers can look at it and agree on what the state of the world is. So the iCache is our solution to make that access scalable. And right now, the iCache infrastructure on the back end is actually largely done. Now, at the point where game side, we're hooking up our game code in order to correctly write out the data efficiently, acquire the data efficiently, as well as fix up some of the game code systems that had previously maybe had some assumptions that don't work correctly within the new context. So that's kind of where we're at at the moment. Effectively, what we're trying to do is cache certain amounts of certain data that we know that we need, and this prevents us from hitting the database, which is a more expensive access. So by caching it, we're optimizing our access, and we're also able to build in a certain amount of fault tolerance in this system through various techniques like duplication and regeneration. So this gives us fast access for the information that we need when we need it.
Chad McKinney · Core Tech DeveloperSo it's effectively like you mentioned, the difference between having a hard disk where the data is stored for long-term storage that's more permanent, and your RAM that effectively gives you faster access that you're happier to write to, but you know that it's temporary. So the iCache storage is storage that's not meant to actually capture and define the state of the world, but rather is an intermediary between the final state of the world, which is the actual database.
Chad McKinney · Core Tech DeveloperYeah, we've been heading in this direction for actually years. So whenever we started on client-side streaming, we knew at that point, actually before that, that we would be ending up here and then the ultimate destination was, spoiler alert later, server meshing. So I think we've made tremendous progress and we've done some rather transformative changes in some of the underlying game engine code that I think is amazing. So I think that we've made a lot of progress. I want to say that we're closer to the end than the beginning, is how I would put it.
Chad McKinney · Core Tech DeveloperSo I would say that some of the most apparent changes that will come about is that we're going to have both a combination of better response on server and service queries for this kind of persistent data, as well as better fault tolerance and scalability. What that will directly manifest as: all the different times in the game where you end up interacting with these databases, which is constantly, it'll be things like when you spawn a ship, whenever you change your loadout, so your helmet or your armor, whenever you change something about your vehicles, whenever you purchase something, whenever you deal with cargo, there's all these places in the game where you're interacting with the persistent state of the game. And in all those places, right now we're struggling both with scaling to support the number of players that we want, especially whenever we have a big event or new release, we're really pushing on the current infrastructure that we have. But on top of it, it'll make it more fault tolerant so that whenever inevitably something goes wrong, the game just doesn't fall over. With the current, what we call the monolithic PCache that we have, if that service goes down, that's just it for all persistence queries for the entire, not just server, but all the servers. So this is clearly not the right place to be. But this new set of services addresses that by not putting all your eggs in one basket, as it were, and allowing for a multiplicity of handlers to be able to react to any requests.
Chad McKinney · Core Tech DeveloperRight, so global persistence is something that is supported now that we have this iCache framework. What it means is that we're going to start writing out data about the state of the world that does not include player account data. Previously, we would only store data that was relevant to a particular player's account. Things like their ships, their weapons, their item port attachments. Perhaps you stole something from somebody, we'll store that. But that's where it stops. We want to do more though. The game that we envision is this incredibly high fidelity, highly persistent world where your actions matter, you have an impact on the world. Well, the only way to make that work is to actually store out the state of those changes and to store out the state of the world itself. So that's what I've been calling global persistence. So the reason why we haven't done this to date, though, is that the amount of data that we're talking about is a magnitude or two more. It's much, much more data. And if we tried to write that out with the current system, it would just keel over and die. It would not go well. So iCache gives us the scalability to actually deal with that scale of data. But on top of that, it's a requirement to support server meshing. Before we get to server meshing, global persistence really is, in my opinion, a feature in and of itself, and it's part of the vision of the ultimate version of Star Citizen. So to me, it was very important that we make sure that we execute on this, that we're finding all the places in the game that we want to support this kind of persistent state and make sure that we update all the game systems to work accordingly. So we actually have a version of global persistence right now. It's just not apparent to players because when they log out of a server and log back in, they'll go into a different server instance often and they'll see something different. And if the server that they're on crashes and recovers, the history that it had is gone. So we actually are maintaining much of this data, but it's just not in a way that players will really ever tangibly see. And where we're going is making it more permanent and on top of that recoverable. So that as you get into the game, we're going to be able to expand out using server meshing the number of players within effectively an instance or what we'll start calling a shard. And as that expands, you'll be playing more and more in effectively the same kind of world line or set of history. And your experience will be that you will see you and your playgroups, your effects on the world become more permanent. And the idea is that for all the players in the game, they end up seeing a version of the game that is effectively permanent for them. From the time they start playing the game to the time that they end, everything is there. And to be clear, though, there are going to be sets of gameplay around maybe dynamic events in the game that happen or there will be places in the game where maybe there's more maintenance that will recover some of the broken and messiness that accrues from all this player behavior. You can imagine that if you have maybe a military base or maybe a high traffic, high cost neighborhood in Area 18 or something like that, they're gonna have more facilities in place to maintain it, to keep it clean and fixed up. Whereas maybe a seedier location perhaps in pirate space, all the mess that's left around is gonna stick around for a while.
Clive Johnson · Network DeveloperWell, currently, when backers play the game, they connect to a server, and it's a single server holding up to 50 clients running the entire simulation on that one server. That's going to be a severe limitation because obviously there's a finite amount of processing available on one computer. So there's a finite number of clients that it can support. So as we want to extend the number of concurrent users, the number of active users in the same simulation at the same time, we need to split that simulation up over multiple servers. So the original idea is to have multiple servers running different parts of the simulation. So you might have a server running the game at Port Olisar, you might have a different server running the game at say Levski. And that's great, they're completely independent. But the problem then becomes how do you get from Port Olisar to Levski or back again if the simulation is running on different servers. So we need to connect the servers together. And what that basically means is we'll have all these servers running different parts of the simulation and they will selectively choose which other servers to connect to so that players can transition back and forth or that they can share parts of the simulation together. And then they'll drop that connection when it's no longer needed and they can make new connections with other servers.
Clive Johnson · Network DeveloperWell, right now, we're working on a couple of things. We're working on getting the time synchronization correct, which probably doesn't sound very server meshing related. But we've been aware for a while that there's been lag, rubber banding, kind of desync issues present in the main game. Obviously, with all these servers connected to each other in the server mesh, that's going to make the problem worse. That's the thinking. So we kind of need to address it a bit now. We can't promise that we're going to completely solve it first go, but we're investigating the situation and trying to make as many improvements as we can. And a lot of the issues seem to come back to time synchronization between the clients and the server. And in server meshing, just to remind everyone, servers will be connected to each other. So one server will be acting as a client of another and receiving updates from it. They could, too, experience rubber banding and desyncs and so on. So we've got to try and make sure that that clock synchronization is right. So it's not just about getting the clocks actually in sync. It's more fundamental than that. Some of the problems we've found involve changing how the engine actually handles the concept of time, which sounds quite grand, maybe. A load of bugs that have crept in over the years. It's quite tricky. It's quite detailed work, but we're getting through it. We're seeing some good results now. Hopefully that will be done soon.
Clive Johnson · Network DeveloperAnd then we're also looking at getting two servers talking to each other, which may surprise some people that that's not the first thing that we do. But it kind of was, because obviously the first thing you try when you try and build a server mesh, you just try and connect two servers together and you see what happens. And we weren't really expecting it to work and we weren't too surprised when it didn't. But it was quite a useful exercise and it showed kind of what didn't work and what we would have to change to make it work. So we kind of built those tasks out and people are working through them right now. And we're getting pretty close to where we can give it another go and have a reasonable shot at the two servers actually being able to communicate. So the fundamental problem there is that the engine's based on the assumption that a server thinks it owns the game session, whereas with server meshing, obviously, they're sharing the game session between them. So we've kind of got to make one server set up and it kind of has the game session and then another server wants to join it and it goes, okay, yeah, you can share into it. You now have as much authority and ownership of this game session as I do. It's a bit of a shift for how the game code and the network code thinks about game sessions and how everything relates to each other. So throughout the game code, well, throughout the engine, there's plenty of checks going, if this is a server, do something like this, if it's a client, do something like this, which doesn't fit with the server meshing model where you've got an authoritative server, you've got a non-authoritative server, and you've got regular game clients. So all these kind of if statements were based on the assumption that you could only ever have one authoritative server. So they all needed to change. We're talking about thousands of locations in the code. So we kind of did an audit. We split that up into hundreds of tasks for people to look at. That got spread out across basically every team that has an engineer in it. And they've been working through it. We're nearly at the end, there's a few stragglers left to do. But nearly at the end, we'll have changed it over to this new API that basically says, am I the machine that has authority over this entity? If I am, I can do it this way. If not, I'm supposed to do this instead. So that was a massive change. And that in itself is getting put out into the new builds and you can't see any changes. And that's a good thing because it shows that they're not breaking stuff by making these changes. So it's an early test. I think we're doing okay.
Clive Johnson · Network DeveloperSo the idea was to try and get as many of the tasks, as much of the work done early, where it might have a knock-on to the team. So like changing how other people write code to decide whether something should be handled authoritatively or not, how they might handle time, any of those kind of changes you want to do as early as possible. So from this point on, people are writing code the correct way and you've got less to go back and kind of revisit. And because there's such a huge amount of code that's already been written, we were kind of expecting it to take a while for those changes to get worked through. So while that's going on, the network team are kind of working on getting the servers connecting to each other. We've also done some other changes to headless clients and other things to try and allow them to be used for automated testing so that when we get these server meshes up and running, we can automatically test them rather than having QA try and fill these things up. So there's a lot of different tracks all kind of progressing in parallel. And we're making progress, but it's not easy.