Where are we at right now with server meshing?

Well, right now, we're working on a couple of things. We're working on getting the time synchronization correct, which probably doesn't sound very server meshing related. But we've been aware for a while that there's been lag, rubber banding, kind of desync issues present in the main game. Obviously, with all these servers connected to each other in the server mesh, that's going to make the problem worse. That's the thinking. So we kind of need to address it a bit now. We can't promise that we're going to completely solve it first go, but we're investigating the situation and trying to make as many improvements as we can. And a lot of the issues seem to come back to time synchronization between the clients and the server. And in server meshing, just to remind everyone, servers will be connected to each other. So one server will be acting as a client of another and receiving updates from it. They could, too, experience rubber banding and desyncs and so on. So we've got to try and make sure that that clock synchronization is right. So it's not just about getting the clocks actually in sync. It's more fundamental than that. Some of the problems we've found involve changing how the engine actually handles the concept of time, which sounds quite grand, maybe. A load of bugs that have crept in over the years. It's quite tricky. It's quite detailed work, but we're getting through it. We're seeing some good results now. Hopefully that will be done soon.

And then we're also looking at getting two servers talking to each other, which may surprise some people that that's not the first thing that we do. But it kind of was, because obviously the first thing you try when you try and build a server mesh, you just try and connect two servers together and you see what happens. And we weren't really expecting it to work and we weren't too surprised when it didn't. But it was quite a useful exercise and it showed kind of what didn't work and what we would have to change to make it work. So we kind of built those tasks out and people are working through them right now. And we're getting pretty close to where we can give it another go and have a reasonable shot at the two servers actually being able to communicate. So the fundamental problem there is that the engine's based on the assumption that a server thinks it owns the game session, whereas with server meshing, obviously, they're sharing the game session between them. So we've kind of got to make one server set up and it kind of has the game session and then another server wants to join it and it goes, okay, yeah, you can share into it. You now have as much authority and ownership of this game session as I do. It's a bit of a shift for how the game code and the network code thinks about game sessions and how everything relates to each other. So throughout the game code, well, throughout the engine, there's plenty of checks going, if this is a server, do something like this, if it's a client, do something like this, which doesn't fit with the server meshing model where you've got an authoritative server, you've got a non-authoritative server, and you've got regular game clients. So all these kind of if statements were based on the assumption that you could only ever have one authoritative server. So they all needed to change. We're talking about thousands of locations in the code. So we kind of did an audit. We split that up into hundreds of tasks for people to look at. That got spread out across basically every team that has an engineer in it. And they've been working through it. We're nearly at the end, there's a few stragglers left to do. But nearly at the end, we'll have changed it over to this new API that basically says, am I the machine that has authority over this entity? If I am, I can do it this way. If not, I'm supposed to do this instead. So that was a massive change. And that in itself is getting put out into the new builds and you can't see any changes. And that's a good thing because it shows that they're not breaking stuff by making these changes. So it's an early test. I think we're doing okay.

So the idea was to try and get as many of the tasks, as much of the work done early, where it might have a knock-on to the team. So like changing how other people write code to decide whether something should be handled authoritatively or not, how they might handle time, any of those kind of changes you want to do as early as possible. So from this point on, people are writing code the correct way and you've got less to go back and kind of revisit. And because there's such a huge amount of code that's already been written, we were kind of expecting it to take a while for those changes to get worked through. So while that's going on, the network team are kind of working on getting the servers connecting to each other. We've also done some other changes to headless clients and other things to try and allow them to be used for automated testing so that when we get these server meshes up and running, we can automatically test them rather than having QA try and fill these things up. So there's a lot of different tracks all kind of progressing in parallel. And we're making progress, but it's not easy.