Back to Search
What are we testing for now [in Server Meshing]? Why has the gap been so long between tech preview tests for this? And is this really working as well as we hope?

So yeah, later in the spring, end of spring, we did a couple of good tests that gave us a lot of good data, right? So what we've been trying to do on the meshing side is be a bit more pragmatic and methodical in how we approach the development. And so these tests really gave us a lot of data, right? Because of the way we conducted them, we were able to try different mesh configurations, look at performance specifically on the hybrid, which is the replication layer of service. And to really be able to hone in on what we need to improve in order to get the scale that we want. And so what came out of these tests in the spring was that there was already things that were touching on the performance of the hybrid and the game server communications that needed improvement. And so this is what we call the components called the NMQ, the network message queue. Now there's a lot of chatter online about, you know, there's a replacement thing and it's called, is it RabbitMQ, whatever. No, so we're not even in that realm. We're talking about really the networking component that transmits data between game server and client, game server and hybrid, hybrid and game servers. It's like this, think of it as the communication library underneath it. So like all games, that technology is using UDP transport to get, you know, eventually consistent updates out as fast as possible. Now, the network message queue had a couple of issues when it got a lot of data in a very short period of time. So these spikes would cause, and that's what we saw in the final server meshing tech previews, was that when large areas would stream in and a lot of players would join in and leave at the same time, the network message queue could go into a tailspin where, think of it as a drain, right? And you're trying to put stuff in the drain. And at some point, if you exceed the bandwidth of the drain, then water starts filling up. Now, the way that manifests itself in the game is interaction delays. So, you know, pressing F trying to get in the ship and it takes seconds before the system reacts, powering on ship takes forever. You know, now I'm not talking about the one offs because you can have interaction delay in your game based on your connection. That can absolutely happen. So it's influenced by your machine, your connection. So if you're playing on a 486 without the turbo button on a satellite connection, you know, you might get interaction delay, right? But in this case, we're talking about widespread, like when an entire shard gets into that state, right? So that's what we saw in the tech preview. So that we had to go a little bit back on the drawing board and actually fast track one of the refactors we wanted to do, which is the introduction of a replacement message queue. Now we call it replication message queue, replacement message queue kind of, you know, we've basically called it RMQ. And so this RMQ is a new implementation of this that has more abilities to adapt to higher bandwidth and react to these large message pushes that happen when we start to stream in large areas or when a lot of players are coming in and out of the game rapidly. And so what we're testing now is RMQ in a server mesh setting. So for you guys who don't know, 3.24, half of the servers on live currently run RMQ. The other ones are running NMQ still. We've been gathering data across both of them to see the performance improvements, the gains. So far, those have been really good, a lot of really good improvements. And so now we want to test this in a server mesh setting. So what changes in a server mesh setting is that now the hybrid is going to manage more than one game server that simulate these key areas. And so there's more traffic going through the hybrid for this at key moments. And so that's what we want to test with this next set of tech previews. So what our hope is, is that the issues that plagued the 800 players last time where there was a lot of interaction delays. Also, some of those tests had a lot of game server crashes at the time that we did the test. But in this case, we're hopeful that all these interactions delays will be basically gone from this bandwidth side of it. So that means we can get to bigger shards and the experience is going to be enjoyable across that. The thing that actually did work really well in our spring test was the actual server meshing tech. So authority transfer, management of territories, all that stuff was fantastic. A really big success. But we did need to take a little bit of a sideways step to optimize the RMQ, which is now done. And also getting mission tested right now by you guys on 3.24.

-
Open in pop-up
Open in new tab
Click to copy link to YouTube
|
Published on this date.2024-09-06
Save Card as Image
ID: aab3cf72dd12c_dashboard_generated_id (Click to copy)
Perma link (Click to copy)
All Q&As from this episode