The biggest factors currently affecting latency are server tick rate, client ping, entity spawning, and the latency of persistent services.

Server tick rate has the biggest effect of these and is related to the number of locations a game server is simulating. Server meshing should help with this by reducing the number of locations each game server needs to stream in and simulate. Fewer locations will mean a much lower average entity count per server and the savings can be used to increase the number of players per server.

Client ping is dominated by distance from the server. We see many players choosing to play on regions in entirely different continents. Some of our game code is still client authoritative, which means that players with high ping can adversely affect the play experience for everyone else. There's not much we can do about this in the short term but it's something we want to improve on after Server Meshing is working.

Slow entity spawning can cause latency by delaying when entities stream in on clients. This can cause undesirable effects, such as locations not fully appearing until minutes after quantum traveling to a location, falling through floors after respawning at a location, ships taking a long time to appear at ASOP terminals, changing player loadout, etc. The bottlenecks with this are mostly on the server. First, entities don't get replicated to clients until they have been fully spawned on the server. Second, the server has a single spawn queue that it must process in order. Third, the more locations a server needs to stream in, the more spawning it has to do. To improve things, we have modified the server spawning code to make use of parallel spawn queues. Server meshing will also help, not only by cutting the load on spawn queues by reducing the number of locations a server has to stream in, but also because the Replication layer replicates entities to clients and servers simultaneously, allowing them to spawn in parallel.

We're still using some of our legacy persistent services, adequate as designed but known to have performance and scalability issues under our demands. This can result in long waits when fetching persistent data from the services in order to know what to spawn, such as spawning a ship from an ASOP terminal, examining an inventory, changing player loadout, etc. Since full persistent streaming and Server Meshing will both dramatically increase the amount of data we need to persist, we knew we needed to do something about this. This is why Benoit and his team at Turbulent have completely reinvented how we will persist data in the form of EntityGraph, which is a highly scalable service built on top of a highly scalable database that is optimized for exactly the kind of data operations we perform. On top of that, we're also developing the Replication layer, which acts like a highly scalable in-memory cache of the current state of all entities in a shard, eliminating the need for the majority of queries we've been sending to the legacy persistent services. That's right, it's going to be highly scalable services all the way down!

To help reduce/eliminate any additional latency the Replication layer may introduce, we're building it to be event-driven rather than on a tick rate like a traditional game server. This means that as packets come in, it will immediately process them and send out the response and/or forward the information to relevant clients and game servers. Once work on the initial version of the Replication layer is complete (the Hybrid service), we'll be doing an optimization pass to make sure it's as responsive as possible. And, although this is ultimately a decision for DevOps, we'll deploy them in the same data centers as the game servers themselves so the on-the-wire network latency due to the extra hop between the Replication layer and game server should be less than a millisecond. Oh, and did I mention the Replication layer will be highly scalable? That means if we detect the Replication layer causing latency hotspots in particular parts of the 'verse, we will be able to reconfigure it to remedy the problem.