Replication Layer Dying: What will players experience if a Replication Layer is shut down/'dies'? We know that the entity graph will collect the seeded information and feed it back into a new replication layer, but will we return to the main menu if the Replication layer dies compared to if a server node dies, or will we have some sort of loading screen that automatically match-makes us into a new layer?

To answer this properly, I first need to give some more detail on what our final architecture will look like. Ultimately, the Replication Layer won't be a single server node. Instead, it will consist of multiple instances of a suite of microservices with names like Replicant, Atlas, and Scribe. One advantage of this is that the Replication layer itself will be able to scale. Another advantage, more relevant to this question, is that although a single node/instance in the Replication layer may fail, it's very unlikely the whole Replication layer will fail at once. From a client's point of view, the Replicant nodes are the most important as it is those that will handle networked entity steaming and state replication between clients and the game. The Replicant is designed to not run any game logic and, in fact, it will run very little code at all; no animation, no physics, just network code. Being built from such a small codebase should mean fewer bugs overall. So, after some inevitable teething troubles, we're expecting Replicants to be pretty stable. It's also important to know that, at any one time, a single client may be served by multiple Replicants (but those Replicants will also be serving other clients at the same time). The final piece of the puzzle is the Gateway layer: Clients won't connect directly to Replicants but instead to a gateway node in the Gateway layer. The Gateway service is just there to direct packets between clients and the various Replicants they are talking to. The Gateway service will use an even smaller codebase than the Replicant so should be even less likely to crash.

So what will a client experience if one of the Replicants serving it suddenly crashes?

The client will remain connected to the shard but part or all of their simulation will temporarily freeze. The Replication layer will spin up a new replicant node to replace the one that crashed and will recover the lost entity state from persistence via EntityGraph. The client gateways and DGS nodes that were connected to the old replicant will re-establish connection with the new one. Once everything is reconnected, the game will unfreeze for the affected clients. At this point the client may experience some snapping/teleporting of entities. We're hoping the whole process will take less than a minute.

What will a client experience if the gateway serving it suddenly crashes?

The Gateway service doesn't hold any game state and will have its own form of crash recovery. Since it's a much simpler service than a replicant, the recovery time should be much quicker, more in the region of seconds. While the recovery is in progress, the client will experience a temporary freeze followed by some snapping/teleporting.

What about the Hybrid service?

During their CitizenCon presentation on Persistent Streaming and Server Meshing, Paul and Benoit talked about the Replication layer in terms of the Hybrid service. The Hybrid service is, as its name suggests, a hybrid of the Replicant, Atlas, Scribe, and Gateway services I mentioned above (but not EntityGraph), as well as a handful of other services not discussed yet. We have chosen to develop this first before splitting it into its component services as it reduces the number of moving parts we're trying to deal with all at once. It also allows us to focus on proving out all the big concepts rather than the boilerplate of having all those individual services communicate correctly. In this initial implementation, the Replication layer will be a single Hybrid server node. If this Hybrid node crashes, then the situation will be similar to what clients experience now when a dedicated game server crashes. All clients will get kicked back to the frontend menu with the infamous 30k error. Once the replacement Hybrid has started, clients will be able to rejoin the shard and continue where they left off. Hopefully, we'll be able to implement it such that the clients receive an on-screen notification that the shard is available again and a single keypress will match them back to the shard (similar to how it works for client crash recovery).