What is the current status of Server-Side Object Container Streaming (SSOCS)?

They had a big summit last year to figure out what work needed to be done to implement SSOCS, and have been working on it ever since, which is almost a year now. The meeting that Chad was in that made him late to this episode was a meeting with CR about their progress, of which the mood is pretty positive because they've made "very meaningful progress". They've made some large changes to Object Containers in terms of how they load entities, separating them into Dynamic Entities and Static Entities, where the former are streamable but the latter aren't. They've made a Star Engine Service, which is written similar to game code but can be deployed as a cloud service. Using this they built the Star Hash Service, which is a kind of coordinate system across the universe (like geohashes), allowing them to partition the universe and know what's streaming when, as well as being able to store those things and look them up. They built a service to keep track of Entity Aggregates in the universe, "which was kind of the roots of entity trees". The Star Hash Service has also been hooked up to the Streaming Manager, which was previously created to govern CSOCS, so it knows if certain aggregates are streamed out, or whether to stream certain aggregate groups in. They've also implemented a Prototype In-Memory Persistence Model, which allows them to write out the state of any Star Hashes to a local cache whenever they're streamed out, so that they can test game code (like service beacons) to see what breaks. For SSOCS to work they need to implement Global Persistence, which completely changes how the game runs, and how entities are spawned and loaded. They've had to revisit existing gameplay features to make sure they aren't broken, which they've done. They next have to implement the real backend for persistence, because Global Persistence will require them to be able to handle a much higher volume of data, which is a Work In Progress. The last big thing is the changes to game code that are needed, which requires all game feature teams to revisit their features. They have unknown unknowns too, because they need to test heavily to know what work will need to be done. They're also considering phasing out SSOCS so it won't need Global Persistence for it's first release, because their Prototype Model is working well. The Player won't notice much difference, but there should be some streaming benefits from this. Server Meshing is a similar situation, but SSOCS is a prerequisite so there's only so much they can do before SSOCS is completed.
The engine they had was CryEngine, and now they have Lumberyard, both of which are level-loading based engines, so there's no entity streaming in them as-is. Years ago they made some large changes to the engine called Object Containers, which is their own grouping of entities as well as level data like brushes. They planned years in advance where they wanted to get to, both now and in the future, which was a huge push for foundational changes. They built on top of Object Containers with Net Bind Culling and Serialised Variables, allowing them at runtime to be able to replicate and unbind entities from Clients. Then last year they deployed Client-Side Object Container Streaming (CSOCS), which is taking the Object Containers at runtime and streaming them in and out, basically unbinding and binding Object Containers. That alone was a multi-year effort.
Now they're doing the same thing on the server (Server-Side Object Container Streaming) and that too is itself a large effort. It started in full force last year - they had a big summit where they got the programming disciplines together in Germany and had a week-long deep-dive on the specific implementation details for the effort across the entire game, including the core engine, game code, as well as the backend services in order to execute this. They've been working on SSOCS ever since then, which is almost a year now. Chad claims that they've made very meaningful progress, and that he just got out of a meeting with Chris Roberts about the progress they're making, and that the mood was pretty positive coming out of it.
Some notable things that have happened so far include: they've done a large set of changes to Object Containers in terms of how they load entities - they've now separated things out into what are called Dynamic Entities and Static Entities, essentially allowing them to differentiate between what is streamable and what isn't; they've made something called a Star Engine Service, which is a type of game service that is written in a similar way to game code but can be deployed as a cloud service; using the Star Engine Service they've built what they call the Star Hash Service, which is their version of a kind-of coordinate system across the universe similar to geohashes. This gives them a way to partition the universe so that they know what sections of the universe are streaming at any given time in an efficient way, as well as storing and looking them up efficiently.
So they built a service that keeps track of all of what they call Entity Aggregates in the universe, which was kind of the roots of entity trees. In addition, the Star Hash Service has been hooked up to the Streaming Manager. The Streaming Manager was previously created to govern the current CSOCS, but now they've done the work to connect it to the Star Hash Service too, to drive streaming events to indicate to it that certain aggregates are streamed out, or to collect certain aggregate groups and stream them in. They've still got a bunch of work to do, but what they've done is they've implemented a Prototype In-Memory Persistence Cache (I think he meant Model instead of Cache). For SSOCS to work they basically have to implement Global Persistence, which is a huge set of changes for the game, completely changing how the game runs and how entities are spawned and loaded. For testing purposes they made the Prototype In-Memory Persistence Model, which allows them to write out the state of any of the Star Hashes to a local cache, whenever they're streamed out. This allows them to test game code such as the mission system, the transit system, service beacons, etc. to see what breaks whenever they start streaming things in and out of the game server.
Due to how significantly they're changing the way the game works, they have to revisit the existing game features to make sure they aren't broken, and to fix them in places that they are, which Chad says they've done. The next big things for them to do are to implement the real backend for persistence, which requires rewriting the way that the game writes out to persistence, and to the backend, the cloud services that handle that data so that they're much better at handling much higher loads - this is necessary because there's already a large amount of persistence data in the game right now, due to persisting Players and ships and items, and this is changed all the time by things like flying around and using fuel, or when something breaks off your ship, but Global Persistence means they're looking at persisting the whole universe for all of the game servers and for the whole game, so there's a much higher volume of data to deal with, thus requiring a really big set of changes on the backend to handle that. This work is a work in progress that continues.
The last big thing (and he adds that there's a lot of little details too but he's covering the big-ticket items) is the game code changes that need to happen, meaning they need all of the game feature teams to revisit their features and make sure they're not broken, which is a big effort. As such, until they start testing heavily, they won't even know of all the work that they're going to have to do, so there's a lot of unknown unknowns. Chad sees the progress they've made so far this year as good, but says however that one particularly positive thing that they've figured out is that their Prototype In-Memory Persistence Model seems to be working very well, so they're considering phasing out SSOCS, so that it doesn't need Global Persistence the first time they ship it. This means that they ship it in such a way that if a server goes down, then its state is lost, but Player state isn't lost, so from the Player's perspective the game won't have changed, but it does mean that they can add in some of the streaming benefits without yet having the Global Persistence benefits, which will be released once they've finished that work. As such, a part of SSOCS could end up on a more accelerated release schedule than they maybe anticipated.
Moving on to Server Meshing, it's something that has also been a long time in the making. The Net Bind Culling and Serialised Variables that were mentioned earlier were a prerequisite for Server Meshing. Similar to SSOCS they have begun significant planning and work on the feature, however SSOCS is a hard requirement for Server Meshing. This means that even whilst they can make progress on Server Meshing, because most of the work for Server Meshing falls on the network team with support from the core engine and game code teams (whereas SSOCS is mainly the opposite with the addition of backend services), they can do some work in parallel but they can't deploy it in full until SSOCS is done because it's a prerequisite - it's how Server Meshing fundamentally works - so it'll come after SSOCS for sure.