Well, persistent entity streaming is basically taking the entire scene graph of the engine, which is the entire hierarchy of the universe, players, all objects, and take that which was previously stored on and most game engine stored in the memory of the server or on the client. We take all of that and we push it to a database. And it gives us the foundation to basically stream the entire entity hierarchy to client servers and to persist. So that means we can shut down all the processes and we can continue where we left off. So it's basically taking this entire chunk of ostate of the universe and pushing it to a persistent database.
And for now, for this very first version, there's not a difference between both because we still run one server per shard. We separate those two words because eventually, the shard will be much larger than just one server. Once we get to server meshing, which will be in later iterations, that's when we have multiple servers per shard. But in this very first iteration, as you said, we basically took the foundation of the engine and we replaced it with this new persistent entity database. We try to get at par with what's there, right? So that means we basically take all the tech we already have, one server per shard, and we replace that with persistent entity streaming.
Well, this is the very, very first part of the question is before server meshing is PES per server. And as I kind of touched on the previous answer, yes, persistent entity streaming is per server. But as you can take items out of the server database into your global database, and we made sure it's the same exact database, the same data structures, there will be more persistence. So you can take persistence from one server to another server by putting items into your inventory and then taking them out of the inventory into a new server. Your second question is, how do we get back onto the same server? For 3.18, we won't address any matchmaking changes. And it again comes back to we want to get at par with what we have so far before we start to adding the new features. Also, matchmaking changes will become a part of our server meshing work, which we will start, obviously, right after persistent entity streaming hits. So for the very first version of persistent entity streaming, you will just use the same matchmaking logic you have right now. That means if you start up your client, you get randomly matched to a server or a shard. I can use those terms interchangeable at the moment because they are pretty much the same thing. Obviously, it's the same as right now. You can play with your friends, so you would join the same instance. If you crash out of a game, we reserve the slot of your current shard. So that means if you crash out, you will be able to have, I think it's 10 minutes, to lock back in and get connected to that shard so you can recover anything. And as I said earlier, all items which you take into your inventory, that's what you will keep with you, which you can bring to any shard. So that is your way to basically make sure the stuff you want to really keep, you can keep by just putting it into your inventories.
So depending on the type of item, and of course if there's legal ownership attached to it, you can stow it and keep it and bring it with you across shards. So that's the first thing. But for items even that you legally own, that you leave, let's say I have a gun that I legally own and I drop it somewhere in Port Olisar and then I stop playing and then I come back later and I get matchmade into another shard. That gun is still in the other shard. It persists per shard. So everything that's in the game world, so unstowed items persist in each server that they're on. Right? That's why there's a mechanism like insurance to get stuff back. If something happens, that's why there's a mechanism like stowing so you can take them across servers. Obviously, that's a bit of a downer because we have a lot of shards, right, for the moment. But the intent and the goal from this foundational work is that we're going to get to very large shards. And so with the matchmaking changes that Paul hinted on related to matchmaking, then we'll be able to have shard affinity. And so the gun that you left on shard X, your chance of getting back on shard X will be extremely high. For the moment, it's not the case. So everything that's unstowed in the game world lives in that server until it's reaped or until it's destroyed or until it's cleaned up. There's many, we're going to talk about a lot of reasons why stuff would go away from a shard.
PaulWell, as, as we hinted earlier if you take a coffee cup and you unstow it or you drop it on the floor in a shot, it stays on that shot. And it stays there as long as we will talk about the, the cleanup process a bit later, but it will stay there basically forever. And everyone can go there and pick it up. And if it doesn't matter if, if, if the player is locked in or out who dropped it, like the moment where you bring something into shot, it's part of the shot and will stay there with the persistent shot forever
Benoît: Yeah. That was one, one confusion I saw throughout the questions was the, like, it's important to notice that it's not like you log into the game world and all of the items you've placed in the world are now suddenly overlaid in it. No, when you put something in one world, one shard, one server, it's in that server. And, and so whether you're online or not at that moment, right? So the, so yeah, that's the confusion. There's no, if I log off, is my coffee cup I placed on Clio still there? Yes. Doesn't matter if you're online or not for that specific, for, for items placed in the game world.
PaulWe're going to, I think the, the only, the only expectation, the only exception to that is that when you log out, obviously your player gets stowed back into, into your player inventory. And if you do a ship that log out, your ship gets stowed together with you into your ship, into your player inventory.
Benoît:And that includes items that are in the ship as well.
PaulIt's always, yes. If we talk about stowing and unstowing, it's always the entire, the entire current state of that object, including everything, which is, you know, I talked about the scene graph and the hierarchy earlier. If you look at a ship and all items placed inside the ship, you can imagine that as like a tree, the hierarchical tree of items. And if you take the, the top, the root of the tree and we put it into an inventory, everything which is under it follows it.
PaulIt does not replicate to other servers at the moment. The only, the only way you can bring something. So the first, the first fundamental point you have to understand is every single item can only exist once, right? So you can't have a copy of the same exact instance of an item in multiple charts. If you have your ship and you bring it to chart A, it is in chart A. That ship can't exist in a different chart. And there's a good reason for it. If it would exist in multiple charts, what happened if in one chart, someone shoots the left wing off in the other shot, someone shoots the right wing off, right? We can't, we can't allow that to happen. So what that means is if that ship is in a chart, it's locked to the chart. It is just part of that instance. If you take that ship and pay, bring it into your global inventory, that's then your way to take this ship and bring it with you to a different chart. And every, every, every shot in this very first iteration is its own isolated instance, basically.
JaredAnd then drawing that line forward to server meshing, server meshing isn't about replication either. Server meshing is about bringing all of these things to all of these servers together in a mesh. And it moves, it transitions, it doesn't replicate.
PaulIf we talk about replication, I mean, obviously if I talk about replication, I probably have a different thing in my head because in server meshing, we do replicate objects between different servers because multiple servers will eventually simulate the same chart. And obviously in that case, you might have two servers and they replicate the same object between each other for simulation reasons, right? So we don't be like, it could be that server A simulates one area and server B simulates another area. There's overlap. In that overlap, you have objects in that overlap area replicated to both servers, right? That's why for me, replication is a bit different word, but you can see a shot again, as it's unique individual instance. There's no cross-shot replication plan. We do, once we talk about player basis and so on, we do have discussions with design right now, how we can do that for like, if a player builds a base on a planet, for example, we kind of want to show that on all charts, not just on the one shot the player's in, but those discussions are still happening.
Benoît: Right. So in the case of, so you have a ship, let's start from the beginning, you go to the ASAP terminal and you unstow a ship. So the ship lives, is first in your, in your location inventory, and it's in the global DB. You, request the ship shows up on a landing pad. That's that actually behind the scenes unstows the ship. So it moves from the global database to the shard. Now you can, you can, now it's being actively simulated by a game server. And so there's thousands of updates on it, right. To keep it up to date in the DB. You go in it, you start flying it out. You put cargo in it, you drop some items in it, whatever. Then a 30k happens, right. Server ran out of memory or, you know, something like this happens. Now when you come back to the game, the, that ship is still in that shard. All right. It's still in the persistent database of shard 24, for example. And so if you come back and you're match made, most likely going to be match made on another server because shard 24 is not available at the time, especially if you log in rapidly back in. When you enter your new shard, that ship will not be available for sure. Right. So, but it will be available for insurance claims. What we are looking at, at this moment for this problem, because we know that's a problem, obviously this is not cool, right. Is that when a server crashes, well, we have a dedicated recovery mechanism that we're actually putting in place right this moment. We're working actively on this right now throughout our current stability testing, which will auto stow back the ship into your location inventory. And so what that means is we don't want you to be stuck with your ship stuck in a shard. That's no longer available, right? You were actively playing with that ship. It's got cargo in it. We want you to be able to grab it. And so in the case of a 30k, your, your ship would get auto stowed same as your player, because there's no difference between a ship and a player in this instance. So, that's the goal that we have for 318 is to make sure that that case doesn't happen. So that in the case of a 30k, your ship is just stowed back in, in the exact state it was in at the moment of the crash. You keep your stuff, right. That's, we don't want you to be, you know, stuck with a ship that's unavailable. And then you need to find the shard again. And listen, you know, that's like a guessing game. So we don't want you to be stuck at that.
PaulI do want to add though that eventually, and that's a great feature of persistent entity streaming, if that server 30ks with persistent entity stream before persistent entity streaming, that means this, this whole world would be dead because the memory is gone. Everything is gone with persistent entity streaming. However, the shard and the state of the shard still exists, right? The server crashed because it ran out of memory or there was a game code bug or whatever. It's always game code bugs, by the way. Yes. It's never the core tech. Anyways, when that happens, we can actually start up a new server, and it can load back that shard and then continue simulation for exactly where we left off. Again, for 3.18, because we don't want to address matchmaking just yet. Um, we will do what Ben just said, where we try to take, locate all the ships and we stole them into your location inventory. So you don't lose your ship and the progress on that ship. However, the ultimate goal is that when a shard server crashes, that we restart a new server for that shard and that matchmaking will allow your player to reconnect to that shard and then matchmaking will be able to do, to resolve the slot for that exact shard. So you start ultimately or not in a next iteration, you will be able to start your client. And there will be a little dialogue, which telling you, do you want to wait for your old shard to come back online? Because obviously it takes a couple of minutes before shard is back online and then you will be able to join back to exactly where we left off. Um, and then eventually the final goal, once we have, uh, complete, uh, the, the server matching and the replication layer and the gateway layers all separated at that point, when a server 30 case, you won't even notice you will still stay connected and we start a new server, or we have a fleet of servers already running, which can take over that work. But that's obviously fine in the future. So what we, uh, what, what we're saying is in this very first version to be on pause, what we have right now, server crashes, we take player and you take the ships, put them all into your inventory. So you don't lose progression. And you do, and you will get exact the persistent state back from the ship at the moment, the server crashed. So you don't lose anything. Um, in the next version, we then try to revive that shard and let you let matchmaking allow you to come back to that shart. And then, ultimately in the, in the final version, it should be all, almost transparent to the player.
Benoît: First of all, that's a bastard move. Right. So not entirely related to PES, I understand the mental link between persistence and spawn points and out the four-ing and stuff. There already have been some changes in the game related to combat logging, which is basically out the four-ing while in combat to save your FPS items. Already the ships stay in the game world in 3.17 and then they basically get destroyed. Basically, that move sort of allows you to save your weapons and armor where you respond outside. So PES is not going to solve that because that's driven by game rules. So even though we have persistence, that's the thing I wanted to stress out, even though entities persist, the game rules and the game code can still decide what to do. They can override any values. If you think about respawning or things like that, same thing for logging. So for combat logging more specifically, which is a problem, obviously. It's something that we're going to want to tackle at some point, but the real fix for that is not a new persistence system. If PES solved all the world problems, that'd be awesome, but it's not the case. The real solution is a proper logout timer, where if you decide to out the four or to kill your client or whatever, you stay in the game world for that logout timer, which will allow your assailants to basically finish the job if they have to. And that makes the game fair. Most other games also do the same thing. So that's the idea that we have on the table now. Obviously, we're not tackling this specifically with PES. Even though there is persistence, it's not going to affect combat logging specifically.
PaulI would add, though, because of PES, the login and logout code has to change slightly. It gives the game code teams actually the opportunity to address this issue. And as Benoit said, we're currently discussing some of the login logout flow. And as Benoit said, adding a proper logout timer to that would be a good opportunity right now to do. Another question I also saw asked a lot is, when I log out with persistent entity streaming, will I log back in where I left off? And again, to our point, let's get back to feature parity. For 3.18, we will keep our current logout system. So when you log out and you log back in, you log back at your last landing zone location. However, PES does enable all of that cool stuff. So with PES, it would be very trivial for game code to change their systems to allow you to log exactly back where you left off. However, it always comes back to how much risk do we want to add for 3.18. And also, the devil is always in the details. If we want to have game code now to completely change, okay, you log back in where you log back out, then that doesn't mean, okay, we have the tech right now, so we can just enable this feature. That means a lot of design work. Like how does it work, for example, if I am suffocating and I log out? I think the design wants you, for example, then to run a timer. And eventually when you log back in, you would be dead because you were suffocating at that moment. You want to spawn back in hospital and so on. So there's a lot of design consideration coming into play. And then also game code systems which need to handle that properly. For example, what if I log out on a planet and then on the next day I log back in? And that exact position is another ship landed at the moment, right? So where game code would need to probably do some position resolving and all those kind of things. So in order to not tackle that right now, we would just keep the log in log out flow how it is, however, as I said, with persistent entity streaming, game code teams now have all the tools at their hands to fully persist whatever they want at the state and restore at exactly the state how they left it
Okay, that's a good question. With our old system, we had this, it's called lifetime management, and it would arbitrarily remove entities once they hit a certain cap. And we realized that's not really great for play experience. For example, I think at the moment, players can only spawn up to three ships. If they spawn another ship, the old one gets removed. For persistent entity streaming, we actually revisit that problem. And we decided, okay, how can we make this a nicer for the player, but also be how can we make this actually work good for persistent streaming and also for server meshing, where maybe it doesn't make sense if if a player has five ships spawned and they're all separated in open space and streamed out, they don't really cost us performance. So why do we still need to arbitrarily limit that? And for that, we have the guys in Frankfurt developed a new system. It's called density management, and it operates on relative cost of an area. So the idea is that instead of just taking arbitrary numbers on objects, we look at areas and we look at the amount of items in that area. And we only start to remove items the moment the relative cost of that area becomes too high. So as long as you spawn your ships in different locations and they're all spread out, we will never touch them or remove them. However, if like now everyone goes to the same spot and you start spawning ships and spawning ships at the same location, that's when the system will find out, OK, now the performance goes down because our relative cost in that specific area goes too high. And then we will start deleting items, which also follows a certain rule set. So we will delete oldest items first. We will also have a relative priority on items. So a player ship, for example, is a very high priority item. So we would not delete a player ship versus a coffee cup, for example, would be the first item to go away. And then with that, we try to get the game to where it's not very invasive to you. So you don't just see stuff disappearing out of the air. Exactly like the approach we're taking is that most of this cleanup will happen during stream out. So if an area streams out, that's when the density management will make a first initial pass and will say, hey, in this area, we had 50 coffee cups and that's too much. So we start removing them. And then we only go into the more aggressive policies once it's really becoming maybe an area never streams out or whatever. All players decided, hey, I mean, I kind of expect that right once we go get PSL probably everyone will like, let's all go to Port Olisar and drop all our coffee mugs on the floor and keep doing that. That's what you guys will do anyways. So basically the system will take care of this, but it will be a smarter system. It's a smarter system than what we have right now.
Benoît: One thing to add to thedensity management too, is that it also has this concept of references. And so for example, if an entity, because this also covers NPC entities, right? So like AIs and whatever else. And so if a specific entity is actually being referenced by a mission, active mission or active player, for example, then there's a ref counting, a reference counting system that will prevent those from being actively reaped. So that, again, I think the goal with density management is really to make sure stuff doesn't pop out of your view and break your game experience. But we still need a density management system to prevent overloads. But that's the approach in general.
So I did actually inquire about this a little bit before we came here because obviously this is game code driven. So it's not something that the PES team actually controls, though we do enable persistence for those things. So I do believe that gimbals modes might persist because they're implemented with a serialized variable. So that means that this would stay. For MFDs and scanning angles, we don't know yet. That needs to be confirmed by the combat team. But if it's implemented with serialized variable, then yes. If it's not and it's a reset, like again, a bit like the spawning location, right? If they every time you sit in the seat, it changes the value, then it doesn't matter if it's persisting. Right. But that's something that all the game teams will be tweaking up until we get there. And if they can benefit from it, I'm sure they will jump on it. Absolutely.