TL;DRPersistence is an expected but unglamorous feature needing huge work: inventory data locked down first, tiered caching and throttling to protect the database, and auditing tools to track every item's history.
Well, I guess I'll say that it's something that's definitely needed. People don't realize the importance of it. It's not a sexy feature to promote PR-wise because it's an expected feature. It's not like, hey, we've got this great new feature over here, it's got a lot of bling and you all are going to love it. It's more of an expected thing, but it takes a huge amount of work. We had to be very careful on how we brought it into the universe. We had to bring it in at the right time. We had to make sure a lot of the game design, the itemization and such, was kind of hammered out. And it's never 100%, like we still have a lot of data that's... But we had to get the player's inventory, their ships, their entitlements, their pledges and such. We had to make sure that there's a lot of things locked down with that before we could even begin a strategy on how to persist it. Because there's a lot more that goes on behind the scenes, especially with this game, considering that your average player is going to have thousands upon thousands of items rather than hundreds as in other MMOs. We had to take very specific approaches and tailor the data in such a way to where we don't get into situations where we're bogged down, we have throughput issues and such like that.
It's definitely an all-hands-on feature because not only do the server programmers have the backend to manage all this data for potentially 50 to 250,000 concurrent players, and just managing all this data and stuff with our caching services, but you got the gameplay guys, you got the UI team that have to have visual displays inside the game so that you can manipulate the different items and move them around and get those changes taking place, push them back up to the persistent cache service. And then we got these little delta timers that actually write to the database. We've got these business tiers to manage the items so that we have complete control of when it writes to the database. If we didn't have those middle tiers, then oh my god, we would blow the database driver out of the water with all these transactions coming through.
Right. We have to have these different tiers because one particular game server may have 30 people or 20 people on it or whatever. And a lot of items are changing and being sent to the caching layer, which holds the data for a certain amount of time and then will write it to the database in a controlled manner. But when we have 100 game servers, that becomes a problem. So we have to have multiple layers of throttling and caching in order to have complete control over the rate at which data comes into the backend system and into the database in a way to where we don't kill the database.
And plus the auditing tools, so that customer support and QA and all that stuff, for every item in the game, we can get a history of where it was at any given time, when it got traded, when it got sold to the vendor, when it was purchased, or deleted from the website and got re-synced with the universe cluster, things like that. So there's a lot of command and control tools in place to manage items.
TL;DRA player will have far more items than in other MMOs, so services are written in C++ and scaled horizontally with load balancing and availability, aiming at a single mega-server universe, though persistence forced rethinking bottlenecks.
A game like World of Warcraft, for example, you can have hundreds of items on the character. And that seems like a lot of items to keep track of and store in your inventory or in your bank and so on. Where in Star Citizen, just with your ships, the things that you buy on the website right now, you are already in the thousands, and we're going to have north of 50,000 items associated with one player. That's a scale that I've never had to deal with before, but it's something that we've been able to put a lot of thought into and figure out the best way to deal with, and it's still going to evolve, because at some point if we have a million online at one time, that's 50,000 per player, that's a lot of data to move around.
So before persistence, our architecture was very first-person shooter-like. Coming from Sony, the initial server cluster kind of looked like some of the models that we had over at Sony, where we had 250,000 concurrent with the simulators with that type of framework. We know where to scale horizontally. If we break from 250 to a million, we know what we need to federate next. Taking those principles from Sony and applying them to CIG, we can most likely hit those same numbers, but once you add persistence, then we have to go back to the drawing board and figure out where the bottlenecks are going to be, what services are going to get federated horizontally and things like that.
Yeah, we're writing all of our services in C++ just for the performance. And any given service can handle a large number of players and a large amount of data passing through it. So one of the ways we're going to scale, like Tom was saying, horizontally, there's two things to consider: load balancing and availability. So we have to make sure that if any service goes down, there's another one there to pick it up. The service is always available no matter what. So we'll have multiple instances of a particular service up at any given time, and if one goes down, there's another one available and so on. And the load balancing aspect would be just to distribute processing across multiple instances of a particular service.
We definitely want to apply the mega server methodologies, where you have a single login and a single universe that you are in where you can play with anybody around the world, rather than fragmenting up the player base on a server-by-server basis where once you lock into a server, that's it, you can only interact with those people. So we're trying to design that upfront with the mega server architecture in mind for both persistence and the regular matchmaking and persistent universe concepts.
And in the services themselves right now, players and objects have a location. So we already view them as a location in one universe. And there's other technical issues that have to be addressed with the game servers and interaction in the instances and stuff like that to really make the players feel like they are in that one large universe. But we've already been fighting that battle on the back end.
Yeah, and this is pretty interesting. It's an interesting challenge because we have to not only deal with U.S. players, we have to consider people in Europe and in Asia and South America and so on.
Oh, yeah, Australia. It's a global mega server, so we have to make sure that the players feel that they are there and that they can play with their US friends, they can play with their European friends, whomever, and it feels like you're next door. You're not transitioning into another realm type of thing.
If I remember right, Spring and Struts are a Java J2EE framework. Our universe cluster is primarily C++ with some third-party libraries that we customize. But the web platform, on the other hand, does use a lot of web-based technologies.
We get asked a lot what we're going to do to prevent cheating and stuff like that. We don't answer those questions because we don't want the bad guys to know what we're doing. So I understand why the good guys would want to know the things that we're doing and the details and how we're going to go about it. It's just one of those Catch-22s where the more info on those things that we put out, the more info the bad guys have to circumvent.
Yeah. The best thing to do, you don't answer the question, but you make sure that they know we're watching, we know what they're going to do, and make them paranoid.
TL;DRServer simulation rate and client frame rate are not locked together; the team says optimization is ongoing at the right time, guided by profiling and PTU testing, while avoiding premature optimization.
So there is not a clear lock point. I mean, that's not my area of ownership; that's the area I'm the very first close neighbor to. I don't believe that we have a very, very specific lockstep where this is what locks the client.
Right. The client frame rate is usually the rate at which you're refreshing the graphics and so on. So on the server side, on the DGS, as we call it, that's the simulation rate, and they're not locked together.
Yeah, absolutely. I know a lot of players are concerned with the FPS. First of all, I want to thank every single person who plays PTU and all the amazing people who play Evocati, and the great people that I woke up during the night, like, hey, I have a new combination I want to test, and people calling each other and coming online. This is amazing. I couldn't have imagined this was even possible. In my interview a year and a few months ago, I was asking, how do you guys test? It's like, we use people. Like, what do you mean? It's like, our players are awesome. You just say you want to test that, and you'll find people coming from nowhere. So what you guys do at PTU is define pain points that should be addressed. Optimization is a deep rabbit hole where you always need to be worried about premature optimization. But also we have a product out in the market that you guys are playing, and we want to make it playable and fun. What I'm trying to do, with my team and with the guys working in Frankfurt and the guys working in the UK and over here and in LA, is to point them as much as we can to that main pain point, like, oh, if you spend some time on this, we might get some advantage. But what we have right now is nowhere near what we would deliver. We have so many tricks in our bag that we know it's not the right time to bring in. But we really need to care about, yeah, you guys were suffering that much and we enhanced it 10%, 5%, 7%; it's still livable. And we're still keeping an eye on what's going on. And also, sometimes we are not willing to enhance anything right now, but we want to see: did our new code logic or complexity tax us a lot? Did it become very expensive or not? So thank you so much, guys, for all the amazing work you do testing PTU.
Yeah, there's RAD Telemetry profiles being generated all the time, and that gets disseminated to each of the teams appropriately. People will have on their radar what is most expensive on the game servers as well as the client, and just tackle it at the right time. But there's just so many features going into the game. Like I said, premature optimization: once you start optimizing code, it looks a little different than normal.
Right. And there's a wide range of things that we can optimize. From the client side, you're going to have things from rendering, you're going to have client-side simulation, and also you're going to have network latency. Then on the back end, there's all the physics and all the other things that happen there. Then there's also things where the game servers may rely on responses from the services. So there's a wide number. We don't want to pre-optimize now, but we optimize certain things over time, the low-hanging fruit and such like that. There's so many different departments that are going to focus on their little areas, but we have to focus on them at the right time. If we pre-optimize now, we could end up painting ourselves into a corner later.
TL;DRThe website and back-end services are being tightly integrated through a fast, bi-directional, stateful link, so purchases like ships can be registered immediately and web features can trigger game-side events and function calls directly.
The website and the back-end services are about to have a very close relationship. Prior to this, it's always been us, the services, and platform. But we're now coming together so the services will look at them as another service and they'll look at us as part of their own system. So we're getting married.
We're using RESTful APIs right now, which work over the internet, but we're co-located now, with a dedicated channel, so we get bi-directional communication.
Yeah, we'll have a set of services that have stateful high-speed communication mechanisms to the platform, and we've been working with them on how to architect both sides to make that as streamlined as possible. And I'll use an example right now. When you buy an entitlement, you purchase a ship on the website, there's no way for the platform to tell us that you purchased a ship. So when you log in, we grab your package, which contains all the ships that you purchased and so on. And then we know about it, and then we can create the actual in-game items and persist them. But with this new high-speed communication system, the moment you buy a ship, they will notify us. We can expand the ship immediately and have it ready for you by the time you log in to the game.
When it comes to community-based features like statistics and achievements and things like that, a lot of those features really are best implemented with a web-based technology. If we have a direct line of communication, we can just trigger those things immediately.
And we'll be able to call functions from them directly as a service and so on. And there's an API we provide for them as well so they can request any data from us. We can generate events back and forth to each other.
Yeah, matchmaking services are ready to go. So we've got it in the QA environment right now and we're currently pushing into production and just trying to figure out how many machines we're going to start out with.
Absolutely. Manuel's been a great help. We had the backend architecture in place, and Manuel was able to really take the matchmaking services and just plug in his matchmaking logic into our framework, and it just worked.
TL;DRGame servers register with the instance manager, and when a player picks a level and mode in the matchmaking UI, the matchmaker scans running servers and places them in a suitable game.
Well, basically, we start up X number of game servers and they register themselves with the instance manager. And so the matchmaker will talk to the instance manager and say, hey, how many game servers are free for games to be played on? Or what are the different player counts and what are the different levels that are currently running on each of the game servers. So when you log in on your own computer and you go to the matchmaking UI, you pick which level you want to play and what game mode, whether it's Capture the Core or Battle Royale, you put in your selection, you hit go, and your message goes up to the matchmaker service, and it's going to scan all the game servers that are currently up and running, and find a game for it to place you into.
A network programmer programs the multiplayer aspect of the game. When it comes to distributed physics and shooting guns and making sure they're all synced up across the network, those are the things I'm responsible for.
TL;DRThey're building the backend infrastructure for the universe cluster, including a universe manager and matchmaking that groups players and moves them between dedicated servers, aiming for seamless transitions.
Well, I'm a server programmer and a network programmer too, but we're creating the initial backend infrastructure for the universe cluster. And what that does is it coordinates all the dedicated servers, so that when you log in, you've got this sharder that's going to place you around the world and things like that.
Well, basically, we've got a universe manager that keeps track of where everybody is in the universe. We've got business rules and matchmaking services that are going to group people together, shard them together, and direct them to go to different dedicated servers, so that you kind of have this experience with a group of people. But when it's decided that you need to move on, whether you left that space or you did something, it does a logical transition. It goes from one dedicated server to the next. And we're trying to do it in a seamless fashion so that it's transparent to the end user.
TL;DRHe isn't worried, since CryEngine's networking already works well out of the box and 16-player dogfights ran early on, though each new engine drop means re-checking that nothing broke.
Well, we're using a lot of great technology with CryEngine. CryNetwork is kind of like what I did at Sony, building up this client library that's going to handle client management and broadcast schedules and monitoring the bandwidth and stuff. We get that with CryNetwork, and it's pretty featureful. It's working out of the box. We had 16-player dogfights when I first came here, before I started working on the backend architecture. We were playing 16-player dogfights already, flying around, shooting missiles, coordinating. And we just got a new code drop, so I'm actually off the backend, making sure it's not broken. Every time we get a drop of the engine, we've got to go back and fix it again. So I've got this level called Epic Space Battle, and that's what I'm using for my playtest. It's like, it's failing to load, what changed? So just kind of going through the checklist.