What can you tell us about building persistence while everybody is sitting there watching?
TL;DRPersistence is an expected but unglamorous feature needing huge work: inventory data locked down first, tiered caching and throttling to protect the database, and auditing tools to track every item's history.
Well, I guess I'll say that it's something that's definitely needed. People don't realize the importance of it. It's not a sexy feature to promote PR-wise because it's an expected feature. It's not like, hey, we've got this great new feature over here, it's got a lot of bling and you all are going to love it. It's more of an expected thing, but it takes a huge amount of work. We had to be very careful on how we brought it into the universe. We had to bring it in at the right time. We had to make sure a lot of the game design, the itemization and such, was kind of hammered out. And it's never 100%, like we still have a lot of data that's... But we had to get the player's inventory, their ships, their entitlements, their pledges and such. We had to make sure that there's a lot of things locked down with that before we could even begin a strategy on how to persist it. Because there's a lot more that goes on behind the scenes, especially with this game, considering that your average player is going to have thousands upon thousands of items rather than hundreds as in other MMOs. We had to take very specific approaches and tailor the data in such a way to where we don't get into situations where we're bogged down, we have throughput issues and such like that.
It's definitely an all-hands-on feature because not only do the server programmers have the backend to manage all this data for potentially 50 to 250,000 concurrent players, and just managing all this data and stuff with our caching services, but you got the gameplay guys, you got the UI team that have to have visual displays inside the game so that you can manipulate the different items and move them around and get those changes taking place, push them back up to the persistent cache service. And then we got these little delta timers that actually write to the database. We've got these business tiers to manage the items so that we have complete control of when it writes to the database. If we didn't have those middle tiers, then oh my god, we would blow the database driver out of the water with all these transactions coming through.
Right. We have to have these different tiers because one particular game server may have 30 people or 20 people on it or whatever. And a lot of items are changing and being sent to the caching layer, which holds the data for a certain amount of time and then will write it to the database in a controlled manner. But when we have 100 game servers, that becomes a problem. So we have to have multiple layers of throttling and caching in order to have complete control over the rate at which data comes into the backend system and into the database in a way to where we don't kill the database.
And plus the auditing tools, so that customer support and QA and all that stuff, for every item in the game, we can get a history of where it was at any given time, when it got traded, when it got sold to the vendor, when it was purchased, or deleted from the website and got re-synced with the universe cluster, things like that. So there's a lot of command and control tools in place to manage items.