Jason Ely

Well, I guess I'll say that it's something that's definitely needed. People don't realize the importance of it. It's not a sexy feature to promote PR-wise because it's an expected feature. It's not like, hey, we've got this great new feature over here, it's got a lot of bling and you all are going to love it. It's more of an expected thing, but it takes a huge amount of work. We had to be very careful on how we brought it into the universe. We had to bring it in at the right time. We had to make sure a lot of the game design, the itemization and such, was kind of hammered out. And it's never 100%, like we still have a lot of data that's... But we had to get the player's inventory, their ships, their entitlements, their pledges and such. We had to make sure that there's a lot of things locked down with that before we could even begin a strategy on how to persist it. Because there's a lot more that goes on behind the scenes, especially with this game, considering that your average player is going to have thousands upon thousands of items rather than hundreds as in other MMOs. We had to take very specific approaches and tailor the data in such a way to where we don't get into situations where we're bogged down, we have throughput issues and such like that.

It's definitely an all-hands-on feature because not only do the server programmers have the backend to manage all this data for potentially 50 to 250,000 concurrent players, and just managing all this data and stuff with our caching services, but you got the gameplay guys, you got the UI team that have to have visual displays inside the game so that you can manipulate the different items and move them around and get those changes taking place, push them back up to the persistent cache service. And then we got these little delta timers that actually write to the database. We've got these business tiers to manage the items so that we have complete control of when it writes to the database. If we didn't have those middle tiers, then oh my god, we would blow the database driver out of the water with all these transactions coming through.

Right. We have to have these different tiers because one particular game server may have 30 people or 20 people on it or whatever. And a lot of items are changing and being sent to the caching layer, which holds the data for a certain amount of time and then will write it to the database in a controlled manner. But when we have 100 game servers, that becomes a problem. So we have to have multiple layers of throttling and caching in order to have complete control over the rate at which data comes into the backend system and into the database in a way to where we don't kill the database.

And plus the auditing tools, so that customer support and QA and all that stuff, for every item in the game, we can get a history of where it was at any given time, when it got traded, when it got sold to the vendor, when it was purchased, or deleted from the website and got re-synced with the universe cluster, things like that. So there's a lot of command and control tools in place to manage items.

Yes. So the way we look at it, we want to implement what's currently known as the event-driven data center. The more I provide what we call sometimes fluid resources, the faster deployment time; they call these the main buzzwords in our business, DevOps, the time to market: how fast would it be for me to scale a game server for you? How easy would it be to obtain resources that I need? How can I repurpose resources? Because the cloud is mostly utility computing. You pay for what you use. And that's a completely different approach than when you own your data center and built it, and now this stuff is yours, and then you use it or you don't use it. That's a different story. So back to my original idea: we want to implement the event-driven data center, where every piece of software that goes through a major event that the infrastructure should know about would fire an event through a wide channel of info buses that have reactors reacting to it and acting upon it. That's the big picture. And we can orchestrate this the way we want, because orchestration is the word here. You need to make all the logic that Jason and Tom and the gameplay guys are writing report what it needs, and my infrastructure reacts to it. Whether by expanding, by replacing, by, oh, that's a bad member of our cluster, we should take it out. Oh, no, this is not going to get auto-healed or self-healed, we've got to wake someone up. That's the plan. Moving to that: the way we usually do it in DevOps and even in backend services shops, you cannot really sit in a room and get this amazing big white paper and draw everything. It's impossible. You will definitely have bad designs. So we go through the MVP, the minimum viable product. Okay, this will deliver my idea. I'm going to scale right now, or I'm going to make the players get in the game. And watch, because maybe there might be a new feature I'm not really putting in my mind. It's the same concept that all the successful startups follow, like how Facebook believes that you should move fast and break things. Until you go out to market, the market corrects you. That also goes really well with your point in the beginning about persistence. If we laid out persistence before seeing the game itself, we'd be just breaking walls and moving around them. While right now, we are more mature. We can start from a point where we understand this is what the game looks like. The game designers had their time. They discussed some ideas, back and forth. The implementation is going to be a lot easier when you have something to test with.

At one point, we didn't even have any of the game data designed. Nobody knew what we had to persist, for example. So we had to actually go through several meetings to figure out what are we going to persist and how are we going to do it.

I remember those meetings, sitting there like, no, I guess the players have this and the players have this, and this needs to sync up with Turbulent and this needs to be pulled down and live in the universe.

Oh yeah, and there's a lot of transitioning on ownership and communication of all that data. I mean, that's still evolving today.

A game like World of Warcraft, for example, you can have hundreds of items on the character. And that seems like a lot of items to keep track of and store in your inventory or in your bank and so on. Where in Star Citizen, just with your ships, the things that you buy on the website right now, you are already in the thousands, and we're going to have north of 50,000 items associated with one player. That's a scale that I've never had to deal with before, but it's something that we've been able to put a lot of thought into and figure out the best way to deal with, and it's still going to evolve, because at some point if we have a million online at one time, that's 50,000 per player, that's a lot of data to move around.

So before persistence, our architecture was very first-person shooter-like. Coming from Sony, the initial server cluster kind of looked like some of the models that we had over at Sony, where we had 250,000 concurrent with the simulators with that type of framework. We know where to scale horizontally. If we break from 250 to a million, we know what we need to federate next. Taking those principles from Sony and applying them to CIG, we can most likely hit those same numbers, but once you add persistence, then we have to go back to the drawing board and figure out where the bottlenecks are going to be, what services are going to get federated horizontally and things like that.

Yeah, we're writing all of our services in C++ just for the performance. And any given service can handle a large number of players and a large amount of data passing through it. So one of the ways we're going to scale, like Tom was saying, horizontally, there's two things to consider: load balancing and availability. So we have to make sure that if any service goes down, there's another one there to pick it up. The service is always available no matter what. So we'll have multiple instances of a particular service up at any given time, and if one goes down, there's another one available and so on. And the load balancing aspect would be just to distribute processing across multiple instances of a particular service.

We definitely want to apply the mega server methodologies, where you have a single login and a single universe that you are in where you can play with anybody around the world, rather than fragmenting up the player base on a server-by-server basis where once you lock into a server, that's it, you can only interact with those people. So we're trying to design that upfront with the mega server architecture in mind for both persistence and the regular matchmaking and persistent universe concepts.

And in the services themselves right now, players and objects have a location. So we already view them as a location in one universe. And there's other technical issues that have to be addressed with the game servers and interaction in the instances and stuff like that to really make the players feel like they are in that one large universe. But we've already been fighting that battle on the back end.

So there is not a clear lock point. I mean, that's not my area of ownership; that's the area I'm the very first close neighbor to. I don't believe that we have a very, very specific lockstep where this is what locks the client.

Right. The client frame rate is usually the rate at which you're refreshing the graphics and so on. So on the server side, on the DGS, as we call it, that's the simulation rate, and they're not locked together.

Yeah, absolutely. I know a lot of players are concerned with the FPS. First of all, I want to thank every single person who plays PTU and all the amazing people who play Evocati, and the great people that I woke up during the night, like, hey, I have a new combination I want to test, and people calling each other and coming online. This is amazing. I couldn't have imagined this was even possible. In my interview a year and a few months ago, I was asking, how do you guys test? It's like, we use people. Like, what do you mean? It's like, our players are awesome. You just say you want to test that, and you'll find people coming from nowhere. So what you guys do at PTU is define pain points that should be addressed. Optimization is a deep rabbit hole where you always need to be worried about premature optimization. But also we have a product out in the market that you guys are playing, and we want to make it playable and fun. What I'm trying to do, with my team and with the guys working in Frankfurt and the guys working in the UK and over here and in LA, is to point them as much as we can to that main pain point, like, oh, if you spend some time on this, we might get some advantage. But what we have right now is nowhere near what we would deliver. We have so many tricks in our bag that we know it's not the right time to bring in. But we really need to care about, yeah, you guys were suffering that much and we enhanced it 10%, 5%, 7%; it's still livable. And we're still keeping an eye on what's going on. And also, sometimes we are not willing to enhance anything right now, but we want to see: did our new code logic or complexity tax us a lot? Did it become very expensive or not? So thank you so much, guys, for all the amazing work you do testing PTU.

Yeah, there's RAD Telemetry profiles being generated all the time, and that gets disseminated to each of the teams appropriately. People will have on their radar what is most expensive on the game servers as well as the client, and just tackle it at the right time. But there's just so many features going into the game. Like I said, premature optimization: once you start optimizing code, it looks a little different than normal.

Right. And there's a wide range of things that we can optimize. From the client side, you're going to have things from rendering, you're going to have client-side simulation, and also you're going to have network latency. Then on the back end, there's all the physics and all the other things that happen there. Then there's also things where the game servers may rely on responses from the services. So there's a wide number. We don't want to pre-optimize now, but we optimize certain things over time, the low-hanging fruit and such like that. There's so many different departments that are going to focus on their little areas, but we have to focus on them at the right time. If we pre-optimize now, we could end up painting ourselves into a corner later.

The website and the back-end services are about to have a very close relationship. Prior to this, it's always been us, the services, and platform. But we're now coming together so the services will look at them as another service and they'll look at us as part of their own system. So we're getting married.

We're using RESTful APIs right now, which work over the internet, but we're co-located now, with a dedicated channel, so we get bi-directional communication.

Yeah, we'll have a set of services that have stateful high-speed communication mechanisms to the platform, and we've been working with them on how to architect both sides to make that as streamlined as possible. And I'll use an example right now. When you buy an entitlement, you purchase a ship on the website, there's no way for the platform to tell us that you purchased a ship. So when you log in, we grab your package, which contains all the ships that you purchased and so on. And then we know about it, and then we can create the actual in-game items and persist them. But with this new high-speed communication system, the moment you buy a ship, they will notify us. We can expand the ship immediately and have it ready for you by the time you log in to the game.

When it comes to community-based features like statistics and achievements and things like that, a lot of those features really are best implemented with a web-based technology. If we have a direct line of communication, we can just trigger those things immediately.

And we'll be able to call functions from them directly as a service and so on. And there's an API we provide for them as well so they can request any data from us. We can generate events back and forth to each other.

It's one of the things that's so awesome about working with a partner like Turbulent.

Those guys are really good. They're awesome.

It's the first time I got on a project where the website was done before the game itself.

Yes. We all think about solving the problems in the same way, so it's a very good match.