Star Citizen: June Subscriber's Town Hall feat. Engineering Team

Well, I guess I'll say that it's something that's definitely needed. People don't realize the importance of it. It's not a sexy feature to promote PR-wise because it's an expected feature. It's not like, hey, we've got this great new feature over here, it's got a lot of bling and you all are going to love it. It's more of an expected thing, but it takes a huge amount of work. We had to be very careful on how we brought it into the universe. We had to bring it in at the right time. We had to make sure a lot of the game design, the itemization and such, was kind of hammered out. And it's never 100%, like we still have a lot of data that's... But we had to get the player's inventory, their ships, their entitlements, their pledges and such. We had to make sure that there's a lot of things locked down with that before we could even begin a strategy on how to persist it. Because there's a lot more that goes on behind the scenes, especially with this game, considering that your average player is going to have thousands upon thousands of items rather than hundreds as in other MMOs. We had to take very specific approaches and tailor the data in such a way to where we don't get into situations where we're bogged down, we have throughput issues and such like that.

It's definitely an all-hands-on feature because not only do the server programmers have the backend to manage all this data for potentially 50 to 250,000 concurrent players, and just managing all this data and stuff with our caching services, but you got the gameplay guys, you got the UI team that have to have visual displays inside the game so that you can manipulate the different items and move them around and get those changes taking place, push them back up to the persistent cache service. And then we got these little delta timers that actually write to the database. We've got these business tiers to manage the items so that we have complete control of when it writes to the database. If we didn't have those middle tiers, then oh my god, we would blow the database driver out of the water with all these transactions coming through.

Right. We have to have these different tiers because one particular game server may have 30 people or 20 people on it or whatever. And a lot of items are changing and being sent to the caching layer, which holds the data for a certain amount of time and then will write it to the database in a controlled manner. But when we have 100 game servers, that becomes a problem. So we have to have multiple layers of throttling and caching in order to have complete control over the rate at which data comes into the backend system and into the database in a way to where we don't kill the database.

And plus the auditing tools, so that customer support and QA and all that stuff, for every item in the game, we can get a history of where it was at any given time, when it got traded, when it got sold to the vendor, when it was purchased, or deleted from the website and got re-synced with the universe cluster, things like that. So there's a lot of command and control tools in place to manage items.

Not to go too far back, when the cloud movement started, people said, it's not for everyone. You cannot just take your app and put it in the cloud in a VM, because it doesn't matter whether you have it on bare metal, have it in a VM in a small host provider, or in one of the big public clouds, you have to use the other abstraction layers that the cloud provides to you. The cloud provides to you object storage, block storage that can move between VMs. VMs can be migrated. Different concepts of firewall, different concepts of VPN, and everything is offered to you as a service. That's a lot of good work happening for you, saving you and your team from doing all this tedious work that's technically reinventing the wheel. However, you are relying on resources shared among other people. So everything about the cloud provider tells you, you gotta design for failure. Expect me to go down at any moment, expect this to happen, expect this to change. And the statelessness. So in websites, most of the time, people try to attain what we call statelessness, like there's no state within the ephemeral resource that might die at any moment, and I have the state saved outside in redundant form that can be accessed in different ways, which is the basic concept of distributed systems. Having this in the web is like being a first-class citizen in a country that talks your language. Everyone does this. And you look at everyone else's playbook and steal a page from them and just do it. Games are the complete opposite. They are very, very stateful. An MMO is extremely stateful. It's not even match-based. It's not session-based. So you expect that you leave your machine running for 24 hours, if you can hold out for 24 hours playing, and you want this to happen. And also, you're not going to tolerate having a 404 or a 502 that you see in websites in the middle of that huge epic game where you get an amazing loot, whatever you are looking for. So given this sort of stability in the cloud, it's not really easy. And dealing with the cloud quirks, because yeah, the cloud does a lot of good work for me, but also takes a lot of control out of my hands. Oh, I could have made this thing work in this way, and stopped the packets here, and only replicated this half, and I can't. I'm using whatever abstraction layer the cloud provider provided me, and the tools I'm using, just like anyone else. So the challenge is, MMO games are an alien in the cloud, while all the web technology are the first-class citizen. So we are this kind of alien; no one else is doing what we are. We have been watching a lot of companies when they go and talk at conferences about their cloud adoption; not a lot of people really manage to have the simulator, which is the dedicated game server, in the cloud efficiently, having everything working fine, and that's the challenge. There are people that did what I mentioned in the beginning: they took their original data center bare metal infrastructure and carried it and put it in the cloud on VMs, the same exact way. That's not what we're trying to do, and it's a challenge because, to get a tiny bit technical, if you want to use a new tool in the cloud or a new service in the cloud in a modern language, and your use case is just a CRUD, something where you create and remove records and all of that, you would find an example within five minutes and you will look at it and you just saved yourself a lot of time and then move from there. Using C++, using a game, that's a different thing to deal with.

Yes. So the way we look at it, we want to implement what's currently known as the event-driven data center. The more I provide what we call sometimes fluid resources, the faster deployment time; they call these the main buzzwords in our business, DevOps, the time to market: how fast would it be for me to scale a game server for you? How easy would it be to obtain resources that I need? How can I repurpose resources? Because the cloud is mostly utility computing. You pay for what you use. And that's a completely different approach than when you own your data center and built it, and now this stuff is yours, and then you use it or you don't use it. That's a different story. So back to my original idea: we want to implement the event-driven data center, where every piece of software that goes through a major event that the infrastructure should know about would fire an event through a wide channel of info buses that have reactors reacting to it and acting upon it. That's the big picture. And we can orchestrate this the way we want, because orchestration is the word here. You need to make all the logic that Jason and Tom and the gameplay guys are writing report what it needs, and my infrastructure reacts to it. Whether by expanding, by replacing, by, oh, that's a bad member of our cluster, we should take it out. Oh, no, this is not going to get auto-healed or self-healed, we've got to wake someone up. That's the plan. Moving to that: the way we usually do it in DevOps and even in backend services shops, you cannot really sit in a room and get this amazing big white paper and draw everything. It's impossible. You will definitely have bad designs. So we go through the MVP, the minimum viable product. Okay, this will deliver my idea. I'm going to scale right now, or I'm going to make the players get in the game. And watch, because maybe there might be a new feature I'm not really putting in my mind. It's the same concept that all the successful startups follow, like how Facebook believes that you should move fast and break things. Until you go out to market, the market corrects you. That also goes really well with your point in the beginning about persistence. If we laid out persistence before seeing the game itself, we'd be just breaking walls and moving around them. While right now, we are more mature. We can start from a point where we understand this is what the game looks like. The game designers had their time. They discussed some ideas, back and forth. The implementation is going to be a lot easier when you have something to test with.

At one point, we didn't even have any of the game data designed. Nobody knew what we had to persist, for example. So we had to actually go through several meetings to figure out what are we going to persist and how are we going to do it.

I remember those meetings, sitting there like, no, I guess the players have this and the players have this, and this needs to sync up with Turbulent and this needs to be pulled down and live in the universe.

Oh yeah, and there's a lot of transitioning on ownership and communication of all that data. I mean, that's still evolving today.

A game like World of Warcraft, for example, you can have hundreds of items on the character. And that seems like a lot of items to keep track of and store in your inventory or in your bank and so on. Where in Star Citizen, just with your ships, the things that you buy on the website right now, you are already in the thousands, and we're going to have north of 50,000 items associated with one player. That's a scale that I've never had to deal with before, but it's something that we've been able to put a lot of thought into and figure out the best way to deal with, and it's still going to evolve, because at some point if we have a million online at one time, that's 50,000 per player, that's a lot of data to move around.

So before persistence, our architecture was very first-person shooter-like. Coming from Sony, the initial server cluster kind of looked like some of the models that we had over at Sony, where we had 250,000 concurrent with the simulators with that type of framework. We know where to scale horizontally. If we break from 250 to a million, we know what we need to federate next. Taking those principles from Sony and applying them to CIG, we can most likely hit those same numbers, but once you add persistence, then we have to go back to the drawing board and figure out where the bottlenecks are going to be, what services are going to get federated horizontally and things like that.

Yeah, we're writing all of our services in C++ just for the performance. And any given service can handle a large number of players and a large amount of data passing through it. So one of the ways we're going to scale, like Tom was saying, horizontally, there's two things to consider: load balancing and availability. So we have to make sure that if any service goes down, there's another one there to pick it up. The service is always available no matter what. So we'll have multiple instances of a particular service up at any given time, and if one goes down, there's another one available and so on. And the load balancing aspect would be just to distribute processing across multiple instances of a particular service.

We definitely want to apply the mega server methodologies, where you have a single login and a single universe that you are in where you can play with anybody around the world, rather than fragmenting up the player base on a server-by-server basis where once you lock into a server, that's it, you can only interact with those people. So we're trying to design that upfront with the mega server architecture in mind for both persistence and the regular matchmaking and persistent universe concepts.

And in the services themselves right now, players and objects have a location. So we already view them as a location in one universe. And there's other technical issues that have to be addressed with the game servers and interaction in the instances and stuff like that to really make the players feel like they are in that one large universe. But we've already been fighting that battle on the back end.

I did have one of these. Every one of us has had one of these. One time... so the way we deploy our environments is very well known in the cloud and DevOps industry as blue-green deployments: we would have a full environment running, whether it's PTU or live, players can see it, and then we stage a shadow environment, complete and full, and then at the moment, we just switch the load balancer and the players get moved. I can't remember, was it 2.2? And I had to edit very, very large files; I mean, we are an alpha project. So sometimes we broke our own rules, like, I'm going to test it right before I do it. Yeah, we have no time. The thing that we have in DevOps, and I think we have in Austin, is we deliver whatever the company needs to be delivered as quick as possible. So yeah, it was one of these things where you hack something really quick, and I had an extra space in a URL that goes to the platform APIs in a very gigantic dictionary that generates a lot of configuration that goes to different services. And this space led to an authentication issue. So QA was testing, trying to get in, and nothing was working. And it took a while, because the error message wasn't very, very clear. So yeah, these things always happen. The good thing is we did not do it in production. We had it in the staging one, so we got to fix it and see it. I'm sure every person running live production servers all the time gets to see some of these human errors.

Yeah, I'm not a huge fan of focusing on the brand, but yeah, we currently use Chef and we use a bunch of our homegrown scripts that do the provisioning. We can bring up a graph that will explain a lot of my work and a lot of what we do. If we start all the way on the right, you can see the build system in Austin. That's where everything is being built. Our current build system is based on a build system called Buildbot. It's highly modified and customized to our own needs; the way it schedules and manages the fanning in, fanning out, everything in the build system is custom made for us. So the outcome from the build system will be a build, which is all the binaries needed to run the client, run the server, run the backend services, and a bunch of other files. That gets uploaded to an object storage in the cloud, as you can see, still on the right. And from there, now we have the build. We want to deliver it as fast as possible. Our current cloud provider gives us a good advantage here: we can create an ephemeral box at this moment, and we pull the build down, and we do whatever we want to do in the OS at this moment, like install packages, make sure that we change the ulimits, make sure that everything is optimized the way we want it, and then we create snapshots from it. This whole process usually takes 20 minutes, 25 minutes. It highly depends on the WAN link. I mean, who's fighting with me over uploading something? It could be me too, uploading somewhere else. So yeah, this is mainly an internet speed kind of limit. And then we get the snapshot. The snapshot can create disks, and these disks can be shared across the whole cluster. So if we scroll back up again, now you understand that the build exists in the snapshot. You as a player sitting here, that's your game client. You start your launcher/patcher, and your launcher/patcher would go grab the build for you from the CDN. The CDN, or content delivery network, is a bunch of endpoints globally distributed, trying to deliver you as fast as possible the files that you need, and you pull them down to your game client. Now you have the build. The build needs to connect. You would have a connection between you and the hub load balancer that takes you to the hub servers. So you have a bunch of different hub servers that handle your connection. You go there over TCP; the hub servers go to the platform, make sure that you are the guy who you claim you are, and you have a connection with the platform. From here, the hub service will take your traffic to most of the services that Tom and Jason did here in the core server. They live in a single box, and we can say why they live in a single box for now. And one of them is the persistence services. We have two main services, persistence cache and the persistence service itself. Those connect to our MySQL database clusters; it has masters and slaves and backups and all this cool stuff. Here you see all the dedicated game servers. All these boxes that are drawn represent an environment. Any cloud environment right now in Cloud Imperium Games has to be three different VMs with three different types of roles: one of them for the hub, one of them for all the core services, and one of them for the game server. This is not anywhere near final. That's what we need now to get everything running and see how we want to scale and change stuff. The difference between QA, PTU and Live, to be quick: QA is one, one, one. PTU is multiple of these, one of these, multiple of these. Live is much larger. The question that he asked also relates to this bunch of servers. We have our infrastructure servers: we have log aggregation systems, we have monitoring systems, tools for dev support, and tools for analytics. We use Chef here, we use a bunch of homegrown scripts here, and over here we have Splunk, we have Graphite, Grafana, StatsD, we have Stackdriver, we have a bunch of core dump analysis, we have GDB server for developers to look at, we have some stuff related to Google Breakpad. I know that we have a lot of backers who come from a technical background, and I see the posts, so I'm trying to keep a balance where I use a lot of technical jargon and then go back to something high level. So yeah, this is not anywhere near final. Actually, right now we are working on a new deployment pipeline that would be the event-driven data center I was talking about, where scaling in and out would be easy and usable under heavy load.

Yeah, this project is running on lots of caffeine and lots of people across the world. This is one of the biggest projects that we have, and it involves people from so many different departments. We have people from DevOps, we have people from IT, engine programmers, people in the platform. Everyone is working together on our own new patch delivery system, and it's not going to only affect the players, it's going to affect even our internal developers, the way that they receive their own builds. Actually, our internal developers have it kind of worse than the players outside in the world, because they pull a lot of builds all the time. So yeah, work is running. I see all the change lists coming in and out. I see all the commits happening. We get to have meetings every now and then to make sure that we all still have the same features that we require from the whole system. It's going to be awesome. We feel really bad that when we deliver three builds for you guys in one single day, you can only play the last one, because by the time you finish the first one... And we want to give you this seamless kind of... I'm a DevOps guy, and all I care about is giving you the content that's ready for you as fast as possible. So having a smaller patch is key for all of us. Yeah, it's been working.

Yes, Mike Pickett. Mike Pickett was one of the very first people that was talking about this issue, having his own design on it. But it extends to a lot of people, because the way we are looking at it, to do everything right, you have to have this system native in all the endpoints that need to use it. So every part should be aware of it to deliver this kind of experience where, I'm downloading, now the game is playable, then it can complete. All this fun, amazing, good stuff needs to be done. And the scale, always the scale is the key. That's not an indie game. I mean, yeah, we're a startup, but we understand how big our community is. So when we build a foundation technology like this, we need to look at all edge cases, make sure everything will work, even if we have one million people downloading right now, because we do not really want to be the victims of our own success. Our game is very successful right now, and we are here just to serve the success.

He hates the light purple network cables.

It brings up good memories, dealing with stuff that I can touch with my hands, which stopped a few years ago, when we had to wire cables and remember which wiring and what colors. Yeah, that's how we started. Nowadays, we forget about all this stuff, and that's the beauty of the cloud. We tell stuff to happen and they happen. They always happen the right way, and it's our job to make sure they always come out the right way. But yeah, I can't remember. I didn't wire a network cable for a while. I didn't touch an RJ45 for a while. A special moment happened after 2.0. A few years ago, none of us would have imagined that one single person would be running 1,500 cores. That's the beauty of the cloud. When 2.0 went up, I'm not going to talk about concurrency, but there were a lot of cores running under Star Citizen's name at our cloud provider. And having this overseen by one or two people, how cool it is that we don't really need to worry about the cables, the racks, all of this stuff at this phase.

So there is not a clear lock point. I mean, that's not my area of ownership; that's the area I'm the very first close neighbor to. I don't believe that we have a very, very specific lockstep where this is what locks the client.

Right. The client frame rate is usually the rate at which you're refreshing the graphics and so on. So on the server side, on the DGS, as we call it, that's the simulation rate, and they're not locked together.

Yeah, absolutely. I know a lot of players are concerned with the FPS. First of all, I want to thank every single person who plays PTU and all the amazing people who play Evocati, and the great people that I woke up during the night, like, hey, I have a new combination I want to test, and people calling each other and coming online. This is amazing. I couldn't have imagined this was even possible. In my interview a year and a few months ago, I was asking, how do you guys test? It's like, we use people. Like, what do you mean? It's like, our players are awesome. You just say you want to test that, and you'll find people coming from nowhere. So what you guys do at PTU is define pain points that should be addressed. Optimization is a deep rabbit hole where you always need to be worried about premature optimization. But also we have a product out in the market that you guys are playing, and we want to make it playable and fun. What I'm trying to do, with my team and with the guys working in Frankfurt and the guys working in the UK and over here and in LA, is to point them as much as we can to that main pain point, like, oh, if you spend some time on this, we might get some advantage. But what we have right now is nowhere near what we would deliver. We have so many tricks in our bag that we know it's not the right time to bring in. But we really need to care about, yeah, you guys were suffering that much and we enhanced it 10%, 5%, 7%; it's still livable. And we're still keeping an eye on what's going on. And also, sometimes we are not willing to enhance anything right now, but we want to see: did our new code logic or complexity tax us a lot? Did it become very expensive or not? So thank you so much, guys, for all the amazing work you do testing PTU.

Yeah, there's RAD Telemetry profiles being generated all the time, and that gets disseminated to each of the teams appropriately. People will have on their radar what is most expensive on the game servers as well as the client, and just tackle it at the right time. But there's just so many features going into the game. Like I said, premature optimization: once you start optimizing code, it looks a little different than normal.

Right. And there's a wide range of things that we can optimize. From the client side, you're going to have things from rendering, you're going to have client-side simulation, and also you're going to have network latency. Then on the back end, there's all the physics and all the other things that happen there. Then there's also things where the game servers may rely on responses from the services. So there's a wide number. We don't want to pre-optimize now, but we optimize certain things over time, the low-hanging fruit and such like that. There's so many different departments that are going to focus on their little areas, but we have to focus on them at the right time. If we pre-optimize now, we could end up painting ourselves into a corner later.

The website and the back-end services are about to have a very close relationship. Prior to this, it's always been us, the services, and platform. But we're now coming together so the services will look at them as another service and they'll look at us as part of their own system. So we're getting married.

We're using RESTful APIs right now, which work over the internet, but we're co-located now, with a dedicated channel, so we get bi-directional communication.

Yeah, we'll have a set of services that have stateful high-speed communication mechanisms to the platform, and we've been working with them on how to architect both sides to make that as streamlined as possible. And I'll use an example right now. When you buy an entitlement, you purchase a ship on the website, there's no way for the platform to tell us that you purchased a ship. So when you log in, we grab your package, which contains all the ships that you purchased and so on. And then we know about it, and then we can create the actual in-game items and persist them. But with this new high-speed communication system, the moment you buy a ship, they will notify us. We can expand the ship immediately and have it ready for you by the time you log in to the game.

When it comes to community-based features like statistics and achievements and things like that, a lot of those features really are best implemented with a web-based technology. If we have a direct line of communication, we can just trigger those things immediately.

And we'll be able to call functions from them directly as a service and so on. And there's an API we provide for them as well so they can request any data from us. We can generate events back and forth to each other.

It's one of the things that's so awesome about working with a partner like Turbulent.

Those guys are really good. They're awesome.

It's the first time I got on a project where the website was done before the game itself.

Yes. We all think about solving the problems in the same way, so it's a very good match.