Star Citizen Live: Builds, Platforms and Publishes

So, it obviously starts with the developer committing a change. Once that goes through that process, we have tools which allow the developers to kick builds from our release streams. Once the build is finished, we go through a manifest process and we create what we call the build manifest. And basically it's just the list of files that dictates what the launcher should patch down. And we upload that and kind of hand off to Ahmed from there.

Yeah, that's where the custody kind of switches from a team to another. But also we do not really have silos. All these teams actually worked together for so many years. So it can be that we can actually be following up on the build and the build system and trying to anticipate what could be wrong. And we could be also looking at deployment and see what could have gone wrong. So from that moment that we have a build, we have a manifest, it depends on the size of the deployment. Every deployment that is going to go to a single region, a small number of audience, a QA or a staging deployment, this is fully automated. You go through a web interface, you click Deploy, everything gets created from scratch, share nothing. VMs get created, a Kubernetes deployment for a new generation of services, a namespace gets created, a Helm deployment chart gets initiated. That chart is a global chart, has other charts underneath it. A lot of moving pieces, so many moving pieces get deployed so you can have that fleet of services and the simulators that indicate the game servers deployed in the cloud, accessible whether it's an internal group of people or external. If it's external, so far due to how rapidly things are changing, we have to have a captain, to have someone who's really running the deployment. Every deployment in that case is a Git branch. You go take our automation repo and you create a branch and that branch will have some of the facts that are specific to this deployment. These facts are always changing. We're allowing very last minute changes because we are a development company. We are really developing as we go. Now we're trying to take away any ability to change things very fast. So that deployment gets created. It's a blue-green deployment, and all the resources get allocated around the world, and it's hidden. At this time, our internal QA starts hitting it. Every deployment has a goal, too. Every deployment has a very specific goal, whether we're delivering a feature, we're testing a feature, we're collecting feedback, and QA hits that part hard, and there is another checklist with other departments. Everyone has to know this is a new game experience where we are exporting or publishing a full game experience. It's not a bunch of virtual machines just being deployed in the cloud, and everyone has to get an agreement, and all the work gets done, and then the players see the light come on in the launcher. And I think that's where you get to see more. I mean, platform is involved in all of that. All these systems are super, super involved. But platform, you get to see more like, okay, we're ready to publish right now. And I think Benoit can explain that better than me.

Yeah, so that's where usually my team comes in, where we've built tools to allow the deployment to go out. So that includes securing the build from who needs to have access to it. As Ahmed was pointing out, there's a lot of builds that players don't see, that are used for internal testing, that are used for QA-ing a specific feature or doing ad hoc testing on a specific refactor or something like that. So that is internal, but the moment it needs to go external, so either it goes on PTU or on production, that's where the launcher release tools come in. And so this allows us to take the build manifest that Miles was speaking about and basically make it available to players to patch. And so that's where the Delta patching technology comes in that's able to rapidly look at the build that you have and the build that's now available and compute the difference and then just give you the objects that you need to run the latest version of the game. Obviously, for that, there's a lot of user-facing services. And every time there's a user-facing service, that means a lot of responsibility related to security, scalability, et cetera. So that's mostly the platform teams that take care of the maintenance of those systems. But the operation of it is fully in the teams. So that's the great part. So that's pretty much the build process in a nutshell.

Sure. We actually maintain dozens of dev streams where individual features live and are housed, and as those features get completed, they're merged up into the more collective streams that get tested by QA as well, so we can kind of see how they play together before creating the release branch eventually. Usually once everything's ready for a 3.10 or 3.11, we'll branch that out and start cutting builds from there. We have a pretty robust inclusions process that allows us to strip it down and make sure things aren't going out that shouldn't be yet. And then as far as... I think I already touched on that. We kind of have a nice user interface that developers, anybody in production can go and request a build. So production a lot of times is the ones responsible for kicking the release builds and maintaining that pipeline of who does what, when, the builds, where. As far as how to get selected things, so obviously bug fixes go in our release streams. We have a bunch of internal automated tools that, as soon as changes are submitted to those streams, we start bringing those changes back down to the lower level developers and distributing them out so that everybody has access to those changes as quick as possible.

I think it's one of those things that maybe isn't always apparent, that everybody's working in these individual feature streams so that the actor feature team can work in their stream and the ship team can work in their stream, the ship art team can work in theirs, the feature team is this one, our team is this one, and they're all working in these different streams so they don't step on each other's toes and overwrite each other's work while they're working. And then that process of integration is where bugs are born.

It's huge. We actually manage quite a few microservices that kind of back the whole build process, and that's probably the most heavily used microservice, that integration process, at the moment. It's thousands a day, easily.

Well, so the answer is complicated, but we do use a plethora of databases. Platform systems tend to rely on structured data. They're going to mostly leverage technologies like MySQL or MariaDB, which is the same kind of engine. But since a lot of the game systems are also made of microservices, then each service can actually use its own data store. And so we have the ability to use the right data store for the right use. So it's not like NoSQL for everything or SQL for everything, right? This use case needs a structured dataset, and so we're going to use this engine. In general, the technologies that are in use in the game today and in platform, both of them share a lot of these technologies, but MySQL is used to persist a lot of the data on both ends. MongoDB is also a big player in a lot of the dynamic variable systems that power the game. And Redis is a real workhorse for a lot of the really real-time stuff like Spectrum unread counts or distributed locks. We're going to use Redis for a lot of this, which is really our Swiss army knife of databases there.

We can tell everything because it's a public offering, everyone knows about it. We use a large variety of VM sizes and families, but the major common factor between all of them is AWS Nitro, which is a key lightweight VM technology that AWS introduced with the next generation or new generation. When it comes to the game servers, the game servers run in C5. If we have some backers watching the livestream from the days before 3.0, we were doing some comparisons between the C4 and the C5. These were changes in the hypervisor, changes in the CPU powering the VM underneath. When it comes to the cloud, you're usually trying to hit a very specific sweet spot for your workload. So for the game servers, we have to meet in two areas. We need to maintain a decent ratio between the memory and the compute. And we need to make sure that we have very fast compute to see if I have fed that very well. We have other services that are single threaded that we need a very, very fast single core. We have other services that require moderate CPU clock speed, but they have larger memory usage. They will be running in R5. So it's all in our availability. A lot of our workload, we try not to be stuck to one single size, because we need to make sure if there is any issues and we cannot allocate VMs from that size, we always have a backup size. We have backup regions behind our regions, but sometimes you have to choose your battle, whether you're willing to go to a second region to allocate resources or you're willing to go for the second most optimal machine size. I hope this is not so boring, but we use a lot of them and we try to use the new ones.

The thing that's interesting about that is that you realize that sometimes you try to get a VM from the cloud and you can't, because they're out of VMs.

Yes, we are very proud of our auto-scaling system. We have a buffer-based auto-scaling system. Game servers are slot-based. So the game instance will take a specific number of slots. You cannot overload that. So when you auto-scale a system like that, you are bound to use what we call a buffer system, where you always have a step ahead of you of VMs or instances, both in the logical and the physical level. That buffer size, how much you maintain ahead of you, how fast it takes an instance from scratch to load the map and be ready for the players, these are adjusted. This auto-scaling system works on a regional level. So the guys in Sydney, it could be a rush hour for them. So we start spawning VMs over there. The US and the EU are not affected. It's a very essential system for us because gamers' behavior is always up and down, and we cannot have resources idle around. These are quite large VMs, so we cannot just have them hanging out for no use.

Sure. Well, I think it's been about four years since we last talked about this topic, so we've pretty much upgraded across the board, be that storage, RAM, adding more CPUs. We've also spent a lot of time utilizing all the hardware a lot better, with better build distribution across the machines. And a lot of that has helped greatly over the last four years. So we used to take at least six hours per build back then. We could only have about two running at any given time. Nowadays, we're sitting at about three to four. We can have up to four builds running simultaneously. And we probably put out 12 to 15 builds a day now, pretty regularly, especially with all the feature stream support that we've added over the last year or two. The builds are getting kicked very regularly, and it's almost a constant queue and stream of builds coming through.

That's interesting. So there's been a lot of work there in the past to try to get that up to speed. Nowadays, there are multiple ways where stability is being tested. So on the platform side, for example, we have a set of tools built on Locust, which is a load testing framework written in Python, which is super nice. It allows us to basically simulate thousands and thousands of users so we can write scenarios about what they do. For example, melt a ship. What if X amount of players melt a ship? How does the system react? We can test that. We've used the same principle in the same framework to build a game-facing Locust system that we call Pestilence, which is basically, yes, I know, the naming is nice, but basically what Pestilence is able to do is mimic what you guys do in game. So it's able to log into the game, connect to the servers, and then do specific actions that we want to test. So let's say we have a service that has an issue and we're trying to see what's causing issues. We can actually write a scenario that does that, and then we can spawn thousands and thousands of locusts doing that in game to simulate what happens when we actually launch a build, because that's usually what happens: there's a big storm of people doing the same action. So we can test stability this way. Now it's getting more and more integrated into the process, where we write scenarios for different services as we go. Recently we've added the ability to even provision account data from it, and so we're able to pre-warm some of the databases to prevent some services from getting hit too hard when the patch is launched, things like that. So that's the major thing that we do for service and platform specifically for load testing. On the game side...

These are headless clients, obviously.

Exactly. Those are even more than headless clients. They're not even running the game itself. They're literally just running the cloud side of the game. But we do have headless clients as well. That's where I was going. There are automated systems, which is called a cloud test launcher, which can actually do that running the game client as well. But the testing there is more focused around testing the game server performance than the MMO side of the game.

Yeah, 100%. We do not discriminate. They all have the same spec. What really happens is player behavior affects a lot what's happening in the game server and also your latency. The dedicated game server is a very, very sensitive workload. It's very sensitive to any kind of overhead, sensitive to any kind of latency issues. So no, everything is at parity. We do not discriminate. Actually, the only workload that we have that always runs on the same exact platform, no change, no matter what the environment is, is the dedicated game server. There are other services that we can give less resources in smaller environments because we know they're not going to get hit hard. When it comes to that simulator, called the dedicated game server, we always make sure that it has the same exact spec everywhere it goes.

I know the second half of the question was, I'm guessing this person is from the Australian region, and they were saying that it seems a little more sluggish, but as far as the actual hardware behind the servers, they are at parity.

It is the same, yeah. We actually care a lot for our customers in Sydney. One time we had issues in the Sydney VPC and we had to allocate some VMs in Mumbai, in India, which is far, but still it was the closest area in AWS, and we got really upset and we're like, super fast, we're going to fix this issue and go back to Sydney. If they're having any issues, I guarantee them it's not related to the hardware.

Well, there's always a lot of load on the system, so that could always cause times to fluctuate quite a bit. There's a lot of uploading that takes place during that, so sometimes the network can be bogged down.

Yeah, I was about to add to that. It's not just compilation time. There's a lot of fan out and fan in. There's a lot of steps. We go do a lot of things in parallel, and then we collect them back in. And these are systems that sometimes are under development themselves. Sometimes we make changes that make things take longer or take shorter. But yeah, if we're talking in general, like Miles said, between three to four hours is our average.

And you're right, Ahmed, in that when a build is finished cooking, it's not ready to be distributed. It's because it needs to be uploaded so that it can be distributed. And that step is fairly long as well. That's why the total time is three or four hours.

The client compiles themselves are relatively quick. The data is typically what takes the longest. And that can go for an hour or two for sure.

Yeah, that's been the idea since the beginning, right, is that the report that's filed and confirmed on the Issue Council gets, if you see the CIG icon on it, then it's been imported to Jira, basically. And so at that point, it can be tracked internally, it can be assigned to, because developers need a Jira ticket to work or check in, right? And so that's the way we legitimize the changes that they have to do. And so, yeah, Issue Council is connected the same way to feed the status back from Issue Council. We're aiming to do improvements to that flow to give more visibility, but we still have a long ways to go before we can hit that. The roadmap is the same thing, the current roadmap, so the cards that you see on the roadmap also come from Jira, so this is a live representation of the production as it stands the moment it gets put up there. So yeah, it is connected, and that's the key: it's the data source and the data destination.

I'll talk about the deployment side. So when it comes to deployment, I kind of touched on that earlier, on the deployment size. If it's a smaller deployment that allocates a reasonable amount of cloud resources, this is fully automated. Anyone can go through a GUI interface and click. And these deployments, you are not allowed to change so much of them. You can only change a few specific facts. And it's based on a system that runs off Git, off a source version control system. And it gets deployed automatically, everything gets built, and everything can get destroyed also by a GUI, by a user interface. When it comes to any large public facing environment, we're talking about a lot of resources. So in the current phase of the project, because we also still have systems or areas in automation that are imperative, where we tell it how to do things, not what we want. That's a very important topic. A lot of people I see in the questions are from the DevOps community. That's a big thing. We're trying to shift a lot of our stuff from telling the automation how to get there to what we want, and let the automation systems and frameworks obtain that state. But to go back to the point, if you have a larger environment, at the initial part of it, you have to have a human decision, you have to have a review, another human decision, making sure that these are the correct values, and then it goes from there and it becomes automated, and autoscaling kicks in and manages the environment from here.

And just to add something real quick, as the project matures, as parts of the project mature, as we know we're not going to be moving a lot of knobs here, we can offset these parts and make them super self-maintained. Self-healing is a big topic in our industry and we want to talk about it. Reactive environment, the environment that reacts to itself, and we might get to talk about that a little bit, about how we are trying to enhance the metrics, or the traces, or the logs, or the insights inside our services, so we can let our infrastructure make decisions based on these insights.

I mean, one key part that's still a human intervention, I think, both on platform and game, is the flip of the environments, right? So the game, as Ahmed was mentioning, is deployed using a blue-green deployment. That means that while you guys are playing the game online on prod, for example, and we're preparing a new build, that's the blue one. And then the green one's being built, or vice versa. And so at some point when the build's validated, the flip happens. And a human says, OK, we're flipping. And that's where the build goes live. It gets available in the launcher. Servers are flipping over and rotating. And so that's still a human doing that. And it actually requires a couple of people saying yes, yes, yes before it actually happens.

It's very similar to the build system as well, which is highly automated, but somebody always has to make the decision to request a build, even if that's a release build.

We do them all. We discussed in the beginning how we do blue-green. Blue-green is usually when we are delivering a lot of breaking changes. A lot of things are like, okay, we cannot have half and half. These things are not going to play nice together. And this is a product currently in development, so we need to get you guys into new stuff so we can see what's going on. From the moment that we publish a big build to a large environment like Live, we can do what we call hotfixes, we can do between 5 and 20 hotfixes. And these are either canary released or rolled. Like in 3.10.0, we canaried six hotfixes for the hub server and we rolled the last one. And that happens in every single aspect. There are areas of the game that can allow for some toggling, where we can toggle features on and off, or hot reloading or hot changes in data. And there are other areas that, because the challenge in running a game in the cloud is the statefulness. The experience between a very immersive game like ours, a game client like ours, and the game server is super stateful. This is a very fragile experience. If you did anything here, you lose that connection. And we are actually always discussing how we can make that less fragile, so people like us do not have so many precious parts of our environments that we cannot touch if the players are on. But we do both. We do canary, we do rolling, and we also initially do a blue-green. Hopefully in the future, as the project matures, we start dealing with the concept of continuous deployment. It's an environment that's always going and there's always pieces moving all the way around. But we still have to do a lot more things before we get there.

To get a build, there's only two. It'd be the developer making the commit and then the developer kicking the build. Depending on what environment you're wanting to deploy to, it could be potentially one more human interaction, which uses the same interface we kick our builds through to deploy to our internal staging environments. I think Ahmed could address outside of that for me.

If it's a larger environment, it goes through people deciding that this is the build we want to go with, and then from then the humans just make the decisions, what we were talking about earlier. Everything is driven by version control, everything is driven by Git, but there is a human saying yes. And we did not abstract that into a GUI yet, we didn't see the need for that. We call ourselves DevOps. I know this is some kind of heated topic for a lot of people. DevOps is a philosophy, not a title, but that's what the market goes for. But we try to really hold the values of DevOps close to us. We share the ownership of that product. This gaming experience is ours. I can spend more than 50% of my time investigating an issue about how a player isn't able to connect to something, because I have the access to all these tools and I can try to go and investigate it. To go back, when it comes to deployment to a larger environment, that's usually where human decisions are being implemented, but all the actual work of provisioning, configuring, securing is automated. It would take days, it would take weeks to deploy the whole environment manually with how it spreads around the world.

I'm thinking 3.0 for me, because the patch before that was so far, right? And to echo what Ahmed was talking about in terms of DevOps philosophy, if you don't deploy for a long time, then you forget how to deploy, right? And so 3.0 was the longest running in-development patch of Star Citizen, I think, that we've had, right? And launching that touched on everything. Everything was new. It was the first for everything. So it's like a brand new game. And so for me, that was by far the most difficult to get across the finish line. And that's when going to quarterly made it a whole lot better, because now we can deploy more often, which is how you get this rhythm going, right? And keeping it going. It also means, to be fair, the more you deploy, the more you break stuff as well, right? So there is a risk there. Don't touch it. It's working. Don't touch it. It's good. But yeah, 3.0 for me by far. I don't know for you guys. How about you, Miles?

My problems are very different usually. I'm more heavily involved at the beginning of the cycle, less at the end. 3.9 was actually pretty challenging to transition to, as far as getting the new stream set up and making sure all the changes were integrated to where they needed to be. We definitely had a good time with that one.

There's a lot of workflow changes in 3.9 for sure.

For me, it's a difficult take between 3.0 and 3.9. 3.0, definitely. 3.0, we rolled out a new hypervisor, a new generation of virtual machines, and a lot of new things, and it was in a very critical time. I think we have an episode of one of our shows, that I'm not very good at keeping up with, where we talk about what happened in 3.0. In 3.0, there was a chat between CR and Tony and us. I still remember a very, very dear message to me, when Tony said, I got to go put some gifts under the tree and come back real quick. That was 3.0. And it was a lot of fun. 3.9 was a lot of fun too. It was like Fleet Week. Wow. We are spoiled by our player base. Our player base are actually developing this game with us. And I'm not just saying that, because you really see that. And 3.9 was a lot of people doing a lot of things. We were learning a lot. We cannot stop learning from what happened. One of the main ideas in DevOps is to measure everything. Testing a game is challenging. It's relatively doable to test RPCs or API calls, but the immersion, how the game runs and everything, that is difficult. In 3.9, we had that. They're testing everything everywhere. They're doing everything. And we were receiving all this data and digesting and reacting. I do not have an accurate number, but the number of hotfixes that went in that week and a half was really, really high.

This actually went pretty well. It was before 3.0, though. We got to run the Delta patcher on PTU for like a month and a half. I just remember my biggest memory about launching the Delta patch system was the moment we had the ability to push a build a day to players, we did that for 48 days straight.

I do have very vivid memories of converting the build system to support that. That was definitely a challenging time as well.

I have a sad memory from 2015 of coming to the office at 1 a.m. so we can prepare the old-school patch so we release it at 6 a.m. And that was voluntarily. We would go and work on it, because it used to take hours to prepare these patches and make sure you have all the diffs between all the old versions. You actually bring good memories about the Delta patcher. And right now, sometimes we look at the CDN graphs and we see these petabytes coming out. Can you imagine if the Delta patcher was not here, how much data we'd be pushing? It's one of these things that we take for granted now, but it was an amazing big change for us.

Critical technology for sure.

Oh, well, that's interesting. So, not really. In fact, the long-term persistence was engineered to basically persist your data that you earn in game between patches, right? So it saves your money and the history of all of the UEC money that you've got. So every transaction up and down, current values of all your ledgers. The inventory system that's part of LTP is also there to store, hey, you own a Hornet, right? Or hey, you own a Constellation, right? The state of these objects, so, oh, this Hornet's missing its left wing, it was abandoned at GrimHEX. So that's stored in the persistence system. So iCache will basically bring more state tracking, more state persistence. But we're still going to persist the same amount or the same kind of records between patches, to be able to rebuild the state when we do a data wipe on the iCache side. And the reason why that's still an issue is because the schemas of those objects are still moving. And so we still need to do that. But the only change that will come from it is that there are some parts of the iCache API that allow better handling of stacking objects. So let's say you buy 50 magazines, right? The way that's handled in iCache is much better. And so the LTP systems will track that better. And so that's not the only change. But the functionality for players should function the same way.

Yes, so as we take some of our legacy systems from imperative deployment, where we are describing to the automation system how to do things, into more declarative systems. We don't want to throw in a lot of jargon, but people who work in DevOps technology probably know what we're talking about here. That is very exciting for us. We're also looking at how we can leverage some of the bare metal resources that a lot of cloud providers allow now. We've always been trying to squeeze as much power as we can get, and we want to see what will happen when we aren't going to have the hypervisor anymore, maybe we'll get some more power out of it. And again, we are not, till we measure... So we measure things and we see benchmarks. But yeah, I'm excited for these two things. One of the most important things I'm excited for, we have two new engineers that will join the team on Monday. So that is a big thing. Hopefully more hands on the keyboard, more minds, more people designing all these challenging solutions with us. So yeah, a lot of good stuff happening.

It's a very collective, democratic call. The way it works: whenever we are in the mode of releasing, we have the live sync meeting, one of my favorite meetings. We get to have it early in our day, towards the end of the day in Europe, and we bring up a lot of data. Whenever we go to PTU, there are several types of reports. One of them is a stability report. The stability report tries to model how much disruption we are causing to the player experience. And then we have player feedback. That comes from the player experience team, and it takes into account all the Issue Council issues, the feel of the game, if there are specific features in the gameplay. QA also goes and takes that report and goes through it and tries to reproduce, by the next morning, all these... Data gets collected, we have performance data, and then everyone has a case, right? And it's a lot of conflicting factors together that decide. Sometimes we want to progress faster between waves because we need more players to see how things are going. And you get to decide what the messaging is going to be like, what's the trade-off. Okay, it's going to be amazing in all these areas, and this feature is going to be really bad. I think our players, our backers will understand that. We can just explain it: avoid this area. It will be all right. And yeah, it's a lot of moving factors. When it comes to benchmark, it's quantified, collected crash information, stability information, performance information, resource usage information, all collected together, all abstracted in a number, in a model that we call the stability score. And on the same side also, we have hard-working teams that try to collect and summarize the game experience. And both of them get together, and then we have different goals and different groups and a very democratic way of deciding how things would go. And yeah, hopefully we make the good decisions every time.

Even development teams like some of the game service guys are going to be watching live PTU builds to see performance metrics and get a sense of it. And so that's also in the mix. So you're right. It is a hodgepodge of different pieces of data.