Ahmed Shaker

I mean, it is pretty close. The fact is we do not just own the infrastructure that the game runs on. We take custody of the game from the moment it comes to us. So we pride ourselves on being experts in how the game runs. Sometimes the developers who work in the code themselves come sit with us and get amazed by how we get to know where the issue might be before we even get to look at logs or look at graphs or look at any kind of data to figure out what's going on. The game is a living creature. It's in its own high fidelity, and the dedication that we see from the community. We also get to see that our community does not deal with it as a rigid system. They try to always interact with it in a very, very unique way. We saw a lot of that, especially after Persistent Entity Streaming was released. And we like the challenge. We like what they bring to us. They try to break our systems. They try to break the game. So there's always a lot of fun going on. And again, it's not like you're hosting any kind of very established static piece of software, making sure it has connectivity and IO and compute. No, this is an ongoing thing that's always evolving. And a lot of the time we learn about the software we're running as it ages. So as a lot of the backers remember, we have publishes that could live for a few days. We have publishes that could live for 90 days. And every single day there is a new thing to learn. So yeah, there's a lot of holes to fill and a lot of plates to spin.

Interesting. So, moving to the US: I'm from Egypt, and during my stay in Egypt, a group of friends and I had our own little shop where we used to support startups around the world with all their technology needs when it comes to infrastructure work, DevOps work, even predating the word DevOps, predating cloud. And one of my specialties was building private clouds before the word cloud was also pretty cool. I've been in this field for quite some time. And moving to a new place, I wanted to find a cool story to attach to. And job hunting, I'm out in Texas trying to find something really unique, and I stumbled upon one of the DevOps positions in Star Citizen. I still remember, I met Francesca a few days ago, and I was remembering when I first sent the cover letter. Back when I was in Egypt, I had one of my clients that was a gaming studio, not anywhere near the size of CIG, but it was one of my favorite clients to visit, because you get to see so many aspects when it comes to computer science. And you also get to see the artistic aspect. That is not really something you find in a lot of other clients when it comes to people that would use infrastructure. So I applied for many jobs and I hoped that this would be the one. And I think a few hours later, I got a phone call. I think the cover letter made an effect. And I think a week later, I was meeting with the team for our first interview, and the journey started from there. In between, I got to consume a lot of video content. I mean, at that time, and still till now, we like to share a lot about our vision. And it was fascinating to see that there is a vision to create an MMO with high fidelity, cloud based, with the latest and greatest technology. And I like being on the bleeding edge. I like challenges. So it kept going from there.

No, I just had one of my clients back in Egypt that was a gaming studio, and it was like one out of 35 clients. But again, there has always been that fascination with gaming and how games are being made. I still remember being very young and sitting after the end of every game to read the credits. And there has always been the aspect that you do not see as many guys with a name like mine in the credits. So it was also another interesting bet: how can I navigate that world? How can I make it there? And still till now, one of the most rewarding areas of working, especially now that I was here attending a summit, is you get to work with brilliant minds. You were relating our field to the medical field. I think it's a very, very good analogy. So we have brain surgeons, we have heart surgeons, we have cancer research guys, and it's a great privilege to work in a shop that really cares about computer science and gets to use the magic of that science in so many areas, and then you get to have the art on top of it. And in our case, the lore of the game is fascinating, to see how it relates to human civilization in different areas, how it relates to some World War II style, how it relates to old civilizations that clashed. So I don't want to sound cheesy, but it's very hard not to sell our project to anybody else. It's one of a kind. So it was not hard for me: once I saw the job posting, I said, this is the one that I'm going to put all my effort in the cover letter, and here we are.

So talking about what John was talking about, about how games push boundaries, that is so true. And it's one of the areas that we're always chatting about; it always comes up. Like right now, everybody's obsessed with AI. When you think about it, AI is powered by GPUs, and GPUs came to exist because of video games. So kind of a similar analogy: when 3.18 got published from PTU to Live, a lot of players did not know that the typical ratio between PTU and Live is usually 25x. The problem with 3.18, though, is we decided...

One of the problems.

One of the problems, that is true. But it starts from the fact that our universe, with all of our moons and planets and players and ships, we decided to persist everything we can as often as we can. And that is a recurring theme in Star Citizen. A lot of players might not notice that we challenge ourselves as early as possible, instead of waiting till we reach the million or the two million or the three million concurrent players. So we try to amplify the challenge all the time by avoiding any kind of hacks or shortcuts. And you see it in every single aspect, but I get to see it myself as well in my field. So I still remember when we first published, our public cloud providers reached out to us and said, what is wrong with you guys? Like, what is wrong with us? And they said that we're moving an insane amount of traffic between zones. Some of our backers are technologists, and they know when you deploy resources in the cloud, you have to split your resources across zones. You cannot sit in a single zone because zones can go down, and they do go down quite often. So the first few days, and the first few days, by the way, we were not peaking concurrency at all. A lot of backers chose not to play the first few days. They like to wait, because a lot of them learned that we hotfix a lot of issues behind the scenes. We were pushing 1.2 petabytes of hot traffic every single day, and that's between zones. That's not between every single server. We still have traffic inside the zone.

You said 1.2 petabytes?

Petabytes every day, between zones, in one region, not in all of our regions. We exist in four regions.

And that's just unheard of.

Yeah, the Amazon guys were like, we've never seen this before. Not like you've gone up from what you're used to. They were like, we don't have any other customers that are pushing this kind of traffic.

A day. And when you think about it, like social media: a human being has to create content, the content has to be uploaded, and it has to be interacted with. And most of that is happening in eventual consistency, running through compression. The fact is that every single simulator that we have... and right now, while we have instances with a reasonable amount of players, we have copies of our universe. All these copies are producing data as fast as the system can handle. And we are a competitive group of people, so none of us is telling everybody else, hey, please slow down a little bit. No, everybody's saying, give me more. And that goes all the way down to the wire, to the database. And I know a lot of players, whenever they hear the word database, they get really upset. The bit that I would like for some of them to understand is we really don't want to hold our innovation behind the scenes for too long. They get to see right now we're testing Server Meshing and server crash recovery in Tech Preview. Each and every one of us cannot wait for that to go to Live. So 3.18.0 is one of these stories where we could have stayed longer till we figured out every single little problem that could happen. But they were hungry and thirsty and really very eager to play that version of the game. So thanks to all of them for staying with us till we figured some of the kinks out. We still have a lot more to go, but we will keep pushing, because that's what we do. That's why we're all here.

The first one was that when I went to my interview, I wore a suit, and I'm standing outside trying to get in, and one of our co-workers comes out in baggy shorts that have paint on them and just runs right by me, and right behind him I see one of our co-workers in a tank top, and I was like, I made an awful mistake, I'm wearing a full suit. And then I got in, and you see a dark space with a lot of artists and a lot of cool-looking people, and you're just walking around with a leather notebook and wearing a suit. And I got to have my interview in what we call the war room. And at that time, the AC used to have a very interesting vibration, and I have a very sensitive ear. So yeah, that was my very first impression. And I came the first day to work, and I'm not sure if Mike will like the story or not, and I found a Windows PC. And I had not been a Windows user for quite some time. It's quite common in my field that people use Linux. And I was like, what is this? They're like, it's your computer. And I think that's how Mike and I got to be friends and know each other. He came to my office and knocked on my desk, and he said, my name is Mike Jones, and that's all you need to know, and walked away from me. And the next night we were hanging out assembling my Fedora Linux machine, and that's how things went. But if you're talking about memorable moments, I still remember our concurrency. We had times when we used to cheer because we had 200 players online, and we're a big magnitude above that right now. I still remember when we first launched Area 18 and the whole QA team had their buggies with them by the elevator for every player to come in and ram into them.

I have a confession on that. So we're about to put Area 18 in for the first time, and in one of my very first conversations with Jake Ross... Now, I'm going to preface this by saying I didn't know what my limits were or what I should be involved in or what I shouldn't be involved in. I was kind of all over the place in 2015. Anytime somebody did something we didn't have a person to do, I'm like, I'm going to do it. I was the events manager for a while. I was the platform guy before Eric Green. Eventually, like, wait, Jared's doing the platform? Oh, we need to hire somebody. Keep Jared away from the platform. But we're a day away from launching Area 18, and I go to Jake Ross and I say, you know what would be fun? Put some Greycat buggies in there. And he goes, oh, yeah, we should put some Greycat buggies in, let people drive around. And it was just this innocent little suggestion, and it took 24 hours to turn around. And then when Area 18 launched for the first time, it was carnage. It was absolute carnage. People were just mowing everybody down with the Greycats. The Greycats, you could look at them intently and they would explode and whatnot. But I remember Jake Ross did not throw me under the bus. When Chris goes, where did the Greycats come from? he was just like, it was a decision made internally, we'll take them out in the next build. But he had a chance to throw me under the bus that day, and he didn't.

Yeah, so many great moments. 2.0, I still remember, I think it's one of the longest times I stayed awake. Not because I was able to do much; we were crashing, and then watching the streamers just counting the crashes every single time, and they keep going back in again. 3.0, I think one of our coworkers, I'm not going to name him, was about to propose. And he was literally on his laptop right in front of their in-laws saying, hey, guys, let's launch the game so I can go inside and meet these people for the first time and prove that I'm a responsible man. So we have a lot of amazing moments, and I think right now, we're talking a lot about the future because we're here to have a summit, laying out the plans for the next two years, and we just keep dreaming of what 1.0 will look like and how we get to see the game get into the shape that we all dreamt of. So yeah, it's very difficult to pick a certain moment, but there's a lot of memories, a lot of sleeping under the desk. I think you and I one time were on a call at 2 a.m. in the morning, and I was leaning back on my desk, and I flipped backward. I'm not sure if you remember or not. And we were like, that's it, Ahmed's gone.

So, it obviously starts with the developer committing a change. Once that goes through that process, we have tools which allow the developers to kick builds from our release streams. Once the build is finished, we go through a manifest process and we create what we call the build manifest. And basically it's just the list of files that dictates what the launcher should patch down. And we upload that and kind of hand off to Ahmed from there.

Yeah, that's where the custody kind of switches from a team to another. But also we do not really have silos. All these teams actually worked together for so many years. So it can be that we can actually be following up on the build and the build system and trying to anticipate what could be wrong. And we could be also looking at deployment and see what could have gone wrong. So from that moment that we have a build, we have a manifest, it depends on the size of the deployment. Every deployment that is going to go to a single region, a small number of audience, a QA or a staging deployment, this is fully automated. You go through a web interface, you click Deploy, everything gets created from scratch, share nothing. VMs get created, a Kubernetes deployment for a new generation of services, a namespace gets created, a Helm deployment chart gets initiated. That chart is a global chart, has other charts underneath it. A lot of moving pieces, so many moving pieces get deployed so you can have that fleet of services and the simulators that indicate the game servers deployed in the cloud, accessible whether it's an internal group of people or external. If it's external, so far due to how rapidly things are changing, we have to have a captain, to have someone who's really running the deployment. Every deployment in that case is a Git branch. You go take our automation repo and you create a branch and that branch will have some of the facts that are specific to this deployment. These facts are always changing. We're allowing very last minute changes because we are a development company. We are really developing as we go. Now we're trying to take away any ability to change things very fast. So that deployment gets created. It's a blue-green deployment, and all the resources get allocated around the world, and it's hidden. At this time, our internal QA starts hitting it. Every deployment has a goal, too. Every deployment has a very specific goal, whether we're delivering a feature, we're testing a feature, we're collecting feedback, and QA hits that part hard, and there is another checklist with other departments. Everyone has to know this is a new game experience where we are exporting or publishing a full game experience. It's not a bunch of virtual machines just being deployed in the cloud, and everyone has to get an agreement, and all the work gets done, and then the players see the light come on in the launcher. And I think that's where you get to see more. I mean, platform is involved in all of that. All these systems are super, super involved. But platform, you get to see more like, okay, we're ready to publish right now. And I think Benoit can explain that better than me.

Yeah, so that's where usually my team comes in, where we've built tools to allow the deployment to go out. So that includes securing the build from who needs to have access to it. As Ahmed was pointing out, there's a lot of builds that players don't see, that are used for internal testing, that are used for QA-ing a specific feature or doing ad hoc testing on a specific refactor or something like that. So that is internal, but the moment it needs to go external, so either it goes on PTU or on production, that's where the launcher release tools come in. And so this allows us to take the build manifest that Miles was speaking about and basically make it available to players to patch. And so that's where the Delta patching technology comes in that's able to rapidly look at the build that you have and the build that's now available and compute the difference and then just give you the objects that you need to run the latest version of the game. Obviously, for that, there's a lot of user-facing services. And every time there's a user-facing service, that means a lot of responsibility related to security, scalability, et cetera. So that's mostly the platform teams that take care of the maintenance of those systems. But the operation of it is fully in the teams. So that's the great part. So that's pretty much the build process in a nutshell.

We can tell everything because it's a public offering, everyone knows about it. We use a large variety of VM sizes and families, but the major common factor between all of them is AWS Nitro, which is a key lightweight VM technology that AWS introduced with the next generation or new generation. When it comes to the game servers, the game servers run in C5. If we have some backers watching the livestream from the days before 3.0, we were doing some comparisons between the C4 and the C5. These were changes in the hypervisor, changes in the CPU powering the VM underneath. When it comes to the cloud, you're usually trying to hit a very specific sweet spot for your workload. So for the game servers, we have to meet in two areas. We need to maintain a decent ratio between the memory and the compute. And we need to make sure that we have very fast compute to see if I have fed that very well. We have other services that are single threaded that we need a very, very fast single core. We have other services that require moderate CPU clock speed, but they have larger memory usage. They will be running in R5. So it's all in our availability. A lot of our workload, we try not to be stuck to one single size, because we need to make sure if there is any issues and we cannot allocate VMs from that size, we always have a backup size. We have backup regions behind our regions, but sometimes you have to choose your battle, whether you're willing to go to a second region to allocate resources or you're willing to go for the second most optimal machine size. I hope this is not so boring, but we use a lot of them and we try to use the new ones.

The thing that's interesting about that is that you realize that sometimes you try to get a VM from the cloud and you can't, because they're out of VMs.

Yes, we are very proud of our auto-scaling system. We have a buffer-based auto-scaling system. Game servers are slot-based. So the game instance will take a specific number of slots. You cannot overload that. So when you auto-scale a system like that, you are bound to use what we call a buffer system, where you always have a step ahead of you of VMs or instances, both in the logical and the physical level. That buffer size, how much you maintain ahead of you, how fast it takes an instance from scratch to load the map and be ready for the players, these are adjusted. This auto-scaling system works on a regional level. So the guys in Sydney, it could be a rush hour for them. So we start spawning VMs over there. The US and the EU are not affected. It's a very essential system for us because gamers' behavior is always up and down, and we cannot have resources idle around. These are quite large VMs, so we cannot just have them hanging out for no use.

Yeah, 100%. We do not discriminate. They all have the same spec. What really happens is player behavior affects a lot what's happening in the game server and also your latency. The dedicated game server is a very, very sensitive workload. It's very sensitive to any kind of overhead, sensitive to any kind of latency issues. So no, everything is at parity. We do not discriminate. Actually, the only workload that we have that always runs on the same exact platform, no change, no matter what the environment is, is the dedicated game server. There are other services that we can give less resources in smaller environments because we know they're not going to get hit hard. When it comes to that simulator, called the dedicated game server, we always make sure that it has the same exact spec everywhere it goes.

I know the second half of the question was, I'm guessing this person is from the Australian region, and they were saying that it seems a little more sluggish, but as far as the actual hardware behind the servers, they are at parity.

It is the same, yeah. We actually care a lot for our customers in Sydney. One time we had issues in the Sydney VPC and we had to allocate some VMs in Mumbai, in India, which is far, but still it was the closest area in AWS, and we got really upset and we're like, super fast, we're going to fix this issue and go back to Sydney. If they're having any issues, I guarantee them it's not related to the hardware.

Well, there's always a lot of load on the system, so that could always cause times to fluctuate quite a bit. There's a lot of uploading that takes place during that, so sometimes the network can be bogged down.

Yeah, I was about to add to that. It's not just compilation time. There's a lot of fan out and fan in. There's a lot of steps. We go do a lot of things in parallel, and then we collect them back in. And these are systems that sometimes are under development themselves. Sometimes we make changes that make things take longer or take shorter. But yeah, if we're talking in general, like Miles said, between three to four hours is our average.

And you're right, Ahmed, in that when a build is finished cooking, it's not ready to be distributed. It's because it needs to be uploaded so that it can be distributed. And that step is fairly long as well. That's why the total time is three or four hours.

The client compiles themselves are relatively quick. The data is typically what takes the longest. And that can go for an hour or two for sure.

I'll talk about the deployment side. So when it comes to deployment, I kind of touched on that earlier, on the deployment size. If it's a smaller deployment that allocates a reasonable amount of cloud resources, this is fully automated. Anyone can go through a GUI interface and click. And these deployments, you are not allowed to change so much of them. You can only change a few specific facts. And it's based on a system that runs off Git, off a source version control system. And it gets deployed automatically, everything gets built, and everything can get destroyed also by a GUI, by a user interface. When it comes to any large public facing environment, we're talking about a lot of resources. So in the current phase of the project, because we also still have systems or areas in automation that are imperative, where we tell it how to do things, not what we want. That's a very important topic. A lot of people I see in the questions are from the DevOps community. That's a big thing. We're trying to shift a lot of our stuff from telling the automation how to get there to what we want, and let the automation systems and frameworks obtain that state. But to go back to the point, if you have a larger environment, at the initial part of it, you have to have a human decision, you have to have a review, another human decision, making sure that these are the correct values, and then it goes from there and it becomes automated, and autoscaling kicks in and manages the environment from here.

And just to add something real quick, as the project matures, as parts of the project mature, as we know we're not going to be moving a lot of knobs here, we can offset these parts and make them super self-maintained. Self-healing is a big topic in our industry and we want to talk about it. Reactive environment, the environment that reacts to itself, and we might get to talk about that a little bit, about how we are trying to enhance the metrics, or the traces, or the logs, or the insights inside our services, so we can let our infrastructure make decisions based on these insights.

I mean, one key part that's still a human intervention, I think, both on platform and game, is the flip of the environments, right? So the game, as Ahmed was mentioning, is deployed using a blue-green deployment. That means that while you guys are playing the game online on prod, for example, and we're preparing a new build, that's the blue one. And then the green one's being built, or vice versa. And so at some point when the build's validated, the flip happens. And a human says, OK, we're flipping. And that's where the build goes live. It gets available in the launcher. Servers are flipping over and rotating. And so that's still a human doing that. And it actually requires a couple of people saying yes, yes, yes before it actually happens.

It's very similar to the build system as well, which is highly automated, but somebody always has to make the decision to request a build, even if that's a release build.

We do them all. We discussed in the beginning how we do blue-green. Blue-green is usually when we are delivering a lot of breaking changes. A lot of things are like, okay, we cannot have half and half. These things are not going to play nice together. And this is a product currently in development, so we need to get you guys into new stuff so we can see what's going on. From the moment that we publish a big build to a large environment like Live, we can do what we call hotfixes, we can do between 5 and 20 hotfixes. And these are either canary released or rolled. Like in 3.10.0, we canaried six hotfixes for the hub server and we rolled the last one. And that happens in every single aspect. There are areas of the game that can allow for some toggling, where we can toggle features on and off, or hot reloading or hot changes in data. And there are other areas that, because the challenge in running a game in the cloud is the statefulness. The experience between a very immersive game like ours, a game client like ours, and the game server is super stateful. This is a very fragile experience. If you did anything here, you lose that connection. And we are actually always discussing how we can make that less fragile, so people like us do not have so many precious parts of our environments that we cannot touch if the players are on. But we do both. We do canary, we do rolling, and we also initially do a blue-green. Hopefully in the future, as the project matures, we start dealing with the concept of continuous deployment. It's an environment that's always going and there's always pieces moving all the way around. But we still have to do a lot more things before we get there.

To get a build, there's only two. It'd be the developer making the commit and then the developer kicking the build. Depending on what environment you're wanting to deploy to, it could be potentially one more human interaction, which uses the same interface we kick our builds through to deploy to our internal staging environments. I think Ahmed could address outside of that for me.

If it's a larger environment, it goes through people deciding that this is the build we want to go with, and then from then the humans just make the decisions, what we were talking about earlier. Everything is driven by version control, everything is driven by Git, but there is a human saying yes. And we did not abstract that into a GUI yet, we didn't see the need for that. We call ourselves DevOps. I know this is some kind of heated topic for a lot of people. DevOps is a philosophy, not a title, but that's what the market goes for. But we try to really hold the values of DevOps close to us. We share the ownership of that product. This gaming experience is ours. I can spend more than 50% of my time investigating an issue about how a player isn't able to connect to something, because I have the access to all these tools and I can try to go and investigate it. To go back, when it comes to deployment to a larger environment, that's usually where human decisions are being implemented, but all the actual work of provisioning, configuring, securing is automated. It would take days, it would take weeks to deploy the whole environment manually with how it spreads around the world.

I'm thinking 3.0 for me, because the patch before that was so far, right? And to echo what Ahmed was talking about in terms of DevOps philosophy, if you don't deploy for a long time, then you forget how to deploy, right? And so 3.0 was the longest running in-development patch of Star Citizen, I think, that we've had, right? And launching that touched on everything. Everything was new. It was the first for everything. So it's like a brand new game. And so for me, that was by far the most difficult to get across the finish line. And that's when going to quarterly made it a whole lot better, because now we can deploy more often, which is how you get this rhythm going, right? And keeping it going. It also means, to be fair, the more you deploy, the more you break stuff as well, right? So there is a risk there. Don't touch it. It's working. Don't touch it. It's good. But yeah, 3.0 for me by far. I don't know for you guys. How about you, Miles?

My problems are very different usually. I'm more heavily involved at the beginning of the cycle, less at the end. 3.9 was actually pretty challenging to transition to, as far as getting the new stream set up and making sure all the changes were integrated to where they needed to be. We definitely had a good time with that one.

There's a lot of workflow changes in 3.9 for sure.

For me, it's a difficult take between 3.0 and 3.9. 3.0, definitely. 3.0, we rolled out a new hypervisor, a new generation of virtual machines, and a lot of new things, and it was in a very critical time. I think we have an episode of one of our shows, that I'm not very good at keeping up with, where we talk about what happened in 3.0. In 3.0, there was a chat between CR and Tony and us. I still remember a very, very dear message to me, when Tony said, I got to go put some gifts under the tree and come back real quick. That was 3.0. And it was a lot of fun. 3.9 was a lot of fun too. It was like Fleet Week. Wow. We are spoiled by our player base. Our player base are actually developing this game with us. And I'm not just saying that, because you really see that. And 3.9 was a lot of people doing a lot of things. We were learning a lot. We cannot stop learning from what happened. One of the main ideas in DevOps is to measure everything. Testing a game is challenging. It's relatively doable to test RPCs or API calls, but the immersion, how the game runs and everything, that is difficult. In 3.9, we had that. They're testing everything everywhere. They're doing everything. And we were receiving all this data and digesting and reacting. I do not have an accurate number, but the number of hotfixes that went in that week and a half was really, really high.

This actually went pretty well. It was before 3.0, though. We got to run the Delta patcher on PTU for like a month and a half. I just remember my biggest memory about launching the Delta patch system was the moment we had the ability to push a build a day to players, we did that for 48 days straight.

I do have very vivid memories of converting the build system to support that. That was definitely a challenging time as well.

I have a sad memory from 2015 of coming to the office at 1 a.m. so we can prepare the old-school patch so we release it at 6 a.m. And that was voluntarily. We would go and work on it, because it used to take hours to prepare these patches and make sure you have all the diffs between all the old versions. You actually bring good memories about the Delta patcher. And right now, sometimes we look at the CDN graphs and we see these petabytes coming out. Can you imagine if the Delta patcher was not here, how much data we'd be pushing? It's one of these things that we take for granted now, but it was an amazing big change for us.

Critical technology for sure.

Yes, so as we take some of our legacy systems from imperative deployment, where we are describing to the automation system how to do things, into more declarative systems. We don't want to throw in a lot of jargon, but people who work in DevOps technology probably know what we're talking about here. That is very exciting for us. We're also looking at how we can leverage some of the bare metal resources that a lot of cloud providers allow now. We've always been trying to squeeze as much power as we can get, and we want to see what will happen when we aren't going to have the hypervisor anymore, maybe we'll get some more power out of it. And again, we are not, till we measure... So we measure things and we see benchmarks. But yeah, I'm excited for these two things. One of the most important things I'm excited for, we have two new engineers that will join the team on Monday. So that is a big thing. Hopefully more hands on the keyboard, more minds, more people designing all these challenging solutions with us. So yeah, a lot of good stuff happening.