Miles · Director of Build Operations

So, it obviously starts with the developer committing a change. Once that goes through that process, we have tools which allow the developers to kick builds from our release streams. Once the build is finished, we go through a manifest process and we create what we call the build manifest. And basically it's just the list of files that dictates what the launcher should patch down. And we upload that and kind of hand off to Ahmed from there.

Ahmed Shaker · Publishing Technology

Yeah, that's where the custody kind of switches from a team to another. But also we do not really have silos. All these teams actually worked together for so many years. So it can be that we can actually be following up on the build and the build system and trying to anticipate what could be wrong. And we could be also looking at deployment and see what could have gone wrong. So from that moment that we have a build, we have a manifest, it depends on the size of the deployment. Every deployment that is going to go to a single region, a small number of audience, a QA or a staging deployment, this is fully automated. You go through a web interface, you click Deploy, everything gets created from scratch, share nothing. VMs get created, a Kubernetes deployment for a new generation of services, a namespace gets created, a Helm deployment chart gets initiated. That chart is a global chart, has other charts underneath it. A lot of moving pieces, so many moving pieces get deployed so you can have that fleet of services and the simulators that indicate the game servers deployed in the cloud, accessible whether it's an internal group of people or external. If it's external, so far due to how rapidly things are changing, we have to have a captain, to have someone who's really running the deployment. Every deployment in that case is a Git branch. You go take our automation repo and you create a branch and that branch will have some of the facts that are specific to this deployment. These facts are always changing. We're allowing very last minute changes because we are a development company. We are really developing as we go. Now we're trying to take away any ability to change things very fast. So that deployment gets created. It's a blue-green deployment, and all the resources get allocated around the world, and it's hidden. At this time, our internal QA starts hitting it. Every deployment has a goal, too. Every deployment has a very specific goal, whether we're delivering a feature, we're testing a feature, we're collecting feedback, and QA hits that part hard, and there is another checklist with other departments. Everyone has to know this is a new game experience where we are exporting or publishing a full game experience. It's not a bunch of virtual machines just being deployed in the cloud, and everyone has to get an agreement, and all the work gets done, and then the players see the light come on in the launcher. And I think that's where you get to see more. I mean, platform is involved in all of that. All these systems are super, super involved. But platform, you get to see more like, okay, we're ready to publish right now. And I think Benoit can explain that better than me.

Benoit Beausejour · Chief Technology Officer

Yeah, so that's where usually my team comes in, where we've built tools to allow the deployment to go out. So that includes securing the build from who needs to have access to it. As Ahmed was pointing out, there's a lot of builds that players don't see, that are used for internal testing, that are used for QA-ing a specific feature or doing ad hoc testing on a specific refactor or something like that. So that is internal, but the moment it needs to go external, so either it goes on PTU or on production, that's where the launcher release tools come in. And so this allows us to take the build manifest that Miles was speaking about and basically make it available to players to patch. And so that's where the Delta patching technology comes in that's able to rapidly look at the build that you have and the build that's now available and compute the difference and then just give you the objects that you need to run the latest version of the game. Obviously, for that, there's a lot of user-facing services. And every time there's a user-facing service, that means a lot of responsibility related to security, scalability, et cetera. So that's mostly the platform teams that take care of the maintenance of those systems. But the operation of it is fully in the teams. So that's the great part. So that's pretty much the build process in a nutshell.