What configuration management tools does the team use to administer and configure servers in the cloud? Also, what kind of log analysis and management tools are used?asked by Disastrophe

Yeah, I'm not a huge fan of focusing on the brand, but yeah, we currently use Chef and we use a bunch of our homegrown scripts that do the provisioning. We can bring up a graph that will explain a lot of my work and a lot of what we do. If we start all the way on the right, you can see the build system in Austin. That's where everything is being built. Our current build system is based on a build system called Buildbot. It's highly modified and customized to our own needs; the way it schedules and manages the fanning in, fanning out, everything in the build system is custom made for us. So the outcome from the build system will be a build, which is all the binaries needed to run the client, run the server, run the backend services, and a bunch of other files. That gets uploaded to an object storage in the cloud, as you can see, still on the right. And from there, now we have the build. We want to deliver it as fast as possible. Our current cloud provider gives us a good advantage here: we can create an ephemeral box at this moment, and we pull the build down, and we do whatever we want to do in the OS at this moment, like install packages, make sure that we change the ulimits, make sure that everything is optimized the way we want it, and then we create snapshots from it. This whole process usually takes 20 minutes, 25 minutes. It highly depends on the WAN link. I mean, who's fighting with me over uploading something? It could be me too, uploading somewhere else. So yeah, this is mainly an internet speed kind of limit. And then we get the snapshot. The snapshot can create disks, and these disks can be shared across the whole cluster. So if we scroll back up again, now you understand that the build exists in the snapshot. You as a player sitting here, that's your game client. You start your launcher/patcher, and your launcher/patcher would go grab the build for you from the CDN. The CDN, or content delivery network, is a bunch of endpoints globally distributed, trying to deliver you as fast as possible the files that you need, and you pull them down to your game client. Now you have the build. The build needs to connect. You would have a connection between you and the hub load balancer that takes you to the hub servers. So you have a bunch of different hub servers that handle your connection. You go there over TCP; the hub servers go to the platform, make sure that you are the guy who you claim you are, and you have a connection with the platform. From here, the hub service will take your traffic to most of the services that Tom and Jason did here in the core server. They live in a single box, and we can say why they live in a single box for now. And one of them is the persistence services. We have two main services, persistence cache and the persistence service itself. Those connect to our MySQL database clusters; it has masters and slaves and backups and all this cool stuff. Here you see all the dedicated game servers. All these boxes that are drawn represent an environment. Any cloud environment right now in Cloud Imperium Games has to be three different VMs with three different types of roles: one of them for the hub, one of them for all the core services, and one of them for the game server. This is not anywhere near final. That's what we need now to get everything running and see how we want to scale and change stuff. The difference between QA, PTU and Live, to be quick: QA is one, one, one. PTU is multiple of these, one of these, multiple of these. Live is much larger. The question that he asked also relates to this bunch of servers. We have our infrastructure servers: we have log aggregation systems, we have monitoring systems, tools for dev support, and tools for analytics. We use Chef here, we use a bunch of homegrown scripts here, and over here we have Splunk, we have Graphite, Grafana, StatsD, we have Stackdriver, we have a bunch of core dump analysis, we have GDB server for developers to look at, we have some stuff related to Google Breakpad. I know that we have a lot of backers who come from a technical background, and I see the posts, so I'm trying to keep a balance where I use a lot of technical jargon and then go back to something high level. So yeah, this is not anywhere near final. Actually, right now we are working on a new deployment pipeline that would be the event-driven data center I was talking about, where scaling in and out would be easy and usable under heavy load.