How do we test and benchmark our systems for stability?
TL;DRStability is tested with Locust-based load tools, including an in-house tool called Pestilence that simulates thousands of players doing specific actions against cloud services, plus a cloud test launcher running real clients to test game server performance.
That's interesting. So there's been a lot of work there in the past to try to get that up to speed. Nowadays, there are multiple ways where stability is being tested. So on the platform side, for example, we have a set of tools built on Locust, which is a load testing framework written in Python, which is super nice. It allows us to basically simulate thousands and thousands of users so we can write scenarios about what they do. For example, melt a ship. What if X amount of players melt a ship? How does the system react? We can test that. We've used the same principle in the same framework to build a game-facing Locust system that we call Pestilence, which is basically, yes, I know, the naming is nice, but basically what Pestilence is able to do is mimic what you guys do in game. So it's able to log into the game, connect to the servers, and then do specific actions that we want to test. So let's say we have a service that has an issue and we're trying to see what's causing issues. We can actually write a scenario that does that, and then we can spawn thousands and thousands of locusts doing that in game to simulate what happens when we actually launch a build, because that's usually what happens: there's a big storm of people doing the same action. So we can test stability this way. Now it's getting more and more integrated into the process, where we write scenarios for different services as we go. Recently we've added the ability to even provision account data from it, and so we're able to pre-warm some of the databases to prevent some services from getting hit too hard when the patch is launched, things like that. So that's the major thing that we do for service and platform specifically for load testing. On the game side...
These are headless clients, obviously.
Exactly. Those are even more than headless clients. They're not even running the game itself. They're literally just running the cloud side of the game. But we do have headless clients as well. That's where I was going. There are automated systems, which is called a cloud test launcher, which can actually do that running the game client as well. But the testing there is more focused around testing the game server performance than the MMO side of the game.