Every patch feels like feature roulette. For example, sometimes auto-loading works in a patch and the next it doesn't. Every time a new patch drops, we're wondering if our favorite gameplay loops will continue to work. So what is being done to prevent feature and fix regressions?asked by Trulin Vaughan

For me, there's three main ways that we're tackling this. Number one, root cause analysis. What is that? So when you're faced with a bug, you can diagnose it sort of surface level, find out what's going wrong and fix it. But unless you follow that through and you go down the rabbit hole and find out what the ultimate root cause of what is essentially a symptom is, there's a very good chance that that is going to come back. So some good examples would be freight elevators, hangars, ASOP, ATC. Freight elevators especially, we've had a decent amount of time of people on our teams really digging into this at a core level, and hopefully people in the community are starting to see the improvements from that.

I mean, that's the thing: root cause analysis changes how we assign our resources when they're debugging things, so that they stay longer on the issue. They go a lot deeper and they document the process before taking action. Then we can identify in that process regressions that have caused the problems to begin with. And then we can also build a comprehensive fix plan, which then feeds into the second point.

So the second point then was expanding our test plan. And that's the thing that you've worked quite a lot on.

Yeah, so basically this is the idea of going from a given change to a larger test plan for regressions.

And what is a test plan?

Well, it's basically a list of steps, a matrix of steps or tests that QA needs to perform, which are much larger now with these expanded test plans.

To validate the fix or to validate the change.

Validate the fix in multiple configurations. We typically used to do these tests, but deferred. Now we're trying to do them pre-submit. And so what that means is that we're doing multiplayer testing with more players pre-submit. We're doing server recovery testing pre-submit. We're covering the happy and the destructive path for each feature pre-submit. We used to do that post-submit.

And that leads into the third point that we mentioned, which is just massively scaling up our suite of tools and frameworks and functionalities for validation. So whether that's in content creation tools or scaling up our use of our automated testing framework.

It's about expanding this testing so that we can simulate live environments basically as much as we can. And so validating the content allows us to make sure that our content and code that are linked actually work properly before submit. But then after that, it's about validating that when you put a lot of players on it, it actually functions the way you would expect it to function.

And that's ultimately what this question is kind of speaking to. It's like feature roulette. You don't want to be rolling a dice.

I agree, 100%.

And I think all of that combined can really give us the confidence that this approach is going to work, and the stability and the robustness of our game, its features, its systems, its content, the waterline of that is just going to continue to rise.