Ray Warner

So I am Ray Warner. I am currently the pillar manager for QA Squadron 42 testing. And that is all I'm going to say about that.

Well, what's a pillar manager?

So I am primarily responsible for all of the project management of Squadron 42 as far as QA is concerned: setting up how we're testing it, covering all of the systems that need to be covered by development as they iteratively develop the game. And pretty much trying to mirror what Phil's doing with the PU for Squadron in the run-up to the inevitable release.

And Phil?

I'm Phil Webster. I'm the PU pillar manager for QA. I look after the Persistent Universe in Star Citizen and oversee all of that and testing for all our live releases. Don't ask me about 3.18 coming out.

And Rodolfo, who are you and what do you do?

I'm the global director of QA, and I'm running the different departments that we have in QA for the different games, and we also do some other things like automation, test automation in the games.

The global director of QA.

Yes, we have QA teams all around the world in different studios. They happen to be the managers that we have here, but the way we're structured, we also manage the people who happen to be in other studios. So for example, if someone is testing something related to Star Citizen in Germany or in the US, it falls to the person who takes care of it, even though we do have people and leads in each studio.

That's a super important thing. And it's something that we really missed in lockdown. Obviously, as everybody moved into that remote situation, we didn't have the ability to just tap a developer on the shoulder and be like, this is a bug. But now that we're back in the office and we're growing the team again, and we have so many embedded positions that are working alongside that, it's super important, not only for those 'look at this bug, this is so broken, can you fix it?' moments, and that on-the-fly 'yes, done, fixed, submitted', getting stuff out faster. But also the relationship and the processes that we carry in, QA have so much more visibility across development. So when people are making these big features or creating new planets, adding new rivers, putting in a volcano, for example, they have people next to them who can say, there are limitations to the way that you can do this. This is what's happening when you spit it out. One of the things that we see with planets quite a lot when they do reworks on planets is the terrain will shift in height and none of the bases will. So you have to have people who are there to go to those bases and know this kind of thing from the history of these kinds of updates, to help development get past those, especially with the turnover that we see, the growth and the turnover that we see in CIG. It's so hard sometimes to maintain developer loyalty to a very specific feature. It just rolls through so many people that we need this kind of point of information. So it's really, really helpful.

It's super easy to get frustrated about patches not coming out for this game. But one of the reasons that happens is so people don't have those situations where they forget how pretty the game is. They get to experience it at the best possible state that we can get it to at the time. We don't get to experience that. The vast majority of our work is repros or hitting crashes and getting the absolute worst of the broken parts out of the game prior to putting it into the hands of backers for the PTU, the Evocati releases, and figuring out the larger-scale problems with server longevity or the amount of players that we have. We want to get the ability to do that before we give it to you, and that's what we face most of the time. So there are elements of playing the game, and I don't want to...

It's a job. Testing isn't just playing, and there can be laborious and monotonous parts to it, but that's not to say we can't have fun and enjoy the game as well, because, like I say, I was a backer and I enjoy playing the game a lot. I'm a role player and I like to role play in Star Citizen. If you ever meet a guy called Stranger in there and he asks you to do something, he might have a decent mission for you to do. The sandbox nature of the game, you can make so much fun with what's there. It's crazy how much you can do, and you can have so much fun, and we do at work as well. But it's a hard job and we have a very important part to play, and it's recognized here at CIG, which is brilliant and fantastic for us.

We have a name for people like Phil and I: QA lifers. You get people who enter QA through several different means. Some people pursue the games industry, some people want to play games. This idea that it's just playing games, this is what pulls a lot of people into QA. But there's a specific skill set that comes really natural to a lot of QA testers, and it's the idea of problem solving and being interested in those problems and finding those solutions. And it's a real passion for that that drives people to continue a career in QA rather than chasing development. One of my favorite aspects, don't get me wrong, is seeing the development of people's careers as they come into QA, and whether they go up in QA or whether they go up in development or production, it's all amazing to watch people grow and gain knowledge of the product. But you get some people who are just wired to find the problem and make that problem worse and then tell people why it's so broken and then get that fixed. It was a particular passion that I found when I moved into QA, and then obviously as I got better and honed that skill in the 10 years that I was at Sony prior to coming to CIG, it became less about how I can branch out and become part of the creative force that make games, do the art or do the programming, and more about becoming someone who gets to see everything. Because that's what you get to see in QA. You are one of the few departments who see literally everything that gets put into a game. And it's such a rare thing to see in the games industry that I just fell in love with it, and ever since, it's amazing. I love it.

The career pathing we have from QA touches everywhere. It can go into any aspect of game development or game production, or staying in QA and leadership and management and looking after people and project management. You've got a finger in every pie, and through that you develop a lot of relationships with all the departments, and you build those relationships, so you know all the different departments there are and you get really good relationships with them, so then you are able to help them better. Because QA is a service, right? It's a service for the entire company. And we have to provide the best service that we can. And we do that through building great relationships. And the best way to do that is have people come from us into their ranks, and they understand because they've been where we are. They know what information is great for us, what helps, what doesn't help. Riffing on that and going forward with that and pushing people forwards and keeping hold of people as well.

It's a really interesting aspect of it, having people move from QA to a production or a development process, because it can change the way that a department or a particular focus works, because they go in with so much understanding of what we're going to do to try and ruin their day at the end of it. We get these huge benefits of people saying, well, I understand how testing works now, I understand the process QA go through, I understand what they're trying to fix, I understand my feature, and this is a big one, my feature has an interaction point and affects other people's work. And that knowledge is something that's very, very rare to get when you come out of university and go into development and you're so siloed in the process that you're doing. So I love pushing QA into production. I love pushing QA into dev, but I do wish sometimes that I could keep them for myself.

We do have guys from QA who have gone to all the areas. We have people who have gone to sound, people who have gone to art. Of course, a lot went to production. It's very, very common that they go to design, to game design, and the skills that they get from QA are very useful for that. But on top of that, we have people who actually code, and we have people who are doing automation, which includes coding and their relation with the code of the game. And these people develop skills that are super useful for development, but also for design. So it's more technical than what people think. It's a whole craft. And I always use an analogy, because this happens in the software industry a lot, that for some reason the QA work is not appreciated enough. The same way there's people who have a mindset that is very good for constructing, a constructive mindset, for QA you need to have a deconstructing mindset, and that is a skill in itself. It's very different, but it's very, very useful. And in other disciplines, it's not like this. For example, in physics, you have the Nobel Prize, right? And you give the Nobel Prize to people who are theoretical physicists, people who make theories, like Einstein. But you also give the Nobel Prize to people who are doing experimental physics, which is basically trying to prove if the theory this guy came up with is valid or not. And you need to make an experimental design of how to test that, how to test the theory of cosmology or quantum physics or whatever. That's exactly what we do. It's just we do it with software. So we're basically empiricists, experimental scientists working with software instead of working with physics or chemistry. That's what we're doing. And there's a lot of techniques behind it, a lot of knowledge and a process. It's not as simple as just playing the game. There's a way you have to approach it to do it in an efficient way, because you cannot find all the bugs in an infinite universe like we have just by trying to go through it. It will take you forever, so you need to do it in a certain way.

Absolutely. That's what I mean when I say there's got to be a very specific mindset to get into QA and remain good at the job. It's about problem solving. And when I say problem solving, it's not about being presented the problem and finding the solution, it's about finding the problem. And that's very much what you're talking about. It's about seeing a piece of software or seeing a feature or seeing an element of gameplay and thinking to yourself, and our backers are great examples of this, how are people going to come in and ruin this? Because they will. The first thing they want to do is find easier ways to do things or better ways to get things, ways to exploit the code, ways to exploit the gameplay. We have to approach our testing methodologies with this in mind from the very, very get-go.

Right from the start, like in the kickoff meetings, the amount of times when designers come in and they do the pitch and someone from QA will go, so that could be weaponized by doing A, B and C, right? And their faces drop and they're like, oh, yeah. But that saves time, and it gets that fixed before it even hits the code base. So it's very important.

We actually had one of those today, right? It's very important because sometimes you can have a design that will not work out in the whole universe, and people don't figure it out until it's coded and gets into QA hands. So we actually test it and it's too late, because you have to drop the code and start again. So it's so much better when you can have this interaction in the early stages of the development lifecycle.

I think one of the easiest draws is some of the dynamic mission stuff that we do. There's a lot of elements to our dynamic missions because they attract so many players, and the unique elements of it, like the Idris attacking or the flight shows. And probably one of my favorites, and I apologize, Phil, because I'm sure you were probably going to say this also, is when we brought the Idris on and they were like, here's the Idris, now everyone can look at it, it'll be amazing. And the first thing QA say is, someone's going to get in it and take it over and then fly around in the Idris. And they're like, no, that's not possible. We've de-physicalized all of the floors, it'll be fine, don't worry about it. And within, what, 10 minutes of the release, someone had clipped through the ceiling, landed in the seat, got in the seat and started flying around in the Idris. And it's like, well, we told you. But that always comes up against that kind of, we've done it, this is it, we're going out live. PTU released, damn it, we're going to have to go back to square one and figure out how to make this work. And it happens constantly. But there's a great satisfaction in 'I told you so'. How about you, Phil?

Well, Ray totally stole the one that I was going to use. So I'll just say, about Ray's particular project in Squadron 42: redacted. So unfortunately, I can't tell you mine.

What about you, Rodolfo? You don't make it to global head of QA without having to crush some feelings.

I have some stories from long ago from other projects. I don't know if I can share that, but we do have them. I do remember, for example, my very first job in QA, when I was a tester many years ago, I was on a mobile game for iOS, and they were pretty sure that it was working perfectly. It was a physics simulator, so in some sense similar to this. You know The Incredible Machine? Do you remember the old-school games? Well, we made a remake back then at Disney for mobile. And I tried to stress it a little bit. So when you start stressing it, you saw that all the cables and things start wobbling around, and you can just do whatever you want. And it's the kind of thing, oh, but no one is going to do that. Well, we did. Maybe the players will do as well. You always have that kind of discussion, right? They only expect people to use it as the feature is intended. And actually players, if they are allowed to do it, they will do it eventually.

At CIG specifically, one of the big problems that we have is we have these universes that have hundreds of players, potentially in the future hundreds of players in a server at any one time. There are things that happen in the back end when you put that many people doing random things. As we say, people get destructive when they play video games, and when you put a hundred of them in and let them do what they want, it's very, very hard to replicate that in a controlled environment. And that is fundamentally the baseline for QA. We need to be in a controlled environment so we can find those steps, find the cases of how it happens, when it happens, what you need to do to make it happen. And when you put a PTU build out and we get 50 crashes, or we get the elevators killing people, everybody knows this, right? We put it into a game. We get 10 people into the game. We say, just ride elevators for a day and tell us what happens. And it's fine. It runs great. And you're like, why is it running great? And it's really, really hard to replicate. And you can do it in the PTU, but you can't do it in a dev build. So you can't get the information you require. And it's super frustrating. It really is.

And we have to deal with multiple builds a day, whereas the servers will be up for days at a time with this amount of people on them, and so it's the soak time and stuff as well that we just can't replicate, because we have to keep on iterating and getting the fixes in, making sure they do work at a base level, so that we can get it out to the backers and they can start playing it and enjoying it as soon as possible.

And then you get Rob Johnson, who comes in and goes, oh yeah, I'll fix that. And it's done. And it's like, I've been locked into this for a week.