Samantha Haley

So when it comes to Star Citizen, it is absolutely massive. And when you're looking for people to hire, you obviously want to try and take on people who have a little bit of QA knowledge to begin with. But recently we have been reaching out beyond people who have maybe that games experience already, into people who have a passion for the project. For example, quite a few of our new starters did actually start off as players of Star Citizen. For example, I know George plays Star Citizen as well, George Stabler that is. So he started off from that. And it's just really kind of inspiring to take people through that process of knowing nothing about games to kind of warming them up, I guess, and getting them out of the myths and the kind of preconceptions they might have around QA, to the point where they're fully ready to get into the job and test our massive universe, basically. So yeah, when it comes to those kinds of, I guess, myths and stuff, a lot of the people who, like I say, don't have that games background may have some conceptions about QA testers and what they do and that kind of thing. So I guess we'll touch on some of those points.

I would love to get paid to just play the game. That would be absolutely wonderful. But sadly, it's not maybe quite as appealing as that. There's the video that goes around in the games circles where there's the girl who comes in and says, "Hey guys, are you done testing that game? I need another one designed." And they go, "Yeah, just tighten up the graphics on level three. Man, I can't believe they pay us to play games." I guess that is kind of the opinion that a lot of people get because of that. But yes, when it comes to actual QA, I think it's more to do with being kind of like an information broker for a developer, and just making sure we know exactly everything about the game, keeping a pulse on it as it goes from the beginning. For example, Jack starts off with the most broken, kind of new, interesting stuff on the core team. Those guys deal with all of that kind of thing. And then as it goes on to the more finalized products, George would see that through to the released product, basically. So as well as getting to play the game, which we do repetitively, but we do get to play the game, we are making sure we're providing good detail to the developers on how to reproduce bugs, debug information for anything that comes up, prioritizing certain checks. For example, if new features are coming in, we need to make sure that we are rinsing those and maybe deprioritizing some of the older stuff. So yeah, that's a little bit more about the QA role.

And that touches on a bit of the things that people are saying there in terms of the 30Ks and stuff. We probably have felt your pain thoroughly every day while we're trying to test stuff as well. It is definitely a struggle, but like I mentioned for the last point, we are information brokers. We deal the information, but we don't get to decide when a developer has time to get in and actually fix the problem. So unless it's something that is really severe and a lot of people are seeing quite consistently, it might be something that people have to deprioritize. QA do try and raise flags about severe issues or things that we find particularly frustrating, gameplay issues or particularly ugly art issues, all that kind of thing. But it is, at the end of the day, down to the developers to find time to fix that kind of stuff. They also have to find time to create new content and jam all of that in there and somehow get that released out to you guys. Honestly, they do incredible work to manage to get that stuff done. So yes, unfortunately, some of the more potent smaller bugs may not get fixed for quite a long time, but we commiserate. And do please keep sending them in to us. We love to try and deal with issues that the community call out particularly. I mean, George will have dealt with some aspects of that with your old job as well, didn't you, with the player relations stuff?

Yeah, so one of my bits in player relations, I believe it's become more of an actual team now, one of my jobs was with Chris Danks back in the day: whether it was a big patch, collate all the major information from the community, like go into chat, look at the Issue Council, everything, and find out what was annoying people most and send the top 10 over to QA to have a look through and get bugged up or linked to bugs which were already in, and make sure the developers were actually aware of the community's pet peeves, the worst ones affecting the people on the PTU.

A lot of the assumptions that we see publicly are that if a bug makes it to live, and this is in any game, we're talking in general here, anything from Cyberpunk at full release to us who are in alpha, there's the assumption that if a bug makes it to live, it must not have been seen, or it must not have been noticed or found. When the reality is often that these things have to be prioritized. Even with 700 staff working in five studios around the world, there are only so many things you can be working on at any given time. And it is often a triage of figuring out what the most aggressive bugs, what the most invasive bugs are, and then the ones that you can have solutions for. Not everything that you encounter has a solution that becomes readily apparent. And some things can't be fixed until other systems come online or other tools come online. You know it's important, you know it's super invasive and drives you and everybody else crazy, but the tool that you need in order to be able to address it won't be ready in time. But one of the things I want to make sure that we demystify here is that they do not go out because they are not discovered. I'm sure that happens sometimes, occasionally, but it is just usually a matter of triage.

My heart goes out to you guys because we feel it too. But yeah, like Jared says, there are occasions where there are things that slip through the net. We have about 100 QA staff across all of our studios, and it's just not enough for a game of this scale when you've got thousands of people out there playing it. So please do keep telling us about your stuff. You never know, it might be something that we've missed along the way, and we would love to hear about your problems.

Or enough reports can help change the prioritization.

Exactly that. And the new Issue Council is absolutely fantastic. So yeah, definitely make the most of that and use it, because those Player Experience guys are doing wonderful work with it.

Not at all. Definitely not. When it comes to devs and QA, obviously, we are there to somewhat ruin the day. And we do come in, and sometimes there'll just be something that they have not expected that crops up. But the vast majority of the time, I think we have a pretty good connection with the devs. None of them have called me names to my face. But yeah, we work very closely pretty much from the beginning of when a feature is being designed. So for example, at the moment, in the run-up to 3.15, we've got the healing and inventory changes that are coming in. We've been working really closely with the Actor Feature team on that. Guys like Chris Parry doing absolutely wonderful work, and they guide us and we help guide them. So we're all striving towards that same goal of having a good quality product. Developers also test their own work. So it's not just us that are coming around and saying, "Hey, it's broken." If they go through and launch it up and something's broken, they also don't want that. So yes, as much as it is kind of like a Tom and Jerry cat-and-mouse comedic relationship in theory, in reality we do have quite a good bond with those guys, which is really fun. And honestly, QA gets to touch on so many different areas of gameplay that we do have a close connection with all of the devs. Say, for example, artists may never get the chance to work with a particular area just because of the nature of their job. So it's really great to have our tendrils in there. And George being directly in the network team, obviously, if you didn't have a good relationship with those guys, that wouldn't work so well. I'm hoping stuff's good.

Yeah, stuff's going very well.

Also, because we touch on everything, we potentially know a bit more about other features than the devs, because they'll be very focused on what they're doing, their one little thing. For example, back when I was doing UI stuff, when I was here before, there was an occasion where literally they did something and it impacted on another system they hadn't even considered. The second we saw it, we were aware and we obviously fed that back. So having that relationship with the teams, you've got the ability to give them knowledge they sometimes don't necessarily have access to, because they're so focused on their one little area.

So like you say, we do have those main QA teams, like Jack's team, like George's team, that are the main gameplay-focused QA guys, and they run through together as a team of QA. Granted, they still do contact developers and work with them that way, but they're more of a QA team. So we've got those main QA teams in the UK and also in Austin, Texas. We have a few testers who work as part of main teams like that, as well as the core and the gameplay teams. We also have a vehicles and FPS team, which I led for a short amount of time. It was really fun to delve into the ships and the shooters kind of stuff, basically. And then we also have the tech team who deal with everything editor-based. So all of the tools that our developers use that also need to work. So not just the game-playing aspect of things, also the game-making aspect of testing too. But yes, like George here, we have multiple embedded testers who basically are part of their teams directly. So George is part of the network team. We have people in the UI team, audio team. And basically they get to be the person who is the point of contact who knows everything about that particular area. When it comes to QA, we are generalists. We try and do our best to provide info across the entire game. But obviously we can't remember everything. There's a lot of stuff in terms of logging, debug. I can't even get started on crashes. There's so much with crashes. We have a crash guide that is literally an hour read on its own, just all different sections. So yeah, it can get quite complex. And having those embedded guys who are literally there to know the exact ins and outs of everything to do with that particular team is so, so useful, for general QA and for the developers themselves. It is a really, really good kind of symbiotic relationship, I guess, that we have going on. George, do you want to talk a little bit about your transition over to embedded?

Yeah, so it was kind of like a weird culture shock in a way. So before joining games, I worked in retail for almost 10 years, did my degree and then started in QA. Moved through a few different departments: before, I was on live release, so what gameplay is now, so we'd test the features before they go live. Then I moved on to the core team, which Jack is on now. Then I moved over to tech, then I moved over to the resource team with Sam for a good couple of months, and then I ended up moving on to the embedded team. So it was quite a weird culture shock, because the first thing that I thought of going into an embedded team was, I need to make sure everything's good. So if I see a texture bug, I need to bug it up, which, that is the premise, but usually when you're on an embedded team, you just focus on the stuff that your team does, because that's why you're there. You're running around trying to bug all of these things up, and it's like, no, just take a minute, calm down, bug your network stuff up, and then relay that to the devs. And that's kind of the way the team works. I think the biggest thing is that when you're a generalist QA, you're not really talking to devs on a day-to-day basis, really, because you're more relating to your seniors or your leads. Whereas going on to an embedded team, you're talking to your lead, who's obviously an engineer or developer, plus your seniors and the junior programmers and all of that on a day-to-day basis. And when Sam was saying before, it's like a catch-22. So when people say, oh, devs must hate QA because they're always finding these bugs and can't they just stop so we can do other things. But at the end of the day, it goes hand in hand. So I'll go to my lead or my senior and be like, oh, I'm not too sure how to use this specific tool or command or something. And then they'll come back and be like, oh, it does this, it means X, Y, Z. And kind of the same vice versa as well. So if devs are in-game and they're not too sure of a specific thing, because a lot of the things in-game do change quite rapidly as well, because the game is huge, then it comes to us: I'm not sure how I do X, Y, Z. So then we'd go and help them do what they need to do in order to check their fixes have taken, basically.

I think we've somewhat touched on the aspects of it from the day-to-day stuff. But yeah, like George says, it's the different dynamics of we have QA teams and then we have our embedded guys. And then we have the Squadron team, which is kind of a hybrid between embedded and also being its own small team of QA, and those guys did great work as well.

Yeah, so obviously the most obvious one I can point to is the stuff where we get to be involved in the CitizenCon playthroughs. That kind of thing is super fun. I know George did some work on the testing for the CitizenCon stuff as well. But yeah, being able to perform that live and do that in front of a crowd is a little bit of a different skill than my day-to-day, just me and the two monitors testing. Very nerve-wracking, but super fun. So yeah, we do things like that. We do gameplay for marketing footage as well. So say, for example, if they've got a particular feature they want to highlight, sometimes we'll get QA to do particular maneuvers or capture particular footage in game. But yeah, live playthroughs and stuff, like I say, on the stage, or we sometimes do recorded playthroughs of things as well for CitizenCon. We play sometimes during review meetings. So say, for example, they're reviewing a feature, or the directors want to see how a particular aspect of gameplay is going in-client, how is it actually looking? We'll sometimes get involved with those. So that can be a bit nerve-wracking as well, but probably doesn't quite match up to sitting on stage with CR there and Todd just there.

It definitely doesn't make it harder, no. It maybe gives us a little bit more work to do, but there's always plenty of work to do with Star Citizen, so that's totally fine. No, we work really closely with the Player Experience guys to investigate those issues that come in from Issue Council. Like I said, those top 10 issues do come to us. We do look into them. We do make sure they're updated on the database. So you can rest assured we are definitely touching your bugs. We do see them and we do appreciate all of the info that you can give us about those particular issues, especially if it's something that's on and off. Sometimes having that stuff from the Issue Council is the key to us finding out exactly where to go and how to provide the right debug. If it's a particular thing that we didn't think of along the way, a step that we were missing, that's super useful to have. Outside of the Issue Council stuff, we do take things from the community, from the forums. If there are feedback things that we think are probably bugs, we do pick up those as well. Videos, for example, on YouTube, all of our content creators and streamers, we do get a lot of that content sent through to us saying, "Hey, have you seen this happen? Is there a bug for it? Help." And we'll usually trawl through our database for that. So yeah, it's really an interesting position here at CIG, especially in comparison to other QA roles I've done in the industry, where we are fully tight-knit with the community because we are releasing constantly. And it's an experience I've just never had before. When you're working on a project that is going to be released and shipped off and nobody else is looking at it, that's different; this gives you a whole other perspective, and you're able to do those big load tests of thousands of people, or 50 people to a server every single time, which we don't often get to do in QA. So that's really an experience I didn't expect to have when I came to this job, I guess, and has been really interesting. It is like having another crew of testers that could help us sweep everything that we need to do. How have you guys found interacting with the community?

It's really, really useful, because I know back in the day there was an issue we had at Levski where occasionally everything would go psychedelic colors. And we saw it three, four times in testing and we couldn't repro it. There was nothing we could do. It just happened every so often. And so we couldn't work out what was causing it at all. And obviously it then went to the PTU to be tested. And actually, because it gives you those extra people throwing at it, we could see what was causing it and pass that on. And it's so useful to have that information to be able to feed back. Certainly in situations like that, with things which are ultra-low reproduction, the chance of us reproducing them in standard testing is just luck, whether we get it and we can reproduce what caused it or even work it out at the time.

So for starters, things like video footage and screenshots. Obviously, if you can provide that kind of physical evidence, it makes it a lot easier for us to see things that perhaps you don't know the particular word for externally, so having those videos is really useful. Particularly mentioning the types of ships and the variant of ship that you are encountering an issue in can be really useful to us, actually. A lot of bugs end up being specific to certain manufacturers or certain types of ships because of the way that they're set up. So if it is a ship bug, maybe go into detail about the ship that you have, whether you've changed the loadout, or if you've changed any of your settings, so keybinds or graphics settings, that kind of thing. Really, honestly, the more info you can provide, the better, basically. Do you guys have any other suggestions for things that people could provide?

Not really. I don't think I've ever seen an Issue Council bug and gone, oh no, they've given me too much information, please add less to your bugs. The more you can give me, the happier I am, because it gives me a much better starting point to try and reproduce what you're seeing in game. If you've given me loads of information, I can just jump in game, get to it straight away. I can still do that if you've given me less information, but it might take me a bit longer. So yeah, the more information, the better, really.

And don't think, oh, that's been there forever, they'll know about it, because we might not. It might be something which has been missed and missed and missed. And if you raise it, we'll then go look and check. And we might even go, oh, that's something which has been there for ages, and then we'll go search and we won't find it. So even if you think we must know, let us know about it.

Or it can be something that was deprioritized, and every time it happens, that can help to up the prioritization of it. So just because you think, oh, they must already know about it, does not stop you. Submit that, submit that, submit that. It helps in a number of ways.

Yeah, absolutely. And like I say, we've been taking on a few people that are coming from the community who don't have any experience or education in games. It can help, obviously, if you have that kind of background. But even I, coming from a place where I had QA experience, learning Star Citizen was its own mountain to climb. So it is something that we thought about in the QA department, and we wanted to prioritize making sure that the people coming through knew the game, knew what they were doing and were able to get into work and test the project quickly, I guess. So yeah, when it comes to a project like ours, with the longevity of the project, it is super worth it to invest a good amount of time into fully training our people from scratch. And just from our guys, Jack, you had a bit of a university games and indie background before coming into QA, didn't you?

Yeah, so I did some game stuff at university, and I was in a much smaller company, four people, before I got to Star Citizen. And I think that does help, but I would say it's in no way required. I don't think during training there was any point where I was like, oh, I did it at university, so I'm so much better than everybody else who hasn't done it. Everybody was kind of just there like, oh wow, there's a lot of stuff to learn here.

Thrown in the deep end. And yeah, George H coming from retail, George S was literally in the Navy before you came to CIG. So you can come to this career from pretty much anywhere, as long as you've got the passion for it and you show us that you are willing to learn. I love to take new people who are just passionate about Star Citizen and want to test it and make sure it's good. So when it comes to the actuality of what we do with the training, we take all of our new starters here on an 18-day crash course through everything Star Citizen. For starters, that first week is just game familiarization. Obviously, we do need to be able to play the game. So for those of us who maybe haven't played Star Citizen before, but even those who have, it's good to get some time to sit down dedicatedly with each area and go through bit by bit. We start off with some game basics: being able to get your ship, fly around, go from place to place. Then we go through missions, so all of the contracts that we have in game through Contract Manager; mining, which now is quite an in-depth thing in terms of, we have the ship mining, ground vehicle mining, FPS mining, all that different stuff. So we have a full day dedicated to just getting those down. Then we touch on our other areas as well, like Arena Commander and Star Marine and Theaters of War. So, you know, that thing that we are working on. So yes, we do touch on the minigame areas as well, to make sure that everybody's up to date with all of those. If you were at the CitizenCon that I was on stage for, you might have got the chance to play Theaters of War there, actually. They had a really cool setup room for that. It's really good stuff, and I'm hoping that you guys can play it soon. But yes, after we've done all of the gameplay stuff, we move on to the technical aspects of things. So that is all of our test commands and what we call CVars, allowing us to get all of that debug information and go from place to place much faster. Say, for example, we have goto commands that allow us to zip from planet to planet in an instant, which saves us a lot of time while testing. As I'm sure you guys are aware, it takes a while to get from planet to planet naturally. How to run our internal builds: like Jack was talking about before, we do spin up our own separate servers and run what we call development builds, which allow us to get a lot more debug information, development information that you can't get on the live servers. Our internal editor as well; we do go over some basics for that, even though most of our testers don't usually dabble too much in editor. We do give them a little bit of basics around how to actually set that up. And then we start to move on to things like QA basics: writing bugs, what kind of evidence you need to provide for certain bugs and what kind of rules you need to follow while writing bugs, searching for bugs, testing existing bugs, that kind of thing. And then lastly, we do crashes, which does take a little while to go through. While you guys are well acquainted with the 30K, they do come in many different flavors internally, and because of that, we have to know how to deal with each one. Which can take a little while, but once you've wrapped your head around the basics... I would say that's probably one of the most technical and difficult things that we have to do in QA. So once you've got that down, you're pretty much on your way.

So since we have started working from home, literally all of these guys have come in during the working-from-home time. I literally have never seen any of these guys in the office, which is a crazy thought to think about now. It's strange. My brain kind of places people in the office as if they were there the whole time. I have no idea what height they are. I just kind of guess. Yeah, it's been a huge challenge to adapt everything to working from home. But honestly, through doing voice chats each day, just while we're working, we'll use voice chats with each other. It almost feels like being in the office these days. And it's really useful for me to be able to screen share while I'm walking through particular things. I do miss the aspect of being able to lean over and help guide someone directly. But we do our best, and the wonders of technology have allowed us to do a lot. How was it for you guys, starting a new job while working from home?

It was actually really good. It was really, really well done. It was a bit weird on the first day, but it was so smooth. So just basically straight into it, it didn't really feel any different from starting a job anywhere else. It sounds daft, but that's good.

It's been an interesting experience. It's been very different. There are times in training, and even after you finish your training, because 18 days sounds like a lifetime of training, but you've barely scratched the surface. There are times when you're new and you're unsure about things that it is a little bit anxious, a little bit stressful, not having a person in the room with you to, if it all goes horribly wrong, just jump in and fix it for you. You do have to be a little bit independent and work with people talking in your ear like, OK, I know it sounds terrible, but it's not actually that bad. We've seen this before. It's fine. What you need to do is this, this and this. So I think during the training period, that is a consideration, but you work through it, and I hope it's all worked out. No one's fired me yet, so I must be doing something okay.

He's doing pretty okay.

That's totally fine. I understand. Yeah, so actually, automating testing for a project like this is quite challenging, to say the least. As it stands, we are moving slowly towards being able to automate certain test cases. We do have a guy who is embedded automation, actually, and he is working slowly towards making it more available to all of us. But we do have a couple of things that we use internally already. George, on your time on the tech team, you might have touched on some of this stuff with TestRunner.

Yeah, that's correct. So TestRunner would grab a build and run through basic sanity checks. It doesn't go too in-depth, because obviously when you're making an internal development build and then running it through TestRunner, it can take a few hours to do that. So it goes through the basic checks. And when we try and grab a build, it'll give us all the information: this was fine, this was fine, this was fine, but this was broken. Sometimes we can get around it because it's not a core thing that would stop us from testing. But there have been cases where we tried to grab builds and it would just break continuously. That's where TestRunner picks things out, so we can identify that before we go into game, to say, we need to do this task, but this build is broken, what do we do? And then usually the devs will go and fix it ASAP and kick a new build, or we just revert back and then carry on the testing.

Yeah, exactly. It's mostly launch sanity checks and that kind of thing. Very basic pass/fails at the moment. But we are working our way towards it, and it is something that we are very interested in. So if you are an authority on automated testing, please do come in. We would love to hear your opinions.

I think Jack can answer this the best, to be honest. You deal with the most crashes on your team, I think.

The stream I work on is notorious for being potentially the least stable one in the game. Patience is obviously a big part of it, but I think the thing I've used the most since I moved on to the core team is my memory, remembering things. A lot of issues tend to maybe disappear and then come back. And after a while, you just remember the really bad ones. So something might appear that's been gone for three months, and you can just look at it and go, oh, I know exactly what that is, I've seen that before. And you'll go and find it, and you'll link it back up, and you'll escalate with the devs and go, this has started happening again. What did you do, please? Please help.

The other one I'd say is curiosity, quite a bit, because you find something which isn't quite right, and you pull the string, and then you find something else, and then you find something else, and then you find something else.

I think that is a lot of what a lot of people don't really understand about QA sometimes. There's one thing, and you think, oh, that's fine, it's an easy fix. But then that will branch off into 100,000 different things that need to be fixed: so does that, so does that, so does that, before this main thing can be fixed. It branches off so much. Game dev is so complex, it's crazy.

Oh, I would have to say the amount of things that we learn. We learn so much in QA, whether it's parts of dev and just the game, and just generally how games are put together. I love learning things. So that's what I love about QA. It's just being a sponge of knowledge.

For me, there's a lot of different things. The thing I think I enjoy the most is the people. It's such a good, vibrant group of people who, even when you're having the horrible crashes, you'll be able to have a joke and have a laugh. And if it wasn't for a good group of people, I don't think I could do it. But it's such a good group of people.

I should have gone before George, because I was going to say the same sort of thing. I think, especially on the game dev stream, which does tend to have some minor issues with stability, you do form a real sense of camaraderie with the people around you, because it's not just you that's having a horrible time, and you've been trying to get into game for four hours and you've not made it past the main menu yet. It's you and seven other people who are also having a horrible time and haven't made it past the main menu yet. We're kind of friends for life, forged in that fire together.

I guess as well as being able to do the same things as those guys, I love learning stuff. I love picking the brains of all the different developers, especially the big-brain people like the directors who just know so much about how to make everything cutting edge. I guess just generally being able to work to support games is something that I really enjoy. When I came out of university, I did an animation degree and I hated animation. I came out of it and I was like, never again in my life. But I knew I still wanted to support developers, and I knew I wanted to support games development and do that kind of stuff, because that is something I'm so passionate about. I didn't have a clue what I wanted to do, and I just fell into QA by pure chance, because it was something that was entry level and everybody could have a go at. And I ended up staying here because I love it so much. So yeah, I think it is something that really is an interesting career, and really is a way to get your fingerprints on a game and put your influence on a game's development without having to necessarily be a creative or something along those lines.

Oh, crashes. Crashes are the main bugbear, especially if you're working on a test, you get right to the end and the server's gone and you're like, no, why?

Absolute pain in my life. You're just trying to get something, you've been all set up, you've been with somebody else, multiple people, you're just set up to get something tested, and nope.

Yeah, you're managing to hook up your thing, you're about to catch the bug that you've been hunting all day, and then the pop-up.

A completely unrelated crash.

Yeah, completely something else, and you're like, no, I don't have time for this. It's never the one you wanted. Yeah, I guess that would probably be my least favorite aspect: when bugs pop up in the search for other bugs, and you just have to go, not now.

Shh. Go to sleep. Add it to my Post-it note of things I'll have to look into later.

The good old sticky note of unloved bugs.