George Stabler

And that touches on a bit of the things that people are saying there in terms of the 30Ks and stuff. We probably have felt your pain thoroughly every day while we're trying to test stuff as well. It is definitely a struggle, but like I mentioned for the last point, we are information brokers. We deal the information, but we don't get to decide when a developer has time to get in and actually fix the problem. So unless it's something that is really severe and a lot of people are seeing quite consistently, it might be something that people have to deprioritize. QA do try and raise flags about severe issues or things that we find particularly frustrating, gameplay issues or particularly ugly art issues, all that kind of thing. But it is, at the end of the day, down to the developers to find time to fix that kind of stuff. They also have to find time to create new content and jam all of that in there and somehow get that released out to you guys. Honestly, they do incredible work to manage to get that stuff done. So yes, unfortunately, some of the more potent smaller bugs may not get fixed for quite a long time, but we commiserate. And do please keep sending them in to us. We love to try and deal with issues that the community call out particularly. I mean, George will have dealt with some aspects of that with your old job as well, didn't you, with the player relations stuff?

Yeah, so one of my bits in player relations, I believe it's become more of an actual team now, one of my jobs was with Chris Danks back in the day: whether it was a big patch, collate all the major information from the community, like go into chat, look at the Issue Council, everything, and find out what was annoying people most and send the top 10 over to QA to have a look through and get bugged up or linked to bugs which were already in, and make sure the developers were actually aware of the community's pet peeves, the worst ones affecting the people on the PTU.

A lot of the assumptions that we see publicly are that if a bug makes it to live, and this is in any game, we're talking in general here, anything from Cyberpunk at full release to us who are in alpha, there's the assumption that if a bug makes it to live, it must not have been seen, or it must not have been noticed or found. When the reality is often that these things have to be prioritized. Even with 700 staff working in five studios around the world, there are only so many things you can be working on at any given time. And it is often a triage of figuring out what the most aggressive bugs, what the most invasive bugs are, and then the ones that you can have solutions for. Not everything that you encounter has a solution that becomes readily apparent. And some things can't be fixed until other systems come online or other tools come online. You know it's important, you know it's super invasive and drives you and everybody else crazy, but the tool that you need in order to be able to address it won't be ready in time. But one of the things I want to make sure that we demystify here is that they do not go out because they are not discovered. I'm sure that happens sometimes, occasionally, but it is just usually a matter of triage.

My heart goes out to you guys because we feel it too. But yeah, like Jared says, there are occasions where there are things that slip through the net. We have about 100 QA staff across all of our studios, and it's just not enough for a game of this scale when you've got thousands of people out there playing it. So please do keep telling us about your stuff. You never know, it might be something that we've missed along the way, and we would love to hear about your problems.

Or enough reports can help change the prioritization.

Exactly that. And the new Issue Council is absolutely fantastic. So yeah, definitely make the most of that and use it, because those Player Experience guys are doing wonderful work with it.

Not at all. Definitely not. When it comes to devs and QA, obviously, we are there to somewhat ruin the day. And we do come in, and sometimes there'll just be something that they have not expected that crops up. But the vast majority of the time, I think we have a pretty good connection with the devs. None of them have called me names to my face. But yeah, we work very closely pretty much from the beginning of when a feature is being designed. So for example, at the moment, in the run-up to 3.15, we've got the healing and inventory changes that are coming in. We've been working really closely with the Actor Feature team on that. Guys like Chris Parry doing absolutely wonderful work, and they guide us and we help guide them. So we're all striving towards that same goal of having a good quality product. Developers also test their own work. So it's not just us that are coming around and saying, "Hey, it's broken." If they go through and launch it up and something's broken, they also don't want that. So yes, as much as it is kind of like a Tom and Jerry cat-and-mouse comedic relationship in theory, in reality we do have quite a good bond with those guys, which is really fun. And honestly, QA gets to touch on so many different areas of gameplay that we do have a close connection with all of the devs. Say, for example, artists may never get the chance to work with a particular area just because of the nature of their job. So it's really great to have our tendrils in there. And George being directly in the network team, obviously, if you didn't have a good relationship with those guys, that wouldn't work so well. I'm hoping stuff's good.

Yeah, stuff's going very well.

Also, because we touch on everything, we potentially know a bit more about other features than the devs, because they'll be very focused on what they're doing, their one little thing. For example, back when I was doing UI stuff, when I was here before, there was an occasion where literally they did something and it impacted on another system they hadn't even considered. The second we saw it, we were aware and we obviously fed that back. So having that relationship with the teams, you've got the ability to give them knowledge they sometimes don't necessarily have access to, because they're so focused on their one little area.

It definitely doesn't make it harder, no. It maybe gives us a little bit more work to do, but there's always plenty of work to do with Star Citizen, so that's totally fine. No, we work really closely with the Player Experience guys to investigate those issues that come in from Issue Council. Like I said, those top 10 issues do come to us. We do look into them. We do make sure they're updated on the database. So you can rest assured we are definitely touching your bugs. We do see them and we do appreciate all of the info that you can give us about those particular issues, especially if it's something that's on and off. Sometimes having that stuff from the Issue Council is the key to us finding out exactly where to go and how to provide the right debug. If it's a particular thing that we didn't think of along the way, a step that we were missing, that's super useful to have. Outside of the Issue Council stuff, we do take things from the community, from the forums. If there are feedback things that we think are probably bugs, we do pick up those as well. Videos, for example, on YouTube, all of our content creators and streamers, we do get a lot of that content sent through to us saying, "Hey, have you seen this happen? Is there a bug for it? Help." And we'll usually trawl through our database for that. So yeah, it's really an interesting position here at CIG, especially in comparison to other QA roles I've done in the industry, where we are fully tight-knit with the community because we are releasing constantly. And it's an experience I've just never had before. When you're working on a project that is going to be released and shipped off and nobody else is looking at it, that's different; this gives you a whole other perspective, and you're able to do those big load tests of thousands of people, or 50 people to a server every single time, which we don't often get to do in QA. So that's really an experience I didn't expect to have when I came to this job, I guess, and has been really interesting. It is like having another crew of testers that could help us sweep everything that we need to do. How have you guys found interacting with the community?

It's really, really useful, because I know back in the day there was an issue we had at Levski where occasionally everything would go psychedelic colors. And we saw it three, four times in testing and we couldn't repro it. There was nothing we could do. It just happened every so often. And so we couldn't work out what was causing it at all. And obviously it then went to the PTU to be tested. And actually, because it gives you those extra people throwing at it, we could see what was causing it and pass that on. And it's so useful to have that information to be able to feed back. Certainly in situations like that, with things which are ultra-low reproduction, the chance of us reproducing them in standard testing is just luck, whether we get it and we can reproduce what caused it or even work it out at the time.

So for starters, things like video footage and screenshots. Obviously, if you can provide that kind of physical evidence, it makes it a lot easier for us to see things that perhaps you don't know the particular word for externally, so having those videos is really useful. Particularly mentioning the types of ships and the variant of ship that you are encountering an issue in can be really useful to us, actually. A lot of bugs end up being specific to certain manufacturers or certain types of ships because of the way that they're set up. So if it is a ship bug, maybe go into detail about the ship that you have, whether you've changed the loadout, or if you've changed any of your settings, so keybinds or graphics settings, that kind of thing. Really, honestly, the more info you can provide, the better, basically. Do you guys have any other suggestions for things that people could provide?

Not really. I don't think I've ever seen an Issue Council bug and gone, oh no, they've given me too much information, please add less to your bugs. The more you can give me, the happier I am, because it gives me a much better starting point to try and reproduce what you're seeing in game. If you've given me loads of information, I can just jump in game, get to it straight away. I can still do that if you've given me less information, but it might take me a bit longer. So yeah, the more information, the better, really.

And don't think, oh, that's been there forever, they'll know about it, because we might not. It might be something which has been missed and missed and missed. And if you raise it, we'll then go look and check. And we might even go, oh, that's something which has been there for ages, and then we'll go search and we won't find it. So even if you think we must know, let us know about it.

Or it can be something that was deprioritized, and every time it happens, that can help to up the prioritization of it. So just because you think, oh, they must already know about it, does not stop you. Submit that, submit that, submit that. It helps in a number of ways.

So since we have started working from home, literally all of these guys have come in during the working-from-home time. I literally have never seen any of these guys in the office, which is a crazy thought to think about now. It's strange. My brain kind of places people in the office as if they were there the whole time. I have no idea what height they are. I just kind of guess. Yeah, it's been a huge challenge to adapt everything to working from home. But honestly, through doing voice chats each day, just while we're working, we'll use voice chats with each other. It almost feels like being in the office these days. And it's really useful for me to be able to screen share while I'm walking through particular things. I do miss the aspect of being able to lean over and help guide someone directly. But we do our best, and the wonders of technology have allowed us to do a lot. How was it for you guys, starting a new job while working from home?

It was actually really good. It was really, really well done. It was a bit weird on the first day, but it was so smooth. So just basically straight into it, it didn't really feel any different from starting a job anywhere else. It sounds daft, but that's good.

It's been an interesting experience. It's been very different. There are times in training, and even after you finish your training, because 18 days sounds like a lifetime of training, but you've barely scratched the surface. There are times when you're new and you're unsure about things that it is a little bit anxious, a little bit stressful, not having a person in the room with you to, if it all goes horribly wrong, just jump in and fix it for you. You do have to be a little bit independent and work with people talking in your ear like, OK, I know it sounds terrible, but it's not actually that bad. We've seen this before. It's fine. What you need to do is this, this and this. So I think during the training period, that is a consideration, but you work through it, and I hope it's all worked out. No one's fired me yet, so I must be doing something okay.

He's doing pretty okay.

I think Jack can answer this the best, to be honest. You deal with the most crashes on your team, I think.

The stream I work on is notorious for being potentially the least stable one in the game. Patience is obviously a big part of it, but I think the thing I've used the most since I moved on to the core team is my memory, remembering things. A lot of issues tend to maybe disappear and then come back. And after a while, you just remember the really bad ones. So something might appear that's been gone for three months, and you can just look at it and go, oh, I know exactly what that is, I've seen that before. And you'll go and find it, and you'll link it back up, and you'll escalate with the devs and go, this has started happening again. What did you do, please? Please help.

The other one I'd say is curiosity, quite a bit, because you find something which isn't quite right, and you pull the string, and then you find something else, and then you find something else, and then you find something else.

I think that is a lot of what a lot of people don't really understand about QA sometimes. There's one thing, and you think, oh, that's fine, it's an easy fix. But then that will branch off into 100,000 different things that need to be fixed: so does that, so does that, so does that, before this main thing can be fixed. It branches off so much. Game dev is so complex, it's crazy.

Oh, I would have to say the amount of things that we learn. We learn so much in QA, whether it's parts of dev and just the game, and just generally how games are put together. I love learning things. So that's what I love about QA. It's just being a sponge of knowledge.

For me, there's a lot of different things. The thing I think I enjoy the most is the people. It's such a good, vibrant group of people who, even when you're having the horrible crashes, you'll be able to have a joke and have a laugh. And if it wasn't for a good group of people, I don't think I could do it. But it's such a good group of people.

I should have gone before George, because I was going to say the same sort of thing. I think, especially on the game dev stream, which does tend to have some minor issues with stability, you do form a real sense of camaraderie with the people around you, because it's not just you that's having a horrible time, and you've been trying to get into game for four hours and you've not made it past the main menu yet. It's you and seven other people who are also having a horrible time and haven't made it past the main menu yet. We're kind of friends for life, forged in that fire together.

I guess as well as being able to do the same things as those guys, I love learning stuff. I love picking the brains of all the different developers, especially the big-brain people like the directors who just know so much about how to make everything cutting edge. I guess just generally being able to work to support games is something that I really enjoy. When I came out of university, I did an animation degree and I hated animation. I came out of it and I was like, never again in my life. But I knew I still wanted to support developers, and I knew I wanted to support games development and do that kind of stuff, because that is something I'm so passionate about. I didn't have a clue what I wanted to do, and I just fell into QA by pure chance, because it was something that was entry level and everybody could have a go at. And I ended up staying here because I love it so much. So yeah, I think it is something that really is an interesting career, and really is a way to get your fingerprints on a game and put your influence on a game's development without having to necessarily be a creative or something along those lines.

Oh, crashes. Crashes are the main bugbear, especially if you're working on a test, you get right to the end and the server's gone and you're like, no, why?

Absolute pain in my life. You're just trying to get something, you've been all set up, you've been with somebody else, multiple people, you're just set up to get something tested, and nope.

Yeah, you're managing to hook up your thing, you're about to catch the bug that you've been hunting all day, and then the pop-up.

A completely unrelated crash.

Yeah, completely something else, and you're like, no, I don't have time for this. It's never the one you wanted. Yeah, I guess that would probably be my least favorite aspect: when bugs pop up in the search for other bugs, and you just have to go, not now.

Shh. Go to sleep. Add it to my Post-it note of things I'll have to look into later.

The good old sticky note of unloved bugs.