Star Citizen Live: Rivers and Tech and Bugs, Oh My!

Will Hain · Engine Programmer 3, Planet Tech

Honestly, I ask the same question. I think it's roughly years of service, but I think it's like which year of service you're in, because I'm approaching my third year, but it's also not at the same time. I've not got a clear answer out of anyone. It means I'm better than a two, I guess, depending on who you ask. I'd hope.

Jared Huckaby · Host, Community

Now, is that real years or game dev years?

Will Hain · Engine Programmer 3, Planet Tech

So that's here years. Game dev years, I guess, if you were coming from somewhere else. But this is my first job.

Will Hain · Engine Programmer 3, Planet Tech

Engine programmer could be anyone really, working on the engine, but more specifically I work on the planet tech team. There's four of us, five depending on who you count, and we work on all of the procedural tools like the rivers, like the planets, like some of the graphicsy stuff, because we're kind of a graphics team and yet kind of not. But generically, I'd say an engine programmer is just going to be working on core tech and low-level engine stuff. Specifically, I work on the procedural, the building big things from small amounts of data very, very quickly sort of side of things.

Jared Huckaby · Host, Community

Such as?

Will Hain · Engine Programmer 3, Planet Tech

Such as the planets, such as the rivers, that's what we do. We have these enormous giant-scale planets that we build from less than 10 megs of data, and we do it at runtime as you fly around them. That's what my team manage and work on and improve and develop new features for.

Will Hain · Engine Programmer 3, Planet Tech

I'm Will. I'm a two-dimensional character. I exist purely for Cloud Imperium Games. I went to uni down in Cardiff, where I studied game design and development. I've done programming since I was about 11. I used to use a program called DS Game Maker where you could make very, very bad games and then put them on one of those R4 ROM cards you can put in your DS with a micro SD card in. And that's my first exposure to programming. But it just kind of snowballed from there. I really wanted to do music for a long time, but I refused to do it academically because I wanted it to be organic. And then I thought, well, I need to do something that will definitely earn me money so that I can afford to do my music, and I thought I'm pretty good at programming, so I'll roll with that. I did the course in Cardiff, which was great, and I did a dissertation in planet terrain generation where I essentially made this GPU-powered planet generator based on ridges and rivers, which would then propagate out to an entire planet. Very bad it was. Anyway, I applied to a couple of jobs a couple of months before I was going to graduate. I applied here. Marco, you know Marco, he picked it up, interviewed me, really liked the stuff in the dissertation. He was like, so we're doing something similar, and just brought up New Babbage on this big screen in the interview room. And I was like, yeah, that's cool. And that's how I got here. That's who I am.

Will Hain · Engine Programmer 3, Planet Tech

Absolutely. I had this game. No one's allowed to copy this. I had this game idea when I was about 11. I wanted to create a game called Infection, where you were essentially a little nanobot flying around in someone's veins shooting viruses, was my concept. Anyway, I spent a very long time in MS Paint making 2D backgrounds of veins, I guess, and this little ship that would fly around. And then tried to make that happen. And you could just about move around. We had, we, I had this, it was we, I didn't have a studio at 11. You could paint the map down and then you could paint a magenta overlay to show where you wanted collision to happen. So there was no real physics. It was just if you hit the magenta, you bounced back sort of thing. And it was tacky and horrible and terrible. But that was the first thing I did.

Jared Huckaby · Host, Community

In one of those programs, the games that let you make other games.

Will Hain · Engine Programmer 3, Planet Tech

It wasn't a game as such. It was made by some other young guy. I think he was like 15 or 16. And it was made in Windows Forms. So it was a real program. And the idea was you used one of those GameMaker-y, Scratch-y sort of programs. You used event blocks to trigger stuff happening, but you could go in and look at the code behind those event blocks, which I think was in C. And so I started copying events and changing little bits to try and make it do something similar. I had no idea what I was doing, but I just experimented until something worked and pestered the guy who made this game maker far too much.

Will Hain · Engine Programmer 3, Planet Tech

Well, rivers is a funny one. So back to that interview with Marco, when I'm sitting there looking at a picture of New Babbage on the screen and he's saying, how do we solve this problem? How do we solve that problem? And then he says, can I have a look at your dissertation? So I bring it up on my laptop and I've got rivers. I mean, they're just trenches at this point. And I say to Marco, have you got rivers? And he says, we haven't got rivers. So about five, six months into working at CIG, Marco asked me to start working on a terrain modification system. You remember we did the first ISC about that, my first, not yours, because we had no way of modifying the terrain currently. It was fixed because of all the shared data and nothing being local, blah, blah, blah. Anyway, so I started working on a solution to that. And then I said to Marco, this is going really well. I could use this to make rivers. Do you mind if I do? And he said, no, that sounds cool. So I did. It was really nice to have that freedom, and obviously I couldn't put that much time into it to start with because it was kind of like a side project. I remember the first version of it, we deformed the terrain and then raised the ocean to fill the gap, and it was a real gimmick. It looked really bad. The waves were going the wrong way. And it was the same mesh, so we had no control over it or anything really. But it just about worked. And anyway, he showed CR and we managed to get some time scheduled for it. And that's how I started working on rivers. It was really cool to go from saying I want to work on this to it being a feature. And I remember it coming up on the roadmap for the first time and being really excited. I worked on something that's on the roadmap, and being able to tell people about it. It was cool. I really like the rivers. I still need to do more on them. They're still my baby. No one else takes the rivers.

Will Hain · Engine Programmer 3, Planet Tech

Yo, Chat. Rivers. The reason I think rivers are really, really cool and really, really important is they are almost just a catalyst to improving the planetary tech in and of itself. So everything that has been built around rivers and built to allow rivers to work has generated improvements on the planetary tech itself. So we didn't have terrain modification. We wouldn't have taken that as far if there wasn't as big a mandate as rivers. We didn't have this specific control over what spawns in what area depending on local features. But because we wanted it for rivers, we drove that and now we can use it everywhere. So it's not just a cool feature in a cool location, and it doesn't just add to the realism of the planets and add unique locations that we really didn't have before. It is a way of not only showing off new development in the planet tech and the procedural tech, but also a catalyst for generating the need for work and the drive to provide these new features. It's all well saying, well, I want to be able to control specifically at what level all these rocks want to spawn. You say, well, why do we need that control? What feature demands that? But actually what you can say with the rivers, you can say, well, we need it for this. So we make it for the rivers and now we've got it for everything and we can use it for all the other locations. And when we start introducing more unique POIs, we have this control and we have this technology that we wouldn't have otherwise.

Will Hain · Engine Programmer 3, Planet Tech

Cliff sides. So the new river cliffs in 3.18 are really cool, actually. So the piece of technology that allows them to exist is this thing called on-demand spawn points. I won't go into too much technical detail, but the spawning system is run by a piece of code called Biome Builder, which I help look after in Planet Tech. And that has lots of cells across the planet, a grid of cells across the entire planet, each of which have a set of spawn points. Spawn points are usually random, and then they take their properties from the terrain in order to decide what to spawn. On-demand spawn points is a little bit of code that allows us to hook into that and say, we want a spawn point here. We want it right here. And we want an object to appear. And you can even specify some overriding stuff as well, which is pretty cool. Now that was developed for the rivers and then for the cliffs, to allow us to say, well, we've got all these rules about where we want such and such a rock, but you can't always just cover for this all in rule sets. Sometimes you need specific art direction. Sometimes you need to say, well, this is a river gorge. We need a wall of river cliffs going along the side. So that's the kind of tech that's driven that. We want to be able to place this on naturally occurring cliffs across the planet as well, which will be really cool, because then when you get those really steep slopes that naturally appear on our planets anyway, we will be able to populate them with the really beautiful cliff meshes. But there's also some rumours and some talks about going a different direction with that and doing more of a technical approach, which I won't spoil, but it would be really cool if we end up doing that. I hope we do. But the on-demand spawn points are driving it at the moment, and they're really cool.

Jared Huckaby · Host, Community

More of a technical approach?

Will Hain · Engine Programmer 3, Planet Tech

More of a technical approach. GPU-y stuff. I'm not going to spoil. That's not my place to spoil.

Will Hain · Engine Programmer 3, Planet Tech

If I could choose what I was going to work on next, God, there's so many things. I really want to go back to the spawning, and I want to make our planets more physicalized, which means nothing. I want the planets to be realistic. So we currently have our climate, we have temperature and humidity. I want more detail, more control, more proceduralness, so that you can click a button with some randomised parameters and generate an entire unique planet just like that. So I want to say, well, this is the local sun of the solar system, and this is where the planet is, and these were the materials that were around when it was forming. Where are we likely to have continents? Where are we likely to have resources in the soil? What plants and rocks are likely to have come here? I'd like to, and I don't know if we'll ever be able to do this, but I'd like to do full terrain simulation. You could do those cool simulations where you have a flat piece of terrain and you start introducing water droplets and it forms the naturally occurring erosion, because that's not what the rivers are. The rivers are finding where a river works given the current terrain, because we can locally edit terrain with a terrain modifier, but we can't completely reshape the planets, because our procedural tech is relying on reading these fixed height maps that we have. We can tweak it here and there, but we can't reshape the whole thing currently. But it'd be really cool to do a full erosion sort of thing. Whether that's ever going to find its home in the PU, though, in our Star Citizen, where we're finding hundreds of planets and stuff, I don't know. But that's the sort of thing I'd love to work on.

Jared Huckaby · Host, Community

It's hard, because I don't think anybody's going to say we don't push technologies in ways that maybe other games sometimes don't. But there are always considerations for the multiplayer aspect, for the MMO aspect, for the server. There are things that we could easily do just for a single player. You can control the perspective. You know it's only being seen from one place.

Will Hain · Engine Programmer 3, Planet Tech

Exactly. I mean, with the planets, the best example is if you've got one planet and you've got 20 gigs of hard drive space, you can go into so much detail and you can have so much locality. But actually, that's not where we need to push the boundaries. What we need to do is push the boundaries so that you can have hundreds and hundreds of planets in minimal, minimal data in one game download. And as well, if you wanted accurate height maps of Earth, that's terabytes. And if you want the level of resolution that we have on our planets on Earth or something smaller even, it's just not possible to read it from raw data. You can't do that. You need to be streaming it in like Microsoft Flight Simulator does. And even that, that's a really, really, really beautiful game. But you go to the terrain and it's not the fidelity that you'd expect when you're walking around the terrain. It's a flying game. It's designed to look good from above. And it really does. But that's not the problem we're trying to solve. We're trying to generate these beautiful, unique planets, and we're trying to do it in such a way where it's scalable, because we don't need a one-to-one replica of Earth that is down to the centimetre in every single place. We need fully explorable planets, and we need many of them, and we need it to not take up several terabytes on each of our users' hard drives or SSDs. So pushing the boundaries in the right way, I think, is the right takeaway from that.

Jared Huckaby · Host, Community

There are probably six or seven people all running to Reddit right now to make a post about how somebody at CIG said they can't do Earth and Earth isn't coming. He's talking about the current tech and the current place that we're at and the current resources we have and the current intentions and the current directions. Obviously, folks who have been following the project for a long time know that Earth still exists in the 30th century. And when it comes time for that point on the development wheel to roll around, we'll be discussing how and when and where we'll be doing that.

Will Hain · Engine Programmer 3, Planet Tech

I'm not unconfirming Earth. I'd say if you're working on planet tech and you want to make Earth one day, then each of the decisions leading up to that needs to take it into account to some degree. You don't want to accumulate all of that tech debt.

Will Hain · Engine Programmer 3, Planet Tech

I want to work more on the spawning stuff, but I want to make it more physicalized. That's one thing. I could work on any number of things. I want to do lava and volcanoes. Lava, volcanoes, that'd be cool.

Jared Huckaby · Host, Community

Did you say volcanoes?

Will Hain · Engine Programmer 3, Planet Tech

I'd love to do them. I'm not saying I will. I'm saying I'd love to.

Jared Huckaby · Host, Community

It's just like a vertical river out of a mountain. Just turn it on its side.

Will Hain · Engine Programmer 3, Planet Tech

Maybe for a lava river you could say that. It's like a slow-moving river with more VFX. But what if you want the lava to set? And what if you want to put on your fire hazmat suit and walk through it? There is tech and there is work and there are more features needed to implement something that looks like an aesthetic change on the surface, because it's all the gameplay. That's what we got with rivers as well. When we got it looking good, when we got it ready to release, the thing I was working on in the months leading up to putting out our first river was not making the rivers look better. It was the gameplay. It was making the grav-lev vehicles recognize it. It was making the player drown if they take off their helmet underwater. It was the VFX not spitting up dirt and stuff when you fly your hoverbike over it. It was all of the gameplay and none of the beautification anymore, because there are so many facets to putting something in a sandbox world like ours. It's not a simple, well, here it is, it's done. You've got to think about everything the player can do to it, everything the player can do there, what's going to look stupid, what's not going to work. It's a big job. It'll be the same for lava when it does come. I think it will come. Someone said it would come. I hope so. I want to work on it.

Jared Huckaby · Host, Community

Look into that camera and say, we are definitely doing lava.

Will Hain · Engine Programmer 3, Planet Tech

That's not my call to make, Jared.

Ray Warner · QA Pillar Manager, Squadron 42

So I am Ray Warner. I am currently the pillar manager for QA Squadron 42 testing. And that is all I'm going to say about that.

Jared Huckaby · Host, Community

Well, what's a pillar manager?

Ray Warner · QA Pillar Manager, Squadron 42

So I am primarily responsible for all of the project management of Squadron 42 as far as QA is concerned: setting up how we're testing it, covering all of the systems that need to be covered by development as they iteratively develop the game. And pretty much trying to mirror what Phil's doing with the PU for Squadron in the run-up to the inevitable release.

Jared Huckaby · Host, Community

And Phil?

Phil Webster · QA Pillar Manager, Persistent Universe

I'm Phil Webster. I'm the PU pillar manager for QA. I look after the Persistent Universe in Star Citizen and oversee all of that and testing for all our live releases. Don't ask me about 3.18 coming out.

Jared Huckaby · Host, Community

And Rodolfo, who are you and what do you do?

Rodolfo · Global Director of QA

I'm the global director of QA, and I'm running the different departments that we have in QA for the different games, and we also do some other things like automation, test automation in the games.

Jared Huckaby · Host, Community

The global director of QA.

Rodolfo · Global Director of QA

Yes, we have QA teams all around the world in different studios. They happen to be the managers that we have here, but the way we're structured, we also manage the people who happen to be in other studios. So for example, if someone is testing something related to Star Citizen in Germany or in the US, it falls to the person who takes care of it, even though we do have people and leads in each studio.

Rodolfo · Global Director of QA

We have an important team, a big team that is growing. We're actively hiring right now.

Jared Huckaby · Host, Community

More than 10?

Phil Webster · QA Pillar Manager, Persistent Universe

Way more than 10. Significantly more than 10. When I started, there was 14. A lot bigger than that now.

Phil Webster · QA Pillar Manager, Persistent Universe

It was a daunting prospect, but one that I was really, really eager to get my teeth into. I was a backer before I started working here, and Star Citizen is the game I've always wanted to play, and being a very big part of that in the QA process was something that was very, very important to me and very exciting. And also being able to build a department pretty much from such a small base to what it needs to be and what it is now, and making it a department where people can come and have fun while they're working. You hear a lot of tales from other studios and places, no names or anything, but you hear horror stories about what it's like to be QA. And we want to make sure that no one ever felt that way when they came to work here. And we pride ourselves on that fact.

Ray Warner · QA Pillar Manager, Squadron 42

That's a super important thing. And it's something that we really missed in lockdown. Obviously, as everybody moved into that remote situation, we didn't have the ability to just tap a developer on the shoulder and be like, this is a bug. But now that we're back in the office and we're growing the team again, and we have so many embedded positions that are working alongside that, it's super important, not only for those 'look at this bug, this is so broken, can you fix it?' moments, and that on-the-fly 'yes, done, fixed, submitted', getting stuff out faster. But also the relationship and the processes that we carry in, QA have so much more visibility across development. So when people are making these big features or creating new planets, adding new rivers, putting in a volcano, for example, they have people next to them who can say, there are limitations to the way that you can do this. This is what's happening when you spit it out. One of the things that we see with planets quite a lot when they do reworks on planets is the terrain will shift in height and none of the bases will. So you have to have people who are there to go to those bases and know this kind of thing from the history of these kinds of updates, to help development get past those, especially with the turnover that we see, the growth and the turnover that we see in CIG. It's so hard sometimes to maintain developer loyalty to a very specific feature. It just rolls through so many people that we need this kind of point of information. So it's really, really helpful.

Ray Warner · QA Pillar Manager, Squadron 42

It's super easy to get frustrated about patches not coming out for this game. But one of the reasons that happens is so people don't have those situations where they forget how pretty the game is. They get to experience it at the best possible state that we can get it to at the time. We don't get to experience that. The vast majority of our work is repros or hitting crashes and getting the absolute worst of the broken parts out of the game prior to putting it into the hands of backers for the PTU, the Evocati releases, and figuring out the larger-scale problems with server longevity or the amount of players that we have. We want to get the ability to do that before we give it to you, and that's what we face most of the time. So there are elements of playing the game, and I don't want to...

Phil Webster · QA Pillar Manager, Persistent Universe

It's a job. Testing isn't just playing, and there can be laborious and monotonous parts to it, but that's not to say we can't have fun and enjoy the game as well, because, like I say, I was a backer and I enjoy playing the game a lot. I'm a role player and I like to role play in Star Citizen. If you ever meet a guy called Stranger in there and he asks you to do something, he might have a decent mission for you to do. The sandbox nature of the game, you can make so much fun with what's there. It's crazy how much you can do, and you can have so much fun, and we do at work as well. But it's a hard job and we have a very important part to play, and it's recognized here at CIG, which is brilliant and fantastic for us.

Ray Warner · QA Pillar Manager, Squadron 42

We have a name for people like Phil and I: QA lifers. You get people who enter QA through several different means. Some people pursue the games industry, some people want to play games. This idea that it's just playing games, this is what pulls a lot of people into QA. But there's a specific skill set that comes really natural to a lot of QA testers, and it's the idea of problem solving and being interested in those problems and finding those solutions. And it's a real passion for that that drives people to continue a career in QA rather than chasing development. One of my favorite aspects, don't get me wrong, is seeing the development of people's careers as they come into QA, and whether they go up in QA or whether they go up in development or production, it's all amazing to watch people grow and gain knowledge of the product. But you get some people who are just wired to find the problem and make that problem worse and then tell people why it's so broken and then get that fixed. It was a particular passion that I found when I moved into QA, and then obviously as I got better and honed that skill in the 10 years that I was at Sony prior to coming to CIG, it became less about how I can branch out and become part of the creative force that make games, do the art or do the programming, and more about becoming someone who gets to see everything. Because that's what you get to see in QA. You are one of the few departments who see literally everything that gets put into a game. And it's such a rare thing to see in the games industry that I just fell in love with it, and ever since, it's amazing. I love it.

Phil Webster · QA Pillar Manager, Persistent Universe

The career pathing we have from QA touches everywhere. It can go into any aspect of game development or game production, or staying in QA and leadership and management and looking after people and project management. You've got a finger in every pie, and through that you develop a lot of relationships with all the departments, and you build those relationships, so you know all the different departments there are and you get really good relationships with them, so then you are able to help them better. Because QA is a service, right? It's a service for the entire company. And we have to provide the best service that we can. And we do that through building great relationships. And the best way to do that is have people come from us into their ranks, and they understand because they've been where we are. They know what information is great for us, what helps, what doesn't help. Riffing on that and going forward with that and pushing people forwards and keeping hold of people as well.

Ray Warner · QA Pillar Manager, Squadron 42

It's a really interesting aspect of it, having people move from QA to a production or a development process, because it can change the way that a department or a particular focus works, because they go in with so much understanding of what we're going to do to try and ruin their day at the end of it. We get these huge benefits of people saying, well, I understand how testing works now, I understand the process QA go through, I understand what they're trying to fix, I understand my feature, and this is a big one, my feature has an interaction point and affects other people's work. And that knowledge is something that's very, very rare to get when you come out of university and go into development and you're so siloed in the process that you're doing. So I love pushing QA into production. I love pushing QA into dev, but I do wish sometimes that I could keep them for myself.

Rodolfo · Global Director of QA

We do have guys from QA who have gone to all the areas. We have people who have gone to sound, people who have gone to art. Of course, a lot went to production. It's very, very common that they go to design, to game design, and the skills that they get from QA are very useful for that. But on top of that, we have people who actually code, and we have people who are doing automation, which includes coding and their relation with the code of the game. And these people develop skills that are super useful for development, but also for design. So it's more technical than what people think. It's a whole craft. And I always use an analogy, because this happens in the software industry a lot, that for some reason the QA work is not appreciated enough. The same way there's people who have a mindset that is very good for constructing, a constructive mindset, for QA you need to have a deconstructing mindset, and that is a skill in itself. It's very different, but it's very, very useful. And in other disciplines, it's not like this. For example, in physics, you have the Nobel Prize, right? And you give the Nobel Prize to people who are theoretical physicists, people who make theories, like Einstein. But you also give the Nobel Prize to people who are doing experimental physics, which is basically trying to prove if the theory this guy came up with is valid or not. And you need to make an experimental design of how to test that, how to test the theory of cosmology or quantum physics or whatever. That's exactly what we do. It's just we do it with software. So we're basically empiricists, experimental scientists working with software instead of working with physics or chemistry. That's what we're doing. And there's a lot of techniques behind it, a lot of knowledge and a process. It's not as simple as just playing the game. There's a way you have to approach it to do it in an efficient way, because you cannot find all the bugs in an infinite universe like we have just by trying to go through it. It will take you forever, so you need to do it in a certain way.

Ray Warner · QA Pillar Manager, Squadron 42

Absolutely. That's what I mean when I say there's got to be a very specific mindset to get into QA and remain good at the job. It's about problem solving. And when I say problem solving, it's not about being presented the problem and finding the solution, it's about finding the problem. And that's very much what you're talking about. It's about seeing a piece of software or seeing a feature or seeing an element of gameplay and thinking to yourself, and our backers are great examples of this, how are people going to come in and ruin this? Because they will. The first thing they want to do is find easier ways to do things or better ways to get things, ways to exploit the code, ways to exploit the gameplay. We have to approach our testing methodologies with this in mind from the very, very get-go.

Phil Webster · QA Pillar Manager, Persistent Universe

Right from the start, like in the kickoff meetings, the amount of times when designers come in and they do the pitch and someone from QA will go, so that could be weaponized by doing A, B and C, right? And their faces drop and they're like, oh, yeah. But that saves time, and it gets that fixed before it even hits the code base. So it's very important.

Rodolfo · Global Director of QA

We actually had one of those today, right? It's very important because sometimes you can have a design that will not work out in the whole universe, and people don't figure it out until it's coded and gets into QA hands. So we actually test it and it's too late, because you have to drop the code and start again. So it's so much better when you can have this interaction in the early stages of the development lifecycle.

Ray Warner · QA Pillar Manager, Squadron 42

I think one of the easiest draws is some of the dynamic mission stuff that we do. There's a lot of elements to our dynamic missions because they attract so many players, and the unique elements of it, like the Idris attacking or the flight shows. And probably one of my favorites, and I apologize, Phil, because I'm sure you were probably going to say this also, is when we brought the Idris on and they were like, here's the Idris, now everyone can look at it, it'll be amazing. And the first thing QA say is, someone's going to get in it and take it over and then fly around in the Idris. And they're like, no, that's not possible. We've de-physicalized all of the floors, it'll be fine, don't worry about it. And within, what, 10 minutes of the release, someone had clipped through the ceiling, landed in the seat, got in the seat and started flying around in the Idris. And it's like, well, we told you. But that always comes up against that kind of, we've done it, this is it, we're going out live. PTU released, damn it, we're going to have to go back to square one and figure out how to make this work. And it happens constantly. But there's a great satisfaction in 'I told you so'. How about you, Phil?

Phil Webster · QA Pillar Manager, Persistent Universe

Well, Ray totally stole the one that I was going to use. So I'll just say, about Ray's particular project in Squadron 42: redacted. So unfortunately, I can't tell you mine.

Jared Huckaby · Host, Community

What about you, Rodolfo? You don't make it to global head of QA without having to crush some feelings.

Rodolfo · Global Director of QA

I have some stories from long ago from other projects. I don't know if I can share that, but we do have them. I do remember, for example, my very first job in QA, when I was a tester many years ago, I was on a mobile game for iOS, and they were pretty sure that it was working perfectly. It was a physics simulator, so in some sense similar to this. You know The Incredible Machine? Do you remember the old-school games? Well, we made a remake back then at Disney for mobile. And I tried to stress it a little bit. So when you start stressing it, you saw that all the cables and things start wobbling around, and you can just do whatever you want. And it's the kind of thing, oh, but no one is going to do that. Well, we did. Maybe the players will do as well. You always have that kind of discussion, right? They only expect people to use it as the feature is intended. And actually players, if they are allowed to do it, they will do it eventually.

Ray Warner · QA Pillar Manager, Squadron 42

At CIG specifically, one of the big problems that we have is we have these universes that have hundreds of players, potentially in the future hundreds of players in a server at any one time. There are things that happen in the back end when you put that many people doing random things. As we say, people get destructive when they play video games, and when you put a hundred of them in and let them do what they want, it's very, very hard to replicate that in a controlled environment. And that is fundamentally the baseline for QA. We need to be in a controlled environment so we can find those steps, find the cases of how it happens, when it happens, what you need to do to make it happen. And when you put a PTU build out and we get 50 crashes, or we get the elevators killing people, everybody knows this, right? We put it into a game. We get 10 people into the game. We say, just ride elevators for a day and tell us what happens. And it's fine. It runs great. And you're like, why is it running great? And it's really, really hard to replicate. And you can do it in the PTU, but you can't do it in a dev build. So you can't get the information you require. And it's super frustrating. It really is.

Phil Webster · QA Pillar Manager, Persistent Universe

And we have to deal with multiple builds a day, whereas the servers will be up for days at a time with this amount of people on them, and so it's the soak time and stuff as well that we just can't replicate, because we have to keep on iterating and getting the fixes in, making sure they do work at a base level, so that we can get it out to the backers and they can start playing it and enjoying it as soon as possible.

Ray Warner · QA Pillar Manager, Squadron 42

And then you get Rob Johnson, who comes in and goes, oh yeah, I'll fix that. And it's done. And it's like, I've been locked into this for a week.