TL;DRQA works alongside developers and designers, who call testers over to try new systems; their input is given on the spot and sometimes leads to immediate changes.
Well, actually we have a pretty integrated work environment where we're working alongside the developers and designers as they're coming up with the system. So there's plenty of times when somebody will be working on something new and they'll call us over and say, "Hey, can you check this out? Is this intuitive? Does this seem like it's cool?" And we're able to give our input just right there while they're working on it. And sometimes even see them make changes at the same time: "Okay, well, what if I changed it to this? Can you try it now?" So we're in the trenches with them, which is a great thing for us, particularly coming from... I worked in publisher QA, where there was a big wall between developers and QA and you were pretty much just there to generate bug reports. We're right there with them. And it's kind of rewarding to actually be able to see, "Hey, I did that. I told them to do that" sort of situation.
BL
Ben LesnickDirector of Community Engagement, Online Strategy and Spaceships
That's one of the fantastic things about Star Citizen in general: there's this respect for everybody's thoughts. I'll watch Skype and I'll see Toast having suggestions for things, and there's not another game company in the world where your lawyer both cares and has good ideas.
TL;DRCompared with publisher-side QA, where a wall separated QA from developers, Vince Sinatra says CIG lets him work directly beside developers and talk to them in person, which he prefers.
Well, we kind of already hit on it, because like I said, I was publisher-side QA. So there's kind of this wall between publisher QA and then the internal dev QA. And you're pretty much just checking stuff, making sure things could go to code and pass submission for Xbox and Sony and stuff like that. So it is totally different to be there, like I said, in the trenches the whole time, being able to just walk up to people and talk to them, as opposed to, "Oh, I got to send an email to a producer," and then the producer sends the email to the actual developer's producer, and then eventually, three months later, you may or may not get a response. I actually kind of have a bad reputation here because I will just walk up to any developer, no matter what they're talking about, and I'll just interrupt and say, "Hey, what's going on with this thing? What's going on with that?" I just like getting my nose in other people's business sometimes when it comes to what they're working on, because I know I'm going to have to look at it in a little bit, and I'd rather get ahead of the game when it comes to certain things as opposed to waiting until it's dropped in my lap.
TL;DREveryone in QA touches everything, but testers are encouraged to specialize in areas they enjoy (ships, FPS, servers, Squadron 42), giving both QA and developers an expert to turn to.
Well, pretty much everybody touches everything, but we're really big proponents of a specialist system where you find a niche, something that you like. Like, I came to this company for spaceships. I was five years old, dreamed of flying an X-Wing, and so I was like, this is what I wanted to do, I wanted to work with the spaceships. So that's kind of what became my focus when I first started here. Other people have different interests, the first-person shooter aspect. So it's like, "Oh, hey, you really like FPSs? Well, you can become one of our Star Marine guys." Other people have expertise when it comes to server creation, monitoring bandwidth and all that other type of stuff, helping with the netcode. So we really encourage testers to find something that they like, something that they're interested in working with, and the developers that they have fun working with, and take it and run with it. Right now I'm working primarily on Item 2.0, which is really cool because I get to bother Mark Abent and collect his tears, as we like to say in QA, whenever his code doesn't quite work the way it's supposed to. But yeah, it's a lot of fun, and we definitely encourage specialization. So there are people who are focusing on Squadron 42, some people on Star Marine, other people on ship combat, other people on the persistent universe. And it works out really well, because that way you have an authority in every single studio who not only the QA team knows to go to, but the devs do too. So if a dev gets a Squadron 42 bug and they haven't been through what's going on, they're going to need a little help with how to get to the area where they need to reproduce the bug.
Oh, that's a good question. So yeah, the Mustang is also going to be getting updated. It poses a few special challenges, because when we first built that ship, we built it and all the variants at once, with three different artists working in two different studios. And so there's a lot that went into building that ship that's completely different from the way we do our ships now. It poses a bit of a challenge in how we tackle doing variants, to bring it in line with the way we currently do variants. So we will be redoing it, but that won't be for a little while yet, actually.
TL;DRMemorable bugs include the CitizenCon demo where the rover appeared to be born from the Constellation, and anything with inverse kinematics producing twisted limbs; other bugs are mostly just annoying.
One of my favorites is definitely during CitizenCon, the Homestead demo. We had a lot of problems getting the rover and the Connie working, and there was one where it basically looked like the Connie was giving birth to it, and it was a troubled pregnancy, obviously. That one got a few laughs. What else? I mean, a lot of them are just annoying, such as the aforementioned Mustang bug that we always refer to. We definitely want to see a lot of those get fixed eventually, but we've definitely had some funny ones. Anything with IK is always hilarious, because you get guys with twisted limbs trying to get into their seats. I don't even know what it stands for. That's what I was trying to avoid.
Inverse kinematics. So essentially what it does is it will figure out the best solution for moving a joint. So you'll give it a target, and then it'll figure out how to rotate the two different bones in order to give you a sensible looking motion. I guess it's an animation thing.
It's going. It's going well. We're trying to crank things out as soon as we can. 2.4 touches a lot of things, a lot of interconnecting parts, so there's, of course, a lot of things that can break. So we're hammering those out for the most part. Persistence, good times.
We're going to do a segment with Paul Reindell here soon about persistence and what exactly persistence is, why it's so important and how it lays the foundation for everything going forward. I think we're filming that on Tuesday next week. So look for it in the week after probably.
TL;DRThere is no answer on the number of systems by the end of beta, because it's too early and development is unpredictable regarding resources, breakthroughs and roadblocks.
No idea, man. We're still in alpha. We're still building out the alpha. It's a horse before the cart thing, man. Who knows how big the company is going to be, how many resources we're going to have, how many breakthroughs we're going to have between now and then. We didn't think we'd be this close to procedural planets until well after release. That's what game development is. It's not game construction. You lay out a plan. You have to have some kind of goal and whatnot, but ultimately, there's no predicting inspiration. There's no predicting when somebody on some team somewhere is going to have that breakthrough and some part of the game is going to leap forward, or when the exact opposite is going to happen, some part of the game is going to hit a roadblock and we're just not going to be able to find an answer for this thing for some time, so it has to be deprioritized while we work on the stuff that we do have answers for. So it's just way too early for that kind of thing.
Yeah, Mark, you want to come over? You want to hang out? We'll play the game with you. I know the actor who plays Morrow. I know he plays because I just set him up with an account the other day. Actually, at like 1 o'clock in the morning, Chris emails me because he's in the UK. So Chris emails me, hey, can we set the guy who plays Morrow up on the account? And I did it. I rolled over and got the email and rolled my happy ass out of bed. Set him up. I could have waited until the morning, but I was like, yeah.
That's a good question. I also have that question. I don't know the answer to that. I have asked and didn't get an answer. I think they're holding on to it to make it part of a bigger thing.
Nothing is final in this game. We're still working on the Constellation, so I know there's a lot of talk about that. So that is being worked on. Hopefully it'll be fixed soon.
Well, obviously, like I said, people are flipping out over the Vanguard in its current performance. Not final, of course. It's not quite operating up to specs as we wish it. That's probably number one on the agenda for 2.1. Obviously, people love the big guns, but they want to be able to swap them out. And it's performing a little too tight when it comes to maneuverability right now. So that'll probably be toned down a little bit.
I personally don't really like the way it's functioning currently. I don't like to look at it, don't like to test it. In fact I got some new testers now we're training up and i'm making them test it because they're the nuggets and that's their job.
I guess personally the party system right now. The party system bothers me but at the same time is I know that these are being worked on and a lot of the feedback from the fans. We've taken tonnes of feedback on things and, so we're definitely working on it. So we're pushing that soon. I cannot say when so do not ask when but if you're even thinking about asking don't even do it!
TL;DRGood bug reports are concise and clear, with steps and as much information as possible, including what you did beforehand, so QA can write good repro steps; adding a release-notes focus request is only an idea.
Yeah, I believe this is probably a QA question and Issue Council type question. So basically what we primarily want is people to be concise, clear, clear steps running through things you started doing but also include ... sometimes you're only reporting the thing that you saw and what you were doing when you saw it, but sometimes what you were doing before that impact things.Like we had certain situations where people were getting disconnected while joining Crusader they would just tell us "hey, I got disconnected when joining Crusader" but, you know what, it was because they were joining Arena Commander before they transitioned to Crusader and their tokens weren't getting cleared and that was the issue.So as much information as possible they can provide is usually the best for us because then we can pick through all that information and get really good steps for the developer so they can actually get a fix in.
TL;DRQA teams in Austin and the UK watch the Issue Council, and the issues that rise to the top through votes get investigated, added to the database and pushed toward fixes.
Well, me personally I'm in LA and we've got a really small team, mostly we're embedded testers, we do a lot helping the developers and stuff like that, I know our Austin and UK we've got great QA teams out there who are always in the Issue Council making sure they're checking issues. Now a lot of those issues are already written up in our database cause we ran into them during our normal sanity checks and smoke tests and sweeps and stuff like that, but the ones that do get rise to the top, those are the ones we investigate, we make sure we get in the database, we make sure we get fixes out there because we know the backers want that stuff fixed as soon as possible.
Issue Council is where the backers and all our Subscribers should be going to report any sort of issues, and then if you run into it you vote it up and the higher it goes in the chain the sooner we take a look at it and get it taken care of.
TL;DRThe environment is quite unstable, so they want "planned chaos" testing, announced to other players to avoid griefing, plus simply logging into the PTU, which gives performance and stability data they can't produce internally.
Yes. It is definitely an unstable environment. In fact more days than not it is quite unstable. There's going to be certain areas in the code you can't even look at. There's new iterations coming down. As far as causing havoc, well when you're out there in the PTU I think we want more controlled havoc...
Planned chaos that's what we want. Because doing strange and weird things is always great. Sometimes you find the best bugs by doing something that the average user would never even conceive of.Like what happens if I land this ship of this ship on top of another ship and you end up with some sort of shipception or something like that. Sometimes you can do that sort of wacky stuff and you could find some really good issues. But you don't want to do that to people who don't know what you're up to. Then it's just griefing.
Basically the point; don't be a griefer don't be dicks. Just be nice to people and if you want to test something crazy seriously just... We were talking about this, just be like...
Yeah. We got a pretty good crazy base going on! People are just like to say "Hey I want to try something wacky today does anybody else want to come here and help me out with this?" And usually we get one or two people to jump in. I know i've done it a live server a couple of times. So that's one of those things that you can do as far as havoc.As far as the PTU and how it helps us with the process. It really helps when it comes to performance, stability. We get those metrics from hundreds, thousands of users and obviously that's something we can't do internally because our Test Teams are only so big.
And also on that. People need to just log in and just be there. Cause even just logging in for a little bit here and there so that we can put the servers through their paces, the Engineers are getting really good information.The Designers are getting good information because the Designer can design something and then all of a sudden it works when you have one or two people there and you see this all the time probably. Five people there, but as soon as you hit the ninth or the tenth person everything just breaks down like crazy.
Whereever you can get those numbers it really helps. Because again the QA Teams are kept small and focused so we can you know 'laser fine accuracy' on what we need to be checking but you know...
SO point take away is: If you want to do some crazy stuff seriously just put it in chat and be like "Hey you guys i'm trying this out does anyone want to join me?"
Just don't kill someone cause we've seen things on Youtube where people are just going crazy and just ramming another ship and the person's not even taken off. And i'm like "Come on people. You know that's griefing".
Because you don't know how long that person has to play. You may be in there for a four house session trying to hit up as much stuff as you possibly can. They're only there for fifteen minutes. They got kids, they've got to change diapers, they got to get the food ready, they got a bunch of other stuff.They just want to jump in and see what we have on the next PTU patch and they don't necessarily want to help you do crazy stuff. So just clear it ahead of time if you're going to involve other people in your wacky havoc causing tests.
The other thing is. We have such a strong community we don't need stuff like that it would just bring it down. Just help each other out. Controlled and planned chaos is where it's at.
TL;DRQA time is a constantly shifting mix of regression, feature-targeted and ad-hoc testing, roughly given as percentages, and new blockers or fixes can reprioritise it from hour to hour.
Well, as we've mentioned, it's largely a moving target. You know, usually we have our set predetermined tests that we're going to do on every single build be it a smoke or a sandy check or something like that, and whenever a new feature is developed of course we're going to go through a gambit of tests that are designed around that feature, but we're also going to ad-hoc areas around that feature, you know as it comes out. I would say it's probably about 40% regression, 40% targetted, 30% ad-hoc, because you know QA, we give 110% every time.
So actually we were talking about this earlier, cause like whenever bugs come in and, so we have a little thread in production that's like hey these are being flared out, and it's mostly just production and QA that are on these threads, and it's like hey, this is the latest blocker, this is the latest thing, whether it's found through the community or whether it's found just through us internally testing and all that stuff, these are getting major priority, so as soon as these are sent out the developers get it, and once the developers fix it, a producer's basically hitting up one of the leads like you saying hey, we just fixed this, here's the change list...
Exactly, and then people will just change focus like that and just be like alright, we need this, we need this now, we need this now, so yeah, it's like building in quicksand...
Pretty much, the situation is always fluid, from one day to the next, from one hour to the next, cause all of a sudden someone will come along with something new, and that all of a sudden takes priority and everything else that you had gets shifted down a rung on the ladder and you have to get to it later.