QA developers Vincent Sinatra and Randy Vasquez discuss how QA works: bug reporting tips, Issue Council use, regression versus ad-hoc testing, retesting old bugs, ship feedback threads, unstable PTU builds, and shifting development priorities. They also cover the holotable and party system.
I personally don't really like the way it's functioning currently. I don't like to look at it, don't like to test it. In fact I got some new testers now we're training up and i'm making them test it because they're the nuggets and that's their job.
I guess personally the party system right now. The party system bothers me but at the same time is I know that these are being worked on and a lot of the feedback from the fans. We've taken tonnes of feedback on things and, so we're definitely working on it. So we're pushing that soon. I cannot say when so do not ask when but if you're even thinking about asking don't even do it!
TL;DRGood bug reports are concise and clear, with steps and as much information as possible, including what you did beforehand, so QA can write good repro steps; adding a release-notes focus request is only an idea.
Yeah, I believe this is probably a QA question and Issue Council type question. So basically what we primarily want is people to be concise, clear, clear steps running through things you started doing but also include ... sometimes you're only reporting the thing that you saw and what you were doing when you saw it, but sometimes what you were doing before that impact things.Like we had certain situations where people were getting disconnected while joining Crusader they would just tell us "hey, I got disconnected when joining Crusader" but, you know what, it was because they were joining Arena Commander before they transitioned to Crusader and their tokens weren't getting cleared and that was the issue.So as much information as possible they can provide is usually the best for us because then we can pick through all that information and get really good steps for the developer so they can actually get a fix in.
TL;DRQA teams in Austin and the UK watch the Issue Council, and the issues that rise to the top through votes get investigated, added to the database and pushed toward fixes.
Well, me personally I'm in LA and we've got a really small team, mostly we're embedded testers, we do a lot helping the developers and stuff like that, I know our Austin and UK we've got great QA teams out there who are always in the Issue Council making sure they're checking issues. Now a lot of those issues are already written up in our database cause we ran into them during our normal sanity checks and smoke tests and sweeps and stuff like that, but the ones that do get rise to the top, those are the ones we investigate, we make sure we get in the database, we make sure we get fixes out there because we know the backers want that stuff fixed as soon as possible.
Issue Council is where the backers and all our Subscribers should be going to report any sort of issues, and then if you run into it you vote it up and the higher it goes in the chain the sooner we take a look at it and get it taken care of.
TL;DRThe environment is quite unstable, so they want "planned chaos" testing, announced to other players to avoid griefing, plus simply logging into the PTU, which gives performance and stability data they can't produce internally.
Yes. It is definitely an unstable environment. In fact more days than not it is quite unstable. There's going to be certain areas in the code you can't even look at. There's new iterations coming down. As far as causing havoc, well when you're out there in the PTU I think we want more controlled havoc...
Planned chaos that's what we want. Because doing strange and weird things is always great. Sometimes you find the best bugs by doing something that the average user would never even conceive of.Like what happens if I land this ship of this ship on top of another ship and you end up with some sort of shipception or something like that. Sometimes you can do that sort of wacky stuff and you could find some really good issues. But you don't want to do that to people who don't know what you're up to. Then it's just griefing.
Basically the point; don't be a griefer don't be dicks. Just be nice to people and if you want to test something crazy seriously just... We were talking about this, just be like...
Yeah. We got a pretty good crazy base going on! People are just like to say "Hey I want to try something wacky today does anybody else want to come here and help me out with this?" And usually we get one or two people to jump in. I know i've done it a live server a couple of times. So that's one of those things that you can do as far as havoc.As far as the PTU and how it helps us with the process. It really helps when it comes to performance, stability. We get those metrics from hundreds, thousands of users and obviously that's something we can't do internally because our Test Teams are only so big.
And also on that. People need to just log in and just be there. Cause even just logging in for a little bit here and there so that we can put the servers through their paces, the Engineers are getting really good information.The Designers are getting good information because the Designer can design something and then all of a sudden it works when you have one or two people there and you see this all the time probably. Five people there, but as soon as you hit the ninth or the tenth person everything just breaks down like crazy.
Whereever you can get those numbers it really helps. Because again the QA Teams are kept small and focused so we can you know 'laser fine accuracy' on what we need to be checking but you know...
SO point take away is: If you want to do some crazy stuff seriously just put it in chat and be like "Hey you guys i'm trying this out does anyone want to join me?"
Just don't kill someone cause we've seen things on Youtube where people are just going crazy and just ramming another ship and the person's not even taken off. And i'm like "Come on people. You know that's griefing".
Because you don't know how long that person has to play. You may be in there for a four house session trying to hit up as much stuff as you possibly can. They're only there for fifteen minutes. They got kids, they've got to change diapers, they got to get the food ready, they got a bunch of other stuff.They just want to jump in and see what we have on the next PTU patch and they don't necessarily want to help you do crazy stuff. So just clear it ahead of time if you're going to involve other people in your wacky havoc causing tests.
The other thing is. We have such a strong community we don't need stuff like that it would just bring it down. Just help each other out. Controlled and planned chaos is where it's at.
TL;DRQA time is a constantly shifting mix of regression, feature-targeted and ad-hoc testing, roughly given as percentages, and new blockers or fixes can reprioritise it from hour to hour.
Well, as we've mentioned, it's largely a moving target. You know, usually we have our set predetermined tests that we're going to do on every single build be it a smoke or a sandy check or something like that, and whenever a new feature is developed of course we're going to go through a gambit of tests that are designed around that feature, but we're also going to ad-hoc areas around that feature, you know as it comes out. I would say it's probably about 40% regression, 40% targetted, 30% ad-hoc, because you know QA, we give 110% every time.
So actually we were talking about this earlier, cause like whenever bugs come in and, so we have a little thread in production that's like hey these are being flared out, and it's mostly just production and QA that are on these threads, and it's like hey, this is the latest blocker, this is the latest thing, whether it's found through the community or whether it's found just through us internally testing and all that stuff, these are getting major priority, so as soon as these are sent out the developers get it, and once the developers fix it, a producer's basically hitting up one of the leads like you saying hey, we just fixed this, here's the change list...
Exactly, and then people will just change focus like that and just be like alright, we need this, we need this now, we need this now, so yeah, it's like building in quicksand...
Pretty much, the situation is always fluid, from one day to the next, from one hour to the next, cause all of a sudden someone will come along with something new, and that all of a sudden takes priority and everything else that you had gets shifted down a rung on the ladder and you have to get to it later.
TL;DROld bugs are retested mainly in areas affected by recent code changes, such as landing gear; unchanged areas aren't rechecked, but returning bugs are reopened.
This is actually a really good question. So when it comes to that, usually we focus on areas that have been affected by recent code changes, like recently Mark McCall came to me said hey, we've done a bunch of changes to landing gear, so that immediately tasks us out to start focusing, checking all those old landing gear issues to make sure that those don't recur, because you know of the changes that he just implemented, to make sure that previous fixes didn't get stomped on by new changelists and so on. So i mean, that's pretty much, we try to stay focused on the things that're in motion and are still changing, and then if something hasn't changed build after build after build, we're not necessarily going to be looking at all those old issues again. If they do return we'll reopen them, but we don't necessarily focus on them.
Yeah and that's the thing, it's like I know, once the bugs have been through their paces, they've been resolved and fixed, once it hits a new area if we see it again we'll check the database and be like, okay, this is back, and in all honesty there've been some bugs, especially with landing gear in particular that the landing gear comes back so many timesVS: That's why I brought it up, cause poor Mark has been banging his head against the wall, I swear to god some of these bugs spend time in the Lazarus pit and rise up stronger than they were before, it's just unbelievable with some of them.
TL;DRThey plan to slow down the rate of ship feedback threads to avoid overwhelming developers, and ask for concise, clear, polite, in-scope feedback split by topic, with examples; they can't act on everything instantly.
Yeah, so basically what I'm trying to do is, because it seems like we are getting so much feedback, I mean I had pages and pages and pages of feedback ...
It was very awesome. I mean the community is very passionate and they gave us just tonnes of information but some of it was information overload for developers and designers who are already working hard on new things. And they're like "When am I going to get time to parse through all this data?"So moving forward we're going to try to slow down the rate of those so that we give them more time with each ship for the most part.
Yeah, we're still breaking it down. Like I said, I was rapid fire, machinegun style getting that stuff out. And now obviously that was a little too much, so we're going to try and slow it down a little bit.
I was trying to work on that. But as far as what details, just keep things concise; keep things clear. Try to keep things within scope: we're not always looking for "Hey, please buff my favourite ship" type of stuff. You want to make sure you know exactly the type of thing we're looking for. Perhaps on my part I need to be a little more clear as to what we're asking for. Now we've even had some growing pains through some of the threads wit that ...
Yeah. That's the thing with some of the other stuff we were talking about earlier is I would love for people to break down the feedback from different sections.
So even multiple post so that way like "Hey, this is the arts posts on this". So that way you give everything for art, break it down to design, mechanics ...
... to break it down as much as possible. Yeah, balance-wise like "Alright so I'm flying a 300, I'm flying a 315, they're almost the same ship but not really". But then just breaking it down that thing like as much as ... don't be as verbose. Clear, concise: definitely. But don't be afraid to give more examples or give, at least ... because the examples you guys give, as well as just the detail you guys can give us, anything is useful.
Particularly for the 300, people were like "Hey this is how the redesign could totally work" and I'm like "Is this a resume? Is this a job application? Or is this just normal feedback?". So the detail was crazy good in some regards and then in others people are all like "Hey, give me a buff". For the most part. I want, I want ...
Hey who doesn't? But for the most part as long as it is clear, as long as it is concise. Keep things polite in the threads. If somebody says something you don't agree with, hey we're all trying to make the best game together. That's the great thing about our community is we're working together.
And honestly, the pointing out that thing, if you guys have things different, varying opinions that is perfectly fine. We actually welcome that because then we see different perspectives on how people are looking at the designs, how people are looking at the ships, the role of the ship, everything. Any feedback is good feedback.
We can't always act on everything and we're not going to be acting on it instantly so give us time because we're still working on things. We only have so many people to actually do everything.
Yeah. And I'm going to try and make it easier for the devs and designers to digest when it does come in. Try to get a top 5 bullet points and "Here's the rest of the report when you guy get around to it but these are major things that are either really bothering the community or they're really excited about getting implemented for their particular favourite ships".
TL;DRRandy imagines playing a mercenary/pirate and a miner or trader, while Vincent jokes about becoming a bounty hunter or information broker who hunts CIG staff; it's a humorous answer, not a plan.
So this is kind of a loaded question and the reason why it's loaded is because I really want to have kind of two personas when I go out there. I want to do one where I'm a mercenary/pirate and stuff like that and seriously just get in that whole clandestine thing and like be on that black market. I think that would be hilariously fun cause I never play that kind of character. Usually when I play an online game I'm either support or a healer.Of course the other one I would play would be a miner. Like when I played Eve, I was all about being a miner and a trader. So I think that would be really cool. Just doing some mining, doing some cargo stuff, that's why I like when I was given the ship the Caterpillar... I was like all about it cause it had that different mix of things that I was like, 'I can do this'...
Like you said everything is like a moving target but yeah that's pretty good. I know once the game is out and I have one mission in mind and that is to find Randy Vasquez and kill him. Pretty much. I mean, I like cutting you guys down, that's the most fun of the day...
So once we're actually out there I figure I could be some sort of CIG bounty hunter and I'm just going around letting other people know where CIG people are hanging out...
I think we did a play test what is it, a month or two ago, and basically I think it was McCall or might have been you I think was like, 'by the way guys, Vasquez is at Port Kariya'.
Next thing I know I had like four or five ships coming in and I was like, 'oh god' and then I'm trying to hide from everybody. I shot at a few people but I think I shot Jeremiah.
TL;DRThere is no fixed priority order; priorities shift daily or hourly depending on what's broken, community demand, leadership, and schedules, with different teams working on their own focuses.
Alright so. As we've stated a few times, development is always a moving target and priorities shift daily, if not sometimes hourly based on what's broken or what the fire is and stuff because we want to give you guys a stable game.
So kind of to go off the questions like. Versus ships, versus levels, versus economy, PU, mechanics, systems; we have so many people around the globe working on this game. Everyone has their own focus, everyone has their own priorities. So we have an overall global organization and then we have within that organization we have the different studios and then within that we have the each individual teams were on something where they're Designers, they're Artists, QA, Production, Engineers, Audio, Narrative.We have so many different aspects of this that everyone is just all over the place all the time. There's no ever a priority one thing over another. Like for instance if Crusader is broken, then that's kind of a level design but also design, but it's engineering, not so much Art so that then becomes a focus for the select people.
Yeah, yup so they don't change focus. Now if a ship is completely broken and a ship needs to be redone due to this area, whether they need to be redone for LOD's ship setup issues.
Then it's like the tech designers or artist, they're shifting over to help with this and sometimes they need engineer support. It is a case by case basis on really what the priority is and that is actually coming down from the community because every time the community, like they're yelling for something, they need something, they want something, we're taking that into account.Also the leadership team of CIG is also like "Hey, so." They're kind of always playing the balance game of, "Alright this is what the community wants and this is what they need, this is the leadership team wants and needs," and kind of just doing that balance game of, "This is our priority for right now," and it's our priority until something else changes and we still have like schedules that we're working on, but then...
TL;DRRandy's personal goal is to survive as a Producer and get ships out on a better timetable, and shield rework and balance, which they say is coming soon, is one feature in progress.
So this one for me myself I just want to survive as a Producer. It is so different from a Designer. I'm enjoying it, it's completely different. I'm really loving, I like the communication aspect of it.
So I love that and I love facilitation. So basically making sure the team is taken care of. going to everybody making sure they got everything they need "Hey what's up what do you guys need?"
So i'm really enjoying that but what I want to improve on though. I really want to get the ships pushed out. In a better timetable, I know we're, we push ships out and we're letting you guys know when ships are hitting. And each ship is its own little special snowflake. So it's been hard to nail that down because each ship has its own mechanics, each ship has its own features.
Then there's those features that's tied to each of the ships. That need to be pushed out stuff like that. Like you said the shields and armour balance that Callix and everybody has been working really hard on.
Yea I had to all the tests. Trying to get all those out would be really good and help give us more a final picture of what the game's going to be like, eventually. With each ship that comes down it's alright but that's just a small piece of the greater puzzle.
And that's the thing. The audit you just worked on is part of something bigger too because we're doing, we're redoing doing some of the shields and how they work. Just cause it's going to help optimise the game overall.
All that stuff is just being taken into consideration. So the design's changing a little bit. And it's changing for the better and it's changing to make sure that players are given the proper feedback they need. As well as it has all the functionality necessary things it has to have. Shields are just crazy integral to the whole thing I think as a game. Especially for dog fighting.