How can we support testing better. Should we just try more crazy stuff, actually intentionally cause havoc to break things? Does the wealth of PTU data provide any new insight into the process? How does one prepare to QA the sheer number of future features while simultaneously changing the engine, adding new part, increasing the complexity. I feel this must be a moving target for QA in an unstable environment?asked by Drugh
TL;DRThe environment is quite unstable, so they want "planned chaos" testing, announced to other players to avoid griefing, plus simply logging into the PTU, which gives performance and stability data they can't produce internally.
Yes.
Yes.
Yes.
Yes. It is definitely an unstable environment. In fact more days than not it is quite unstable. There's going to be certain areas in the code you can't even look at. There's new iterations coming down. As far as causing havoc, well when you're out there in the PTU I think we want more controlled havoc...
Actually yeah. We would say controlled havoc or kind of like planned chaos.
Planned chaos.
That was the one we were kind of like..
Planned chaos that's what we want. Because doing strange and weird things is always great. Sometimes you find the best bugs by doing something that the average user would never even conceive of.Like what happens if I land this ship of this ship on top of another ship and you end up with some sort of shipception or something like that. Sometimes you can do that sort of wacky stuff and you could find some really good issues. But you don't want to do that to people who don't know what you're up to. Then it's just griefing.
Basically the point; don't be a griefer don't be dicks. Just be nice to people and if you want to test something crazy seriously just... We were talking about this, just be like...
Just ask people.
Be like "Hey guys I want to do something crazy. Is anyone down for crazy?".
Yeah!
Usually people will be like "Oh crazy? We're in, we're doing this!"
Yeah. We got a pretty good crazy base going on! People are just like to say "Hey I want to try something wacky today does anybody else want to come here and help me out with this?" And usually we get one or two people to jump in. I know i've done it a live server a couple of times. So that's one of those things that you can do as far as havoc.As far as the PTU and how it helps us with the process. It really helps when it comes to performance, stability. We get those metrics from hundreds, thousands of users and obviously that's something we can't do internally because our Test Teams are only so big.
And also on that. People need to just log in and just be there. Cause even just logging in for a little bit here and there so that we can put the servers through their paces, the Engineers are getting really good information.The Designers are getting good information because the Designer can design something and then all of a sudden it works when you have one or two people there and you see this all the time probably. Five people there, but as soon as you hit the ninth or the tenth person everything just breaks down like crazy.
Whereever you can get those numbers it really helps. Because again the QA Teams are kept small and focused so we can you know 'laser fine accuracy' on what we need to be checking but you know...
SO point take away is: If you want to do some crazy stuff seriously just put it in chat and be like "Hey you guys i'm trying this out does anyone want to join me?"
Ask permission before you start using the Glaive to lift up other ships and stuff like that.
Just don't kill someone cause we've seen things on Youtube where people are just going crazy and just ramming another ship and the person's not even taken off. And i'm like "Come on people. You know that's griefing".
Because you don't know how long that person has to play. You may be in there for a four house session trying to hit up as much stuff as you possibly can. They're only there for fifteen minutes. They got kids, they've got to change diapers, they got to get the food ready, they got a bunch of other stuff.They just want to jump in and see what we have on the next PTU patch and they don't necessarily want to help you do crazy stuff. So just clear it ahead of time if you're going to involve other people in your wacky havoc causing tests.
The other thing is. We have such a strong community we don't need stuff like that it would just bring it down. Just help each other out. Controlled and planned chaos is where it's at.