So we're a bit involved in entities dropping in and out. It's a lot of traveling around, but it's also finding all the different entities, how we can spawn them, move them, change the state for one player without changing it for another, bringing the players back together, and checking the states of everything match up. And we're looking for all the edge cases as well at the moment. Because everything's fine apart from stalling when you start loading up, say, Levski. There's a stall as it brings Levski in. Sorry, I'm losing my train of thought.
TL;DRQA checks that a player's client shows the correct state after leaving and returning, and is testing server load using headless "zombie" clients spread around the solar system, given the 50-player limit.
We'll be looking for things like, as you're dogfighting, you've got a mate in your cargo bay throwing a cargo crate out the back and you're blowing up Gordon at the same time. Another player, who, when he left, you were just dogfighting happily, comes back and everything's different. Does his client show everything as it should be? Is the cargo crate there in space? Is your cargo bay open? Is Gordon alive or dead? A lot of different states. And one of the things that I've been looking at myself this week is we need not just two people doing it. Two people seem to be quite easy to keep track of. What happens with the server when there are people or a group of people, not just out in space dogfighting, but you've got someone out in space, people off in Hurston? So we've been using headless clients, which are just low-weight clients that don't actually render the game, but we can give them basic commands. So we're filling the server...
Yeah, putting zombies in. They do basic things. They'll get out of their bed at Olisar. They'll use console commands to zip around the solar system. So we're seeing what's happening with the server now. What's the load like when there's someone everywhere, or nearly everywhere, because the player limit is 50 and it's a big solar system. So you can't get everywhere with 50 people right now.
I overheard someone today use the phrase 'truly amazing'. I don't want to oversell it. I know the backers have complaints about performance, quite legitimately. It's not at the state anyone wants it to be. But we're seeing really raw builds each day. Typically our performance can be pretty terrible. Everyone's reporting improvement. Some people are reporting drastic improvements. It's just a lot nicer to test when you're hitting a really nice smooth frame rate.
What we're seeing now is when you're in Port Olisar, you're not getting updates from Levski, Grim HEX, anything like that. So performance is great. You go to Lorville, your frame rate is still dropping very low because you're still loading Lorville and everything that's in it.
TL;DRSo far it speeds up loading the game for developers, makes the much bigger map loadable on machines with less memory, and lets more game clients run on a single test machine.
Current improvements that we've already seen? From a development point of view, it takes me less time to load the game, which is nice. So if I'm wanting to test very simple stuff, the PU takes less time for me to load because it doesn't have to load all the stuff that's out of range.
It actually makes it possible to load the game as well, because the map is so much bigger now. Not everyone here has quite large memory machines, so some of them can't actually load the full map because it's that big. So they need bind culling on. Well, we've got two maps. We've got a cut-down map and the full map. They either have to work on the cut-down map, or if they want to load the full map, they have to turn bind culling on to be able to actually get it in, working on their mission. It's that much bigger.
The reduced memory is helping us as well. I mentioned earlier about filling a server up, which is effectively using one tester's time across multiple machines. The less memory the game is using, the more game clients we can put on a single machine.
I think the fact that it touches everything means we need to test everything. And it's not just testing everything with bind culling, because when you find an issue on, say, the bind culling server, is it a bind culling issue? Sometimes you can't tell just from the symptoms of the issue alone. You go back to a regular server, you need to reproduce it there. So you're not just testing everything, you're testing everything twice, with it affecting things coming and going. These aren't tests you can usually do by yourself. You need two testers, preferably as many testers as you can get together. And we only have a finite amount, regardless of how many zombies I can... It's been a massive strain on our resources, getting everybody we need to be where we need to be, covering as many things as we can.
See, now you've said funny, all I can think of is crashes. There's been fun ones where you're trying to catch an AI who's decided to run away from you. I think this was earlier on. So he's in range, flying away, and you had an issue where he would reach outside of your streaming window and he'd stop. So then you'd catch up, and eventually you'd get the updates from this AI and he'd fly a bit further, because actually he's even further. So you're constantly having this little stop-start race against the guy, trying to catch up, and you don't even know where he really is.
TL;DRQA says the only drawback is testing time, while Gordon adds that programmers can no longer assume every entity exists, so features must be designed with streaming in and out in mind.
You mentioned earlier some of the challenges involved, like the scanning, having to leave the markers when something is culled so that people can still see it.
Yeah, I don't want to generalise, but most gameplay programmers are quite happy to just assume everything exists all the time. I'd be very happy if I just had every single entity on the client at the same time, because it'd make my job a lot easier, whereas now, I've got to make sure every entity I'm accessing exists, and if it stops existing while I'm talking to it, what do I do? Lots and lots of more technical challenges. It just requires you to think about things, not that you don't think about things, but you've got to be very aware of the implications that a lot of stuff might stream out. If I'm designing this feature, will it work when not everyone's here? Do I need to move it to exist on the server rather than the client? And tweak things to make them work.
TL;DRThe speakers stress that it took about three years, is a company-wide effort touching every part of the game, will keep being improved, and will cause edge-case breakage, so testers should submit crashes.
It's been, as you said earlier, three years since we started work on various parts of it. Although not everyone's been working on it flat out for those three years, it's taken a massive amount of time, a lot of changes, tons of people. It's not down to just us in this room or our teams in particular. It's a company-wide effort. It's been a long road and it's hopefully worth it when we finally get there.
Yeah, I think just to reiterate that it's touched every single element of the game, and it is still currently touching every single element of the game. Like I said, it's a collection of features, but there's never been a feature where there have been as many emails that have come out to absolutely everyone going, is this going to break anything for you? It's needed everyone to pitch in. I see a whole lot more 'BCC everyone' on OCS threads than on just about anything else. And I think as well, as Clive said, it's not been a short road. Before even Object Container Streaming, there had to be object containers themselves, and that started as I was starting, so that's three and a half years ago. So that was kind of the very beginning of it, and this has obviously been on the plan since then, and it's just taken three and a half years of tech and improvements.
No, this is the kind of thing that's going to be ongoing forever, probably. There's always going to be things to improve and more stuff that is added to it. The server streaming is obviously the next major step, and then there's stuff beyond that as well.
Just to reiterate that it touches everything, and from the QA standpoint... everywhere. Yeah, it touches everything. There's going to be a lot of edge cases that, with our department, as cool as everyone is, there's only so many of us. There's only so many Evocati, there's only so many PTU testers as well, and there are a lot of things that are going to break, I think. Please submit to the Issue Council and always submit your crashes.