What are some of the unique infrastructure problems you deal with pushing large game builds across the world all day long?
TL;DRLatency across distant studios is the main problem, addressed with differential build copying and local caching at each studio, though faster solutions just lead to more builds and renewed demands.
Storage, speed.
Latency.
Latency is the biggest hit, because going from Austin to the UK and Frankfurt, I mean, it's a physics issue. You can't get faster than the speed of light. And so just that latency causes a lot of issues for us. There's a lot of different technologies that we've tried that don't really react well to high latency connections. And so that whole process of getting the builds out to the studios is, again, something that came from the Gamescom slash CitizenCon events. It's the precursor to the differential patcher. We developed it internally, the copy build system, where when the build is finished here in Austin, it sends it out to the other studios, but it does it in a differential packaging, where each studio is only downloading the things that have changed since the previous build on a binary level, which gives us a lot better speed at sending that kind of data across. And then once we had that completely finished and refined with the copy build, they took that and morphed it into the differential patcher.
Yeah, and bandwidth doesn't... we have a lot of bandwidth, but these distance problems aren't solved by bandwidth. Another thing I will add, for anybody who really wants to geek out: when we move builds across long distance, we cache them, much like a proxy server methodology. So that way, the first person pulls that differential, and the next person doesn't have to pull it over distance. They pull it from their local, which means it's much faster. It'd be like LAN speed for them. And so it's a combination of solutions; we just keep refining, keep improving. But what it has done is, like I've said before, every challenge that we get and we eventually solve it, we think, wow, this is going to give us some real headroom, so now they're not going to bother us about that again. Well, the minute that you give them this new solution, then suddenly, oh, well, that means that now instead of doing eight builds a day, we can do 24 builds a day. And so suddenly you're back up against the same thing, and we continue to tune.