TL;DRBuilds break because many people check in changes across interacting systems and ships, so a build system runs continuous integration, but only one QA tester exists and more QA is going to be added.
I mean, it's a ton of people all checking in stuff, so sometimes it stomps something, or sometimes you change a little thing not thinking it's going to affect this system over here, and it does. And you're just like, well, why did that happen? So everyone has to kind of work together: I'm working on this and you're working on that, and we have to make sure that we're all working on the same stuff together so that we're not stomping each other or checking in something that breaks everything. And so we have a build system that does a lot of continuous integration to make sure that whatever we check in doesn't just break everything. But we don't have a test system. So Justin is our only QA tester. Every time we make a new build, he's testing it. And there's a lot of stuff there that needs to be tested for one person.
A lot of the thing is that it's a linear process. You can't just take any element out of it without conceivably domino-effecting the whole thing. And we have a lot of people working on a lot of different things in a lot of different locations. So to coordinate all that and have everything come in at a single point is a huge amount of management. Especially the ships, because the ships involve everyone, all the way down: from the XML files, loading in the weapons and the swap-outs, to actually setting up the ship through engineering, and then art. Animations. So all that stuff is integrated in these ships. And so, again, if you find something that's messed up or there's a bug, then you fix it, and conceivably you've broken other things. It just happens that way. Things that you can't foresee happen because there's so many moving parts. It's like taking out a spark plug of your car. Well, conceivably, if you keep it that way, it's going to break your car. There's a lot of things that cascade down from that.
With the build system, once we generate a build, we create a patch of it. There's a couple different files that are associated with that patch that basically tell the patcher how to patch from where you are to where you need to go to. When we upload stuff to the CDN, we need to make sure that all of those files are there and that they're actually generated. We found that there was actually a bug in the patch building system that didn't generate the right to-from patch. So we had to, after the fact, fix that, and that's what ended up solving that issue. But we found a bug in the system. I mean, it was deep down in there; we had no idea it was there until we ran an automated promotion system that we had just developed this week.
TL;DRThe slow build may have been caused by a large added video file, but the team wasn't sure, and other unnoticed background issues, including Visual Studio toolchain bugs causing slow or hanging builds, turned out to be involved.
H
HarryRelease Testing Developer
Well, one thing we were adding was the Aurora video, the commercial, which was a fairly sizable file. And even though the patch system does a diff and doesn't try to download the whole patch every time, still, that's a big add. So we weren't sure if that big of an add of a single file was what was causing this. So we were kind of trying to do some investigation, trying to figure out, is this what's slowing it down? Is that the one thing? We weren't sure, and it turns out there were other things going on in the background that we weren't aware of at the time that were biting us.
We basically have to work around bugs in the Microsoft Visual Studio toolchain that cause issues like just hanging builds, and they look like they're progressing, but they're just progressing slowly. You're not sure what's going on.
Yeah. There are issues that Visual Studio has with parallel builds. So to speed up our build time, we do a lot of compilation in parallel. And Microsoft really doesn't support that very well, at least it seems. So we have things like that to work around. Sometimes it's just a simple restart of one of the build slaves, and sometimes we have no idea why.
I actually was going to go into the film industry, and I was looking for jobs when I came out of college. And I saw the post for Lightbox. Lightbox was actually my first job out of college.
I'd have to say the quality that they want to bring to a sort of MMO-type universe, plus the amazing community that they've already developed. That's one of the hardest things for MMOs to generate these days, is to get a community together of people who all want the same thing. The model that they're using for development is going to be awesome.
No, I do not. I originally went to school for animation. Couldn't really find any jobs when I came out of college, and then found a technical artist position, which is a little bit like animation. You do the rigging and stuff like that. And through that, that's where I learned about programming and Python, and I started making tools for the art team, and then I built up that knowledge, and now we're here, tools programmer.
Yeah, so the whole back-end system of it. Zane, obviously, he was the one who designed the front end and everything. But from the build system, which creates the CryEngine build that we distribute out to everyone, we use this software called Jenkins, and then there's a system that we use for taking that and actually making the patches and making them as small as possible, and then that's what we actually put up on the CDN for people to download. It's a lot of hand-wavy stuff and stuff goes on and it just works for you guys.
Yes, it's just not enabled. It has something to do with the third-party company that we're using for the software. There's lawsuits around it, so if they ever sort that out, it will just magically start working.
Yeah, people have asked that question a lot, and we would love to use that system, but right now it's disabled on their end, and so until the internet patent troll lawyers get finished with their stuff.