Paul Vaden

So a typical day is basically: get in. If I've not got a phone call saying something's broken before I get in, go through tickets, make sure anything that's a blocker or critical gets completed. Get that done first. Just making sure the other guys in my team have things to do and basically making sure they're doing a good job.

Day to day, I get in, check to see if anything's alerting, and then go through the calendar, what's coming up for the day. Sometimes I'll have a call with Hassan to kind of sync up with what's going on in Europe. I still do quite a bit of the network engineering, so running maintenance on our Perforce servers and that kind of stuff as well.

You said alerting. What kind of alerts are we talking about?

Usually it's disk full alerts. The amount of data that we consume is just ridiculous. And so we're constantly adding storage to servers because it just grows so quickly.

Here in L.A., we're pretty much downstream from everybody else, because Germany pretty much starts off the day and they get a full work day, and then the U.K., and they're almost done with their day, and Austin's pretty much in the middle of their day. So we try to pick up on a lot of the tasks that they've already started or they need us to do, and then we kind of close out the evening. So sometimes some of our work comes down as we're trying to reorganize and reorient for Germany to pick up the next day. So we're helping the artist teams and the engineering teams basically be ready to hand off all their materials and the projects they're working on to Germany and start the day over. And what people don't understand is that CIG is a 24-hour outfit. I mean, everybody in this room is pretty much on call 24-7. So if something happens, if it's something we can do remote, we do it from there. And if not, then we have to drive into the office and do it. And then outside of that, the one unique group that we house here is pretty much GVP and the narrative and, like this show. So we try to give Jared additional support if he needs it, and also the GVP team that runs a lot of the other content that's coming up. So a lot of times, what these guys need finished up, we try to finish it for them for the next day.

So my first experience at Cloud Imperium was, I started about a month before we did the PAX Australia event. And part of my role is that I am at all of the public events. So the phrase we like to use here is drinking from the fire hose. So my introduction to CIG was, okay, get everything ready and ship it all off to Australia to do an event out there. And I was like, what's an event? But it's probably one of the most rewarding experiences in my career, because each event has its own separate challenges, and just being able to show up and know that I've got one of the best support teams available. To describe the situation: we generally get access to the event space the day before. And so we have to come in and set up. Recently we've been doing a whole lot of multiplayer demos, so that's a 24 PC LAN. Plus we've got demo machines for player demos, but also the onstage demo machines. We have to network all of that. And then we always have to have a method of getting the latest updated builds. One of the, I don't know, insider information about the events is we're usually getting new builds right up to about an hour before the event starts. I don't think that's a secret. Because the events aren't punctual. We don't normally start exactly on time, but that's because...

We did last year. And we ended on time, too. First time we ever ended on time. Sorry, go ahead.

Yeah, and the reason we do that is it's a very clear message from Chris that we want to show the best stuff that we can. And so the whole team, the whole company really kills itself to get everything in and working so that we can show off the stuff to the fans and the backers. And so there's been a couple of shows where they were literally typing up the PowerPoint five minutes before he was going on.

Which was 30 minutes after he was supposed to be.

Exactly. And so it's just very frenetic. But having the team that we have, it allows me a lot of confidence to know that when our event planner says, hey, you're only going to have a day ahead of time to set up at this venue, I'm like, okay, that's fine. Because I know that the team on this call has all done it before, plus many of our other IT staffers have all been part of the events at some point. And it's like a commando strike team. We come in, lay everything out, get it all working, and then just start pulling builds and getting everybody going.

Storage, speed.

Latency.

Latency is the biggest hit, because going from Austin to the UK and Frankfurt, I mean, it's a physics issue. You can't get faster than the speed of light. And so just that latency causes a lot of issues for us. There's a lot of different technologies that we've tried that don't really react well to high latency connections. And so that whole process of getting the builds out to the studios is, again, something that came from the Gamescom slash CitizenCon events. It's the precursor to the differential patcher. We developed it internally, the copy build system, where when the build is finished here in Austin, it sends it out to the other studios, but it does it in a differential packaging, where each studio is only downloading the things that have changed since the previous build on a binary level, which gives us a lot better speed at sending that kind of data across. And then once we had that completely finished and refined with the copy build, they took that and morphed it into the differential patcher.

Yeah, and bandwidth doesn't... we have a lot of bandwidth, but these distance problems aren't solved by bandwidth. Another thing I will add, for anybody who really wants to geek out: when we move builds across long distance, we cache them, much like a proxy server methodology. So that way, the first person pulls that differential, and the next person doesn't have to pull it over distance. They pull it from their local, which means it's much faster. It'd be like LAN speed for them. And so it's a combination of solutions; we just keep refining, keep improving. But what it has done is, like I've said before, every challenge that we get and we eventually solve it, we think, wow, this is going to give us some real headroom, so now they're not going to bother us about that again. Well, the minute that you give them this new solution, then suddenly, oh, well, that means that now instead of doing eight builds a day, we can do 24 builds a day. And so suddenly you're back up against the same thing, and we continue to tune.

I sometimes ask myself the same question. How do I manage it? But no, I guess, as you know, the staff here are very capable of self-managing. But with 250-odd people, it does come down to the wire with managing or trying to fix different problems at the same time for 250-odd people. But the team is growing. We're up to three people in the UK, in the Wilmslow studio, and we've got two, or one and a half, in the Frankfurt studio, and we're ever expanding. But yeah, as things develop more and people ask for more requests, I'm asking for more information. Yeah, it's coming to the point... I've got white hair now. I thought I wouldn't get any white hair. I'd go on to grey, but my hair's going white. It's good fun, don't get me wrong, and that pressure and that challenge keeps the fire alight.

To toot Hassan's horn for him, that studio has really had explosive growth in the last year, year and a half. And it kind of caught us by surprise. Up until last week, Hassan was supporting that whole studio with just him and our other IT guy there, Kyle. It's difficult to describe the workload that they have had to deal with, but we've gotten an extra help desk guy in there, Nathan, and we've got two more slots open out there to try to get him a lot more support, because 250 people for three support staff is quite a bit.

And the thing is, it's ever growing as well.

Well, I guess when I got here, I was the only one. The company was much smaller. We were just building out the L.A. studio, and shortly after that, we got the U.K. office. In those early days, it was not easy. We didn't have very much. And like I said, it was just me. But of course, there were a lot less people. So we put it around, and sometimes we'd hire some contractors to help out a little bit, but we very quickly grew the team. One of the things that we figured out was that with a company growing this fast, we have to get real with our infrastructure and the hardware, especially where it comes to network connectivity, connecting the studios, firewalls, switches, things of this nature, to really build out a robust network. It didn't take very long for the company to not be very happy about downtimes. So what we started doing was we built a standardized basic package of the minimum gear that you need for a studio to hook in, so that you could make it part of the mesh. And so when other studios started coming online, it was easier then to say, here's your basis, what are your other requirements, we'll add to it. And so Paul was a big help with that, with his experience in network engineering and design. Mike helped a lot, kind of a coordinated effort as everybody came together. But then what that meant was we started establishing these systems. So it really becomes a matter of delivery time, planning the time to get the equipment in. And then another big one is ISPs, especially when you talk about different countries. Sometimes it takes a really long time to establish that fiber network presence. But we usually have everything lined up and pre-programmed, so once we hook in that network, they pretty much come online. And nowadays, with the systems that we have in place that Paul has been talking about a little bit, it's a lot easier than that even. I mean, we even have smaller subset packages. So, this is an extreme example, but you can even set up a CitizenCon as a mini studio. So we'll hook up the security tunnels, we'll hook up all the networks, so that we can deliver the build packages, but also any other kind of normal communication. It really helps the devs that way, so that when they sit down, they can just log in. They don't have to worry about things like VPN clients and this and that. So I guess that gives you kind of an idea of how we do it, at least in overview. Of course, if you let me, I can go into a lot more detail.

Yeah, that crash cart kind of thing that we did, that's what the Derby studio started with. We had come up with this design for one of the public events, where we could just set up anywhere, boom, here's a quick mini studio. And the Derby studio started off pretty much just as a remote mocap shoot, and so we sent it down there. And when they actually leased the space and turned it into a studio, we had very little to do to upgrade it to make it part of the long-term group. It was basically buying an extra infrastructure switch, and that was it.

Like you said, L.A. with GVP. I think the U.K. is just purely a development studio, where we've got a team of artists, a team of programmers, just concentrating on developing the game most of the time. Whereas Austin is the backend; they deal with the build system and whatnot. So basically the whole infrastructure starts from Austin. And uniquely, the UK studio is the one where we've got the most developers developing the games.

Yeah, we started off with each studio kind of having its own niche, where LA was video, Austin was core operations, and DE was mainly game engine and internals, and UK was mainly art and some design and some programming. But just the scale of the UK office now, I mean, they do everything there. Mocap was the other thing that was mainly UK. But yeah, that's not so much the case anymore, where each studio has people in every discipline of the game-making process. But I would say just the fact that most of the tech comes out of Austin, where we've got the build system, we've got our source control system, all of that's housed here in Austin.

Yes. We need two network engineers, two senior systems engineers, and then most likely another help desk here or there.

I would add that those are the ones that we're about to post, or have had posted and then took them down for a while, but we're pretty much open to anything. And so I think anybody who's interested in this kind of stuff, especially with a game company... it's different. IT support in this company is different than any other game company I've been in. And of course, it's way different than your typical IT in your typical institutions, banks, insurance companies, things like that. This is fast paced. When he said fire hose, that's a silly thing, but you really do learn fast, and the challenges are different. But yeah, everything from all skill sets, junior to senior. One of our system engineer positions should have a focus on Linux. And this is just because a lot of our support infrastructure and servers, a good 50% of it, runs on Linux. So I just want to encourage everybody, if you've got an interest, you should send in that resume. We're looking at all of them. Obviously we can't take everybody, but it can't hurt to try.

So Perforce is a revision control system. It's a document repository in essence, where if you think of it like a library, you check out a file to edit and then you check that back in. And so it keeps track of all the revisions. So if you check out a file and update a paragraph and then you resubmit it, then that's version two of that. And if you decide, I really liked that previous version, you can check out revision one and make that the current active. So in development, of course, instead of documents and stories, it's all code and scripts and artwork and stuff like that. The main reason we use Perforce, well, it's a really robust system, but most of our senior developers... I've been in the industry for 25 years now, and back then, Perforce was by far the best revision control you could use in games. SourceSafe and CVS were some of the other ones that were problematic, let's say. And so I think that's one of the reasons why Perforce has remained one of the de facto standards in the game industry, at least, for revision control. Git is absolutely a very good system for that, but we're fully embedded in Perforce, and trying to convert to Git would be a very monumental task.

The developers being in the game industry and Perforce being the standard, it would be a big jump for them to make that transition as well.

Yeah, there's a knowledge level.

We do have specific small applications where we use other source control systems like Git. So it's like outside-the-game reasons, right? You may use it for controlling configurations on certain infrastructure and things. So we'll specialize that use case.

Yeah, we don't exclude anything. If Git is the best solution for what we're trying to do, we use Git. Perforce is the primary repository for source code, and everything we need to build the game goes in through Perforce.

I think that one of the biggest things for me is the amount of support, the amount of infrastructure, the amount of work that this IT team does with the small number of people and the size of this company. To me, that's probably one of the biggest achievements. There's a lot of little things that we could talk about, but we run on a really thin crew, and that's because we have very experienced people. I'm talking about a culmination of a lot of different experience, not just game companies, but from all over the place. And so it means that we can work so well together. What do they say? You start something and the other person just picks it up like they could read your mind or whatever. We're constantly doing things, making process and policy so that users are able to support themselves when possible. If they can't, they come to us, and then we can be quick about it and solve a problem. So I think, to me, that's a big deal.

Well, for me, my most memorable experience was in Australia, at PAX Australia, because it was when I really grokked the scope of what we're doing. I mean, I had worked with Chris back in the Origin days on Wing Commander, and I've been in the game industry a long time. So I'm not a stranger to there being fans of a game. But when I got to the Australia venue, they said, okay, well, here's your team of volunteers that are going to help you set up. And I'm like, well, what do you mean? It's like, well, these are backers that have volunteered to help with the operations of setting up and whatnot. And I was like, what do you... I mean, they've already bought the game and they're volunteering their time and energy and everything. And they're like, yeah. And the enthusiasm that they put in, I've worked in places where paid employees didn't have the amount of enthusiasm that they had. And just the scope of it, that this is a game that people... it's not just a game to them. And every event we go to, there's volunteers and even just backers that come up and say, this is going to be an awesome thing. And then also the amount of openness that we have, since we are crowdfunded. We are very open with all of our development, and that was a big change for me, because traditionally in the game industry, you may be working on a game for two years and nobody's heard of it because the company hasn't even announced that they're working on it yet, and they want to have a big splash at the Christmas sale time of here's this game that we've been working on. Whereas here, we're just open to everything. Having backers come on studio tours, it's just all these things that were never a factor in my career previously. It's just awe-inspiring sometimes.

For me, my most exciting day was... basically, I was a backer of the game. I heard of the game, I immediately backed because the idea was just phenomenal. And I knew Chris from back in the early days when he made Wing Commander, as a fan, not as a person. And I put my resume in and I was like, there's almost no way that I'm going to get picked out of, I'm sure, hundreds of people that are applying for it. And I got a phone call from John Erskine and we had a great talk, and I was like, wow, he's really interested. And he said, well, I want you to talk to our new technical guy, Mike Jones. I'm like, oh yeah, I saw Mike Jones on the show. And so he set up the call for the next day. And we started off with a little bit of a technical talk about my experience at Carl's Jr. headquarters and stuff like that. But within three minutes, I knew Mike Jones was all in on the project, because we left all the technical stuff behind and we just started talking about how passionate we are for what this vision was and what we could do to make it. And I was expecting a little bit more corporate, organized talk, but what really fired me up was Mike's passion for the game. And I was like, oh my God, he's just like me. He's a total fan. And his passion is so powerful that this guy is going to make this game happen regardless. And I got just so fired up, and I hoped I made it through the process. It was a long process because there was a lot going on. LA was just kind of getting up and running. Mike had only been there for about three or four months, and he was reshaping the IT department to try to expand it. And he called me up and said that I was hired, and it just blew my mind. To go from somebody who was so excited about a project, and then find somebody who was that passionate and dedicated to the dream of the game that we're making running the entire technical part, and that he wanted me along for the ride. At that point I said, regardless, I give you a solemn promise that I'm going to work here and give my all until this game is finished. That's what we're doing to this day, five years later. And we're going to keep going until we're done. So that's why I'm here.