Mike Jones

Okay, so when we say infrastructure, we mean the things that it takes for overall operations. So that could be anything from the network that we use to connect to the internet, or the secure network that we use to connect studios to each other so that people can collaborate. The servers that we have our software running on internally to help developers do their work. Storage systems. All the security that goes around that. In corporate technology, we also have a pretty large operation that is our global licensing. This is a pretty massive undertaking, considering we have over 10,000 unique licenses that we need to maintain for every different development operation, every accounting function, any number of things. And my team is the one that not only orchestrates all of that, works those deals, makes sure that everybody has what they need, and also monitors for compliance, so that we're not ever out of compliance on a license or behind on a firmware update. Infrastructure means a lot of things that are behind the scenes. So it's the stuff that makes it work, if that helps.

You cannot say enough about these teams. In the earliest days, we just had a couple of us that would go to a show. I think my first one was in 2013. And I think we had four demo machines and a server. Not much. We even had to string our own cabling just to get internet. But it's really grown since then. And so now, it's lots of people. But basically what we do these days, we learned over time that the needs of this game, and really the vision that we're trying to present at these shows, usually exceeds the expectation of any venue. So we have had to invent ways of getting around that. And a good example of this is last year's CitizenCon, where we had certain constraints that meant that we couldn't hit the vision that we wanted to achieve. So we basically, well, I say we, my team of super expert specialists built out pretty much a whole studio, condensed it down into a traveling package so that we could take with us everything we needed, so that we didn't have to lean on anybody else. And by the time we were done, I felt really proud of what the team put together, with 150 PCs, a stack of servers. We had a lot of help though.

You're talking about, and I don't want this to get lost in the conversation, you and Ahmed and all your team, you built a microcosm of the entire Star Citizen development environment and brought it on site to LACC so that we could download right from the data store. So check-ins are replicated, and so all those 150 computers didn't have to hit the internet, a convention center internet; they could all run locally on that.

Essentially, you could think about a CitizenCon as, like you're saying, in a week, we build a sixth studio on site with networking, security, build system, distribution. This year, we had private cloud for game hosting, Wi-Fi, everything you would expect a studio to have. And then, like you said, at the end of the show, we got eight hours to tear it down and get it in a box and get it out of the building. So that's essentially what the week looks like for us.

So first of all, I had just stumbled into this. I had no aspirations. In fact, I didn't even realize that games were made. It didn't occur to me. I played them, but I never thought about what it takes to make a game and all of that. This is really a long time ago. You said where did the journey start, right? I got an opportunity to tour a video game development company, Sierra Online. That's way back. And at the end of the tour, it was really kind of blowing me away how cool this was. And I got to meet this guy, and it was a job offer, and he said, I have two openings: it's customer service and quality assurance. And I remember thinking, well, I don't even know. Is this for real? Am I actually getting a job here? I didn't even come to get one. But I guess they were desperate. And so I asked, which one pays better? And he said, well, customer service starts out better, but it's pretty hard. So then I remembered, here's what you're supposed to do, at least it's what they taught us: how do I get your job? And he said, well, if that's what you want to do, if you want to be in leadership, you need to start out in QA. So I did. And it went from there. I ran QA teams, and then I found that I had this real strong talent and interest in technology. And so I just kept focusing on that and learning as much as I could, and always going to the right conventions and shows to learn more about tech and get ahead of the curve. And then, I always enjoyed MMOs, so that was part of the journey. But what really brought me to CIG... because I had known about Wing Commander, I knew about Chris, but being on the West Coast, I wasn't as connected. And so this video came up on YouTube, and it was Chris at, I think, the first CitizenCon. And he was introducing his team and talking about what they were doing, and I was just blown away. How is this so transparent, so different than everything that I have learned in my career leading up to this? But the live stream started cutting out. They were having bandwidth problems. And I thought, good lord, whoever's in charge of IT there is probably having a really bad day. And you know what? I think it was two days later, John called me up and said, hey, do you want to come down here? We may have something we want to talk to you about. And so that was the real beginning of my journey. And I couldn't be happier, because of anything that I've done in technology or IT, it felt like it was always on rails. You didn't have a lot of creativity. Here, we're doing something no one else would try. And that means for me, I get to design solutions and work on things where we're actually encouraged to solve problems that other companies don't have. That's candy land for us guys.

So talking about what John was talking about, about how games push boundaries, that is so true. And it's one of the areas that we're always chatting about; it always comes up. Like right now, everybody's obsessed with AI. When you think about it, AI is powered by GPUs, and GPUs came to exist because of video games. So kind of a similar analogy: when 3.18 got published from PTU to Live, a lot of players did not know that the typical ratio between PTU and Live is usually 25x. The problem with 3.18, though, is we decided...

One of the problems.

One of the problems, that is true. But it starts from the fact that our universe, with all of our moons and planets and players and ships, we decided to persist everything we can as often as we can. And that is a recurring theme in Star Citizen. A lot of players might not notice that we challenge ourselves as early as possible, instead of waiting till we reach the million or the two million or the three million concurrent players. So we try to amplify the challenge all the time by avoiding any kind of hacks or shortcuts. And you see it in every single aspect, but I get to see it myself as well in my field. So I still remember when we first published, our public cloud providers reached out to us and said, what is wrong with you guys? Like, what is wrong with us? And they said that we're moving an insane amount of traffic between zones. Some of our backers are technologists, and they know when you deploy resources in the cloud, you have to split your resources across zones. You cannot sit in a single zone because zones can go down, and they do go down quite often. So the first few days, and the first few days, by the way, we were not peaking concurrency at all. A lot of backers chose not to play the first few days. They like to wait, because a lot of them learned that we hotfix a lot of issues behind the scenes. We were pushing 1.2 petabytes of hot traffic every single day, and that's between zones. That's not between every single server. We still have traffic inside the zone.

You said 1.2 petabytes?

Petabytes every day, between zones, in one region, not in all of our regions. We exist in four regions.

And that's just unheard of.

Yeah, the Amazon guys were like, we've never seen this before. Not like you've gone up from what you're used to. They were like, we don't have any other customers that are pushing this kind of traffic.

A day. And when you think about it, like social media: a human being has to create content, the content has to be uploaded, and it has to be interacted with. And most of that is happening in eventual consistency, running through compression. The fact is that every single simulator that we have... and right now, while we have instances with a reasonable amount of players, we have copies of our universe. All these copies are producing data as fast as the system can handle. And we are a competitive group of people, so none of us is telling everybody else, hey, please slow down a little bit. No, everybody's saying, give me more. And that goes all the way down to the wire, to the database. And I know a lot of players, whenever they hear the word database, they get really upset. The bit that I would like for some of them to understand is we really don't want to hold our innovation behind the scenes for too long. They get to see right now we're testing Server Meshing and server crash recovery in Tech Preview. Each and every one of us cannot wait for that to go to Live. So 3.18.0 is one of these stories where we could have stayed longer till we figured out every single little problem that could happen. But they were hungry and thirsty and really very eager to play that version of the game. So thanks to all of them for staying with us till we figured some of the kinks out. We still have a lot more to go, but we will keep pushing, because that's what we do. That's why we're all here.

Oh, I started in the basement, and it wasn't a hoot. Because by the time I started, it was closer to the end of the basement. And so it was overcrowded. And if I had to deploy a PC, you put it up over, because otherwise there's not enough room. You can't carry things around in there. Oh, the stern looks I'd get from certain devs if I bumped their chair leg or something. I came to the company right around the time that we launched the hangar module for the very first time. And we didn't really have much in the way of infrastructure. We had, I think, two tower servers. That was it. That's what ran the company. And there was no server room. I was introduced to my office, which was really a Harry Potter-style under-the-staircase office, or closet, and this is also the server room. And I have to tell you this funny part, because you're fresh in a new job, you want to make a good impression, but it's not making a very good impression on me, because they have this black contractor tarp over the top of the servers. And I wondered, why is this? Well, they pointed out that that's because the sewer pipes go over your head, and if they leak, we don't want that getting on the servers. But I didn't get a tarp. Anyway, that office, we got out of there pretty quick. I'd like to say that I was instrumental in moving the company to the new office, and literally the physical act of moving. So one of my first hires, a man named Chris, I had worked with him before, and I brought him down from Canada to help me with this move. And he did such a great job, the company asked for him to stay, and he moved his whole family down. But he and I carried the core of the company, like all of our intellectual property, source code, everything. And I'll never forget, because he would always say, slow down, we're coming up on a curve. You don't want to lose Perforce. That's the company back there. Be careful.

I do have quite a few people, and I don't want to put the whole list. I would need some time for that. The way we're set up, we do have key functions and the teams that support them. There's IT managers at each of our studio locations. Our largest location has two. Each of those are supported by a really cool team. And these are people that are really passionate about what they do, and you can tell when you interact with them. And then in Austin, we have additional specialists, from network engineering to systems engineering, administration, security, cybersecurity, even specialists. And so rather than naming them all out, I'd like to call them out in that way.

The people who manage all the licenses and stuff.

Well, yeah, I already said his name. That's Chris.

You mentioned the licenses earlier. We build tools to let these softwares talk to each other. But I imagine what that must be like when someone's like, hey, we need to add another piece of software.

We're a really interesting company even in the game space, because we are equal parts game company, technology company, and a production facility. I mean, we're in a beautiful set here that you operate in. You have very specialized needs that are very different from Ahmed's team or the average game designer or the finance department or whatever. So as a corp tech publishing organization, we're supporting everything from mocap to legal document needs to cybersecurity to all kinds of stuff. Like we've said, it makes it really interesting because no two days are ever the same.

Well, yeah, it takes a while to get used to. It kind of just doesn't stop. We have a little window, for me, in my time zone, it's between about midnight and two o'clock when it's quiet. And the other thing is, these guys here talk about the things that they do, but a lot of what we do is responding to emergent issues, or system down or communication failure or some kind of a server not responsive, things like this. And we don't have an internal NOC yet, like a 24-7 team that responds to those types of things. So for now, I'm it. And so if anything happens in one of the other time zones, or maybe they can't publish a build or they can't see a particular network share or something, if it's bad enough, I'll get the call. And then that means, because I have a pretty good idea of who could solve what and in what time zone the quickest, I start dialing out and waking people up. And so that's kind of how we handle this around-the-clock type thing. And I think someday we're going to expand that. But it takes a lot of people to be able to man a 24-7. So the guys here, this team in particular really does spend a lot of extra hours. We have our normal time that we come into the office, but I can tell you that there's not very many times during the day, even on weekends, that I can't send out a message and one or more of these guys will be right there on it and able to respond to an issue.

It used to be all me. Now everybody contributes to this. We set up a new build for CitizenCon. We try to keep our builds really current, like the really high-end gaming PC builds. Chris is pretty specific about making sure that we have really high-end stuff because we're developing for gamers. So anyway, for CitizenCon, we wanted to put our best foot forward, and so we ended up going with the 8086K, and that thing has a 5 gigahertz turbo boost. The build came out great, so it's a six core, five gigahertz. I think we have 32 gigs of RAM. It's an ASUS TUF motherboard. And it's, of course, got our all-in-one water cooling solution. And Intel gave us an Optane drive, so that made the boot really fast. Anyway, the thing is, that build came out so good that as developers started coming in, in prep the day before the show, they show up and sit down, we'll set up a few machines for them to do their last-minute preps and things. And they're saying, whoa, what is this thing? You have to give me the spec. We'll send it to Hassan. I want a machine like this, built exactly like this for when I get back, because the game hasn't run better. In fact, with that build, we haven't even really had to worry about the super high end having the greatest video card. So we have a good mix of a lot of different video cards because the build is just really stable underneath. Of course, the game has improved. So I hope that answers the question of our kind of basic spec. I will say, though, that we don't do all machines that way. So we do have a good mix of the different video cards. We try to have a decent smattering of different motherboards and configurations so that we can have some semblance of compatibility testing, especially within the QA department. We'll put a lot of different configurations together for QA. And oftentimes they'll ask us, can you swap this out? Because we're getting a lot of community complaints about this particular setup. We'd like to set that up so we can verify and get better information to the devs.

Storage, speed.

Latency.

Latency is the biggest hit, because going from Austin to the UK and Frankfurt, I mean, it's a physics issue. You can't get faster than the speed of light. And so just that latency causes a lot of issues for us. There's a lot of different technologies that we've tried that don't really react well to high latency connections. And so that whole process of getting the builds out to the studios is, again, something that came from the Gamescom slash CitizenCon events. It's the precursor to the differential patcher. We developed it internally, the copy build system, where when the build is finished here in Austin, it sends it out to the other studios, but it does it in a differential packaging, where each studio is only downloading the things that have changed since the previous build on a binary level, which gives us a lot better speed at sending that kind of data across. And then once we had that completely finished and refined with the copy build, they took that and morphed it into the differential patcher.

Yeah, and bandwidth doesn't... we have a lot of bandwidth, but these distance problems aren't solved by bandwidth. Another thing I will add, for anybody who really wants to geek out: when we move builds across long distance, we cache them, much like a proxy server methodology. So that way, the first person pulls that differential, and the next person doesn't have to pull it over distance. They pull it from their local, which means it's much faster. It'd be like LAN speed for them. And so it's a combination of solutions; we just keep refining, keep improving. But what it has done is, like I've said before, every challenge that we get and we eventually solve it, we think, wow, this is going to give us some real headroom, so now they're not going to bother us about that again. Well, the minute that you give them this new solution, then suddenly, oh, well, that means that now instead of doing eight builds a day, we can do 24 builds a day. And so suddenly you're back up against the same thing, and we continue to tune.

Well, I guess when I got here, I was the only one. The company was much smaller. We were just building out the L.A. studio, and shortly after that, we got the U.K. office. In those early days, it was not easy. We didn't have very much. And like I said, it was just me. But of course, there were a lot less people. So we put it around, and sometimes we'd hire some contractors to help out a little bit, but we very quickly grew the team. One of the things that we figured out was that with a company growing this fast, we have to get real with our infrastructure and the hardware, especially where it comes to network connectivity, connecting the studios, firewalls, switches, things of this nature, to really build out a robust network. It didn't take very long for the company to not be very happy about downtimes. So what we started doing was we built a standardized basic package of the minimum gear that you need for a studio to hook in, so that you could make it part of the mesh. And so when other studios started coming online, it was easier then to say, here's your basis, what are your other requirements, we'll add to it. And so Paul was a big help with that, with his experience in network engineering and design. Mike helped a lot, kind of a coordinated effort as everybody came together. But then what that meant was we started establishing these systems. So it really becomes a matter of delivery time, planning the time to get the equipment in. And then another big one is ISPs, especially when you talk about different countries. Sometimes it takes a really long time to establish that fiber network presence. But we usually have everything lined up and pre-programmed, so once we hook in that network, they pretty much come online. And nowadays, with the systems that we have in place that Paul has been talking about a little bit, it's a lot easier than that even. I mean, we even have smaller subset packages. So, this is an extreme example, but you can even set up a CitizenCon as a mini studio. So we'll hook up the security tunnels, we'll hook up all the networks, so that we can deliver the build packages, but also any other kind of normal communication. It really helps the devs that way, so that when they sit down, they can just log in. They don't have to worry about things like VPN clients and this and that. So I guess that gives you kind of an idea of how we do it, at least in overview. Of course, if you let me, I can go into a lot more detail.

Yeah, that crash cart kind of thing that we did, that's what the Derby studio started with. We had come up with this design for one of the public events, where we could just set up anywhere, boom, here's a quick mini studio. And the Derby studio started off pretty much just as a remote mocap shoot, and so we sent it down there. And when they actually leased the space and turned it into a studio, we had very little to do to upgrade it to make it part of the long-term group. It was basically buying an extra infrastructure switch, and that was it.

Yes. We need two network engineers, two senior systems engineers, and then most likely another help desk here or there.

I would add that those are the ones that we're about to post, or have had posted and then took them down for a while, but we're pretty much open to anything. And so I think anybody who's interested in this kind of stuff, especially with a game company... it's different. IT support in this company is different than any other game company I've been in. And of course, it's way different than your typical IT in your typical institutions, banks, insurance companies, things like that. This is fast paced. When he said fire hose, that's a silly thing, but you really do learn fast, and the challenges are different. But yeah, everything from all skill sets, junior to senior. One of our system engineer positions should have a focus on Linux. And this is just because a lot of our support infrastructure and servers, a good 50% of it, runs on Linux. So I just want to encourage everybody, if you've got an interest, you should send in that resume. We're looking at all of them. Obviously we can't take everybody, but it can't hurt to try.

So Perforce is a revision control system. It's a document repository in essence, where if you think of it like a library, you check out a file to edit and then you check that back in. And so it keeps track of all the revisions. So if you check out a file and update a paragraph and then you resubmit it, then that's version two of that. And if you decide, I really liked that previous version, you can check out revision one and make that the current active. So in development, of course, instead of documents and stories, it's all code and scripts and artwork and stuff like that. The main reason we use Perforce, well, it's a really robust system, but most of our senior developers... I've been in the industry for 25 years now, and back then, Perforce was by far the best revision control you could use in games. SourceSafe and CVS were some of the other ones that were problematic, let's say. And so I think that's one of the reasons why Perforce has remained one of the de facto standards in the game industry, at least, for revision control. Git is absolutely a very good system for that, but we're fully embedded in Perforce, and trying to convert to Git would be a very monumental task.

The developers being in the game industry and Perforce being the standard, it would be a big jump for them to make that transition as well.

Yeah, there's a knowledge level.

We do have specific small applications where we use other source control systems like Git. So it's like outside-the-game reasons, right? You may use it for controlling configurations on certain infrastructure and things. So we'll specialize that use case.

Yeah, we don't exclude anything. If Git is the best solution for what we're trying to do, we use Git. Perforce is the primary repository for source code, and everything we need to build the game goes in through Perforce.

I think that one of the biggest things for me is the amount of support, the amount of infrastructure, the amount of work that this IT team does with the small number of people and the size of this company. To me, that's probably one of the biggest achievements. There's a lot of little things that we could talk about, but we run on a really thin crew, and that's because we have very experienced people. I'm talking about a culmination of a lot of different experience, not just game companies, but from all over the place. And so it means that we can work so well together. What do they say? You start something and the other person just picks it up like they could read your mind or whatever. We're constantly doing things, making process and policy so that users are able to support themselves when possible. If they can't, they come to us, and then we can be quick about it and solve a problem. So I think, to me, that's a big deal.

Well, for me, my most memorable experience was in Australia, at PAX Australia, because it was when I really grokked the scope of what we're doing. I mean, I had worked with Chris back in the Origin days on Wing Commander, and I've been in the game industry a long time. So I'm not a stranger to there being fans of a game. But when I got to the Australia venue, they said, okay, well, here's your team of volunteers that are going to help you set up. And I'm like, well, what do you mean? It's like, well, these are backers that have volunteered to help with the operations of setting up and whatnot. And I was like, what do you... I mean, they've already bought the game and they're volunteering their time and energy and everything. And they're like, yeah. And the enthusiasm that they put in, I've worked in places where paid employees didn't have the amount of enthusiasm that they had. And just the scope of it, that this is a game that people... it's not just a game to them. And every event we go to, there's volunteers and even just backers that come up and say, this is going to be an awesome thing. And then also the amount of openness that we have, since we are crowdfunded. We are very open with all of our development, and that was a big change for me, because traditionally in the game industry, you may be working on a game for two years and nobody's heard of it because the company hasn't even announced that they're working on it yet, and they want to have a big splash at the Christmas sale time of here's this game that we've been working on. Whereas here, we're just open to everything. Having backers come on studio tours, it's just all these things that were never a factor in my career previously. It's just awe-inspiring sometimes.

For me, my most exciting day was... basically, I was a backer of the game. I heard of the game, I immediately backed because the idea was just phenomenal. And I knew Chris from back in the early days when he made Wing Commander, as a fan, not as a person. And I put my resume in and I was like, there's almost no way that I'm going to get picked out of, I'm sure, hundreds of people that are applying for it. And I got a phone call from John Erskine and we had a great talk, and I was like, wow, he's really interested. And he said, well, I want you to talk to our new technical guy, Mike Jones. I'm like, oh yeah, I saw Mike Jones on the show. And so he set up the call for the next day. And we started off with a little bit of a technical talk about my experience at Carl's Jr. headquarters and stuff like that. But within three minutes, I knew Mike Jones was all in on the project, because we left all the technical stuff behind and we just started talking about how passionate we are for what this vision was and what we could do to make it. And I was expecting a little bit more corporate, organized talk, but what really fired me up was Mike's passion for the game. And I was like, oh my God, he's just like me. He's a total fan. And his passion is so powerful that this guy is going to make this game happen regardless. And I got just so fired up, and I hoped I made it through the process. It was a long process because there was a lot going on. LA was just kind of getting up and running. Mike had only been there for about three or four months, and he was reshaping the IT department to try to expand it. And he called me up and said that I was hired, and it just blew my mind. To go from somebody who was so excited about a project, and then find somebody who was that passionate and dedicated to the dream of the game that we're making running the entire technical part, and that he wanted me along for the ride. At that point I said, regardless, I give you a solemn promise that I'm going to work here and give my all until this game is finished. That's what we're doing to this day, five years later. And we're going to keep going until we're done. So that's why I'm here.