John Erskine

I'm John Erskine. I'm the head of the publishing organization for CIG. I've been with the project since essentially the very beginning of the project. I joined in the beginning of 2013. At CIG, publishing encompasses everything technical that's not related to the game engine itself. So essentially, after a developer checks their code into our source control, Perforce, then it goes into our build system, and the build system is the start of the publishing pipeline. So when a build gets generated, it goes through a series of automated tests, and it eventually goes to a QA department. It eventually gets published to some kind of environment, whether it's an internal development environment, a staging environment, or maybe a test environment. And that's all part of the publishing technology group. Then we have also our live QA team and ultimately our customer support team that are supporting the game once it's either on PTU or on live. People have questions about the game afterwards, problems with their purchases or all that sort of stuff. So all of those variety of things at CIG we call publishing in one form or another. So that's my group. There's a whole team of us. We have people all over the world. Principally we're based in Austin, where I'm based, or at least most of the leadership team is based there. We've also had a really long and fruitful relationship with the platform team at Turbulent, which is based in Montreal. So I head up a lot of the functions there related to the e-commerce and the sort of technology behind the platform there. So we're dispersed all over the place and do like a dozen different things, and it's pretty exciting.

I always saw publishing, at least for CIG, as everything that was outside, underneath, and around the games itself. Basically, we've got our developers and our five studios around the world, and you and your team basically make it possible for them to do their stuff. Whether that's providing the IT infrastructure for them to do their work, to the publishing infrastructure to, okay, now you've done your work, let's get it into the hands of the backers, to the platform stuff, like you said, to track pledges and entitlements and stuff. Really, you run this entire cloud that just sits around the game itself.

That's a good description, because it's sort of like the game. We have entire technology teams devoted to the game itself, a lot of the services that run the game, the game engine and everything that happens there. But I like your description that we sort of surround all of that, maybe in a warm fuzzy blanket, and make it possible for people outside the company ultimately to play the game.

You cannot say enough about these teams. In the earliest days, we just had a couple of us that would go to a show. I think my first one was in 2013. And I think we had four demo machines and a server. Not much. We even had to string our own cabling just to get internet. But it's really grown since then. And so now, it's lots of people. But basically what we do these days, we learned over time that the needs of this game, and really the vision that we're trying to present at these shows, usually exceeds the expectation of any venue. So we have had to invent ways of getting around that. And a good example of this is last year's CitizenCon, where we had certain constraints that meant that we couldn't hit the vision that we wanted to achieve. So we basically, well, I say we, my team of super expert specialists built out pretty much a whole studio, condensed it down into a traveling package so that we could take with us everything we needed, so that we didn't have to lean on anybody else. And by the time we were done, I felt really proud of what the team put together, with 150 PCs, a stack of servers. We had a lot of help though.

You're talking about, and I don't want this to get lost in the conversation, you and Ahmed and all your team, you built a microcosm of the entire Star Citizen development environment and brought it on site to LACC so that we could download right from the data store. So check-ins are replicated, and so all those 150 computers didn't have to hit the internet, a convention center internet; they could all run locally on that.

Essentially, you could think about a CitizenCon as, like you're saying, in a week, we build a sixth studio on site with networking, security, build system, distribution. This year, we had private cloud for game hosting, Wi-Fi, everything you would expect a studio to have. And then, like you said, at the end of the show, we got eight hours to tear it down and get it in a box and get it out of the building. So that's essentially what the week looks like for us.

My journey in the game industry started... I was living in Austin and I was managing a retail store that specialized in outdoor gear and travel and sort of exciting stuff like that. My roommate was one of my lifelong friends, who is also a CIG employee now. His name is Paul Vaden. He's in our Corp Tech department with Mike, and he's a critical member of our team. But anyway, Paul and I have been friends since we were in almost kindergarten. We have birthday party pictures together from when we were kids and stuff. To make a long story short, he worked at Origin Systems in Austin and was working on an early version of Ultima Online. And we were talking one day at the apartment, and he was telling me... I guess it started before that, because I would go hang out there at Origin with the guys and play Quake on the LAN because they had a great connection and stuff. And so at some point, I was probably spending too much time up there after hours playing games and goofing off. And they sort of said, well, hey, we have an open position working on Ultima Online. And I sort of said, well, what's that? And they gave me a beta key, and I went home and I just played the beta version of UO nonstop for a few days. And I said, well, yeah, sign me up. So I went through an interview and got a job as a game master on the beta version of Ultima Online. And that was the start of it, really: working at Origin, getting to know the team. I was particularly fascinated with online games as a vehicle for connecting people. These were the really early days of what started to become massively multiplayer online games. There's a long history before this, starting with MUDs and bulletin board games, and I'd played those games since I was a little kid, early MUDs and stuff like that. That was really cool. And so I ended up working at Origin and Electronic Arts for a few years and advanced my career internally. It was sort of funny, because one of the things that was attractive to me about going to work there was that I felt like I had a lot of responsibility when I was managing this store, and I felt like I'd have less responsibility just basically being a customer service person. And in about six months I was managing the department again and had a lot of responsibility, and sometimes wished I was back at the store, to be fair. But fast forward, I left there in 2001 to work with Richard Garriott and a small team to start a new company called Destination Games. And pretty soon after we started that company, we actually joined forces with NCSoft and became NCSoft's Western division, brilliantly called NCSoft West. And we built a publishing organization for NCSoft to service their rest-of-the-world venues. And I ended up staying with that project for nine years and left with Richard and a small team to form a company called Portalarium. And we focused on social and mobile games, and we had built some really, really cool tech, but never really had a product that hit the market super strongly. And so in 2012 we had been talking with Chris about some of his ideas about getting back into the game industry, and ultimately, when he launched the initial crowdfunding campaign at the Game Developers Conference in Austin in 2012, he was hanging out in our office space in downtown Austin and using that as kind of a base of operations. And so we sort of chatted a little bit about some of his ideas and what he wanted to do with Star Citizen and the crowdfunding campaign. And a few months later, I was working with Chris instead of Richard. And here I am. It's been about 11 years since then, and we've come a long way since those early days. Who would have thought, right? To be honest, and I don't want to make too big of a thing out of it, but I remember I worked with this guy as a game master in the very beginning, and he sort of had this attitude of, this is stupid, we're doing grunt work, if they fix these bugs, you wouldn't need us, and whatever. And I distinctly remember having this conversation with him and saying, this is the future of video games that we're doing here today. This is really going to mean something, and you should take it seriously. And so I sort of had that attitude from the beginning of, this is really where this is going. And I think it's interesting to think about the aspects of those early games that we thought were really revolutionary. We had stories of people who had military family members that were deployed, and they would use Ultima Online as a way to talk to one another after hours and stuff. Now we take that for granted as a super mainstream activity. 20 years ago, that was kind of fringe. And so I think a lot of the things that we consider mainstream today, social networking and all this kind of stuff, really have their roots in these kinds of games. So I thought that was exciting, even when I was first getting into the game. Not that I've got perfect future vision, but I was at least smart enough to realize this is really meaningful when it comes to how people interact virtually, and it's going to mean something in the future.

So talking about what John was talking about, about how games push boundaries, that is so true. And it's one of the areas that we're always chatting about; it always comes up. Like right now, everybody's obsessed with AI. When you think about it, AI is powered by GPUs, and GPUs came to exist because of video games. So kind of a similar analogy: when 3.18 got published from PTU to Live, a lot of players did not know that the typical ratio between PTU and Live is usually 25x. The problem with 3.18, though, is we decided...

One of the problems.

One of the problems, that is true. But it starts from the fact that our universe, with all of our moons and planets and players and ships, we decided to persist everything we can as often as we can. And that is a recurring theme in Star Citizen. A lot of players might not notice that we challenge ourselves as early as possible, instead of waiting till we reach the million or the two million or the three million concurrent players. So we try to amplify the challenge all the time by avoiding any kind of hacks or shortcuts. And you see it in every single aspect, but I get to see it myself as well in my field. So I still remember when we first published, our public cloud providers reached out to us and said, what is wrong with you guys? Like, what is wrong with us? And they said that we're moving an insane amount of traffic between zones. Some of our backers are technologists, and they know when you deploy resources in the cloud, you have to split your resources across zones. You cannot sit in a single zone because zones can go down, and they do go down quite often. So the first few days, and the first few days, by the way, we were not peaking concurrency at all. A lot of backers chose not to play the first few days. They like to wait, because a lot of them learned that we hotfix a lot of issues behind the scenes. We were pushing 1.2 petabytes of hot traffic every single day, and that's between zones. That's not between every single server. We still have traffic inside the zone.

You said 1.2 petabytes?

Petabytes every day, between zones, in one region, not in all of our regions. We exist in four regions.

And that's just unheard of.

Yeah, the Amazon guys were like, we've never seen this before. Not like you've gone up from what you're used to. They were like, we don't have any other customers that are pushing this kind of traffic.

A day. And when you think about it, like social media: a human being has to create content, the content has to be uploaded, and it has to be interacted with. And most of that is happening in eventual consistency, running through compression. The fact is that every single simulator that we have... and right now, while we have instances with a reasonable amount of players, we have copies of our universe. All these copies are producing data as fast as the system can handle. And we are a competitive group of people, so none of us is telling everybody else, hey, please slow down a little bit. No, everybody's saying, give me more. And that goes all the way down to the wire, to the database. And I know a lot of players, whenever they hear the word database, they get really upset. The bit that I would like for some of them to understand is we really don't want to hold our innovation behind the scenes for too long. They get to see right now we're testing Server Meshing and server crash recovery in Tech Preview. Each and every one of us cannot wait for that to go to Live. So 3.18.0 is one of these stories where we could have stayed longer till we figured out every single little problem that could happen. But they were hungry and thirsty and really very eager to play that version of the game. So thanks to all of them for staying with us till we figured some of the kinks out. We still have a lot more to go, but we will keep pushing, because that's what we do. That's why we're all here.

Well, my first firefight came about five minutes after Chris announced the crowdfunding project and the website went down. I got involved peripherally with that project because, like I mentioned, Chris's team was camping out in our office and we were comparing notes and so forth. At that time, we didn't have an office. Chris was still getting things off the ground. There were a few members of our team that had already been working with Chris on the prototype, and I think everybody was working remotely or from around the world. So in January of 2013 was when we got our first office, and it was in Austin. There are these historic homes that are like three or four stories tall, and it's funny, because this place where CIG had our first office, my wife had actually worked in that building before, in an environmental firm that had offices there on the first and second and third floor. And when I went to meet the team and really talk about switching from Portalarium to early CIG, they gave me directions. And I said, well, I know right where that is. That's perfect. And so I go up and I knock on the door of the first floor, and the nice lady that met me there said, oh, no, you've got to go downstairs. And she directed me to the basement. So it was funny: earlier I told my wife, hey, they're officing in your old building. And I had to go home and say, actually, they're in the basement of the building. And she said, oh, that's where the archaeology department was. And that was kind of how it was for us in the early days. When I joined, I think there were about 12 people that worked there. And there were some people in Los Angeles; Chris and a few other people were in Los Angeles at the time. In the early, early days, we had an art outsource team that was in Monterrey, Mexico, CGBot, with Sergio's team. We had another outsource team that was in Montreal at Behaviour. And that's actually how we met Benoit and the team at Turbulent, by a referral from Behaviour, to help us work out these website issues that we had because we overwhelmed our initial platform. So anyway, that first office was a hoot. It was a perfectly functional office. We did have a few windows, even though it was mostly subterranean. For those veteran backers, I think there's a couple of walk-around studio tours of that office that may be still hanging out on YouTube. And we used to broadcast the original Wingman's Hangar weekly show from there, with the green screen and all kinds of stuff. That was an interesting experience. We stayed there for about a year. We grew to the point where I think we had about 35, maybe almost 40 people in that space. And it was very crowded. We literally had, almost to the point of people sitting on each other's laps and so forth. So it got out of hand before we moved. But we moved into a nice new office in Austin in January of 2014. And in fact, we're still in that same office in Austin to this day.

I do have quite a few people, and I don't want to put the whole list. I would need some time for that. The way we're set up, we do have key functions and the teams that support them. There's IT managers at each of our studio locations. Our largest location has two. Each of those are supported by a really cool team. And these are people that are really passionate about what they do, and you can tell when you interact with them. And then in Austin, we have additional specialists, from network engineering to systems engineering, administration, security, cybersecurity, even specialists. And so rather than naming them all out, I'd like to call them out in that way.

The people who manage all the licenses and stuff.

Well, yeah, I already said his name. That's Chris.

You mentioned the licenses earlier. We build tools to let these softwares talk to each other. But I imagine what that must be like when someone's like, hey, we need to add another piece of software.

We're a really interesting company even in the game space, because we are equal parts game company, technology company, and a production facility. I mean, we're in a beautiful set here that you operate in. You have very specialized needs that are very different from Ahmed's team or the average game designer or the finance department or whatever. So as a corp tech publishing organization, we're supporting everything from mocap to legal document needs to cybersecurity to all kinds of stuff. Like we've said, it makes it really interesting because no two days are ever the same.

Yeah, well, we've talked about quite a few things, and I can't think of everybody, but I'd like to call out for sure Eric Green and our publishing operations team. They handle a lot of our web platform: everything that you see on the site when there's a major sale or a major event, their team has had a hand in coordinating and preparing that for public consumption. Our marketing teams and art teams and stuff, they come up with the ideas and the content and so forth, but Eric's team really makes it ready to consume. His teams also include our release management team, which is critical for the day-to-day operations of Star Citizen. In a given day, that team is probably considering three to five different builds that may go out to the PTU environment or to one of our special testing channels. And so Jeff and Marissa and Scott, they're up early talking with the team here, Francis and Matt, who have been coordinating the development activities throughout the day here. They wake up in Austin, we have a handoff, they take the baton, and then by the time the evening rolls around, we've got probably a new build that's out there for people to try or not, as the case may be. We have Will Leverett, who heads up our player experience and our live QA teams. When we do have a build candidate that we're thinking about putting out on PTU, the Live QA team, with Bayor and Waka and some of these names that the community will know very well, they're instrumental. We have a whole team of people that will test this build internally, make sure that hopefully there's no spoiler content in there, that it runs as expected, that there's not problems with all the different backend systems that have to be checked. And then Will's got other leadership team, John and John. There's two different people named John. We have a joke amongst us; we've got a chat room called the Three Johns. So we all need to come up with snappy nicknames, I guess. Anyway, they handle customer support and e-commerce support and all the different pieces that go along with that. We've got our whole team in Montreal that's handling all sorts of different platform technology development and web development and e-commerce development, and that's headed up by Doreen. So we've got our build department that handles a variety of things. It's headed up by Miles. Our development process is very rapidly evolving, and so they're always providing build system support and making sure developers can have access to the latest builds. I was just looking at a report: in the fourth quarter, we've generated thousands of builds of various flavors; every team has their own custom stream that they're working in, for the animation team and the audio team and whatever. So his department really keeps that all running smoothly, or as smoothly as possible. We have our data solutions team, and the list goes on and on. So it's exciting to see how much it takes to make it all work.

Sure. Those are different topics. Fleet View is one and Management is another. As we've looked at how to address those things over the last year and a half or two years, part of the reason we arrived at what Spectrum is today is because we want to do a really good job of those things and we want them to have meaning. Managing your org isn't all that relevant if you don't have a good way to communicate or if there's not alerts or things to do. So what will happen is in Spectrum, as we move through the different objectives that we have over the next, let's say, nine to 12 months, we'll be taking the management and the administrative functions that exist today on the website and bringing those into the Spectrum interface itself, so that you'll be able to do everything you can do today plus a lot more. The technology framework that we're using allows us to develop very rapidly, especially with UIs and some other functionality. So we'll be able to achieve the objectives that we have shared before, plus some other cool stuff that all these notifications and real-time chat and this sort of game presence allows. So we still will be able to deliver those goals, and I think there's a lot of cool new opportunities that come along that'll be good as well.

Fleet View is a little different. We've looked at, there are some tools that people have made today that we really like, the sort of maps of all the ships and different things. We're kind of working with the game designers to understand how best to address that, because we've gotten different feedback from different people. So the primary thing is that we understand orgs want to be able to control the privacy of how those things would show up, because there's got to be some strategic advantage to either knowing or not knowing what your org has, or maybe there's some privacy that, like, I may have a ship personally that I don't want to be part of the org and all this. So we're sort of waiting on the game design on that side to come along. But like I said, we'll be able to do something really cool with the UI functionality that we'll have. So there's really neat stuff.

Spectrum continues to enhance the immersion, the idea of where this is going to blend in there. Going back to the original adage of building the universe, keeping us all connected.

Absolutely. The goal is also to make it so that your experience with Star Citizen is really ambient all the time. You don't necessarily have to be logged into the game itself, but if you want to connect with people through your mobile device or through the browser or through whatever, you could sort of do that whether you're logged into the game or not logged into the game.

That actually adds a lot of other interesting potential, like what we've talked about in the past. Everything from a friend of yours needs some money and you're on a phone, so you obviously can't play the game, but you can certainly receive a request from him, respond affirmatively, etc. Your friend can invite you into a party so that as soon as you're home, all you do is join the game and you're automatically routed in with your friends who are already in and playing. Many, many other possibilities to just enhance the overall gameplay experience by bringing people together and allowing them to communicate regardless of whether they're at that particular moment in the game or not.

Yes. We're building Spectrum on a technology stack that allows for that. Principally, we're using React Redux, which is actually an open source framework that was developed by some Facebook engineers. And so it's really amazing everything that's supported in this technology. So bots and that sort of technology are available. As we've looked at the state of the art today with chatbots, there's not a lot of compelling examples in the real world that we can point to, but we can imagine lots of stuff that you could do with a chatbot in Spectrum, especially as an organization. You could potentially manage org banking, or, well, there's just a lot of stuff that we could envision. It's not something that would be available, obviously, in the first release of the product, but it's definitely something that we're looking at. We're particularly interested in finding some examples in the real world that we think are compelling that we could draw from.

Are you asking for submissions for the...

Yeah. Especially over the last few days, I had a number of really engaging conversations at the event Sunday night with people who had lots of ideas about Spectrum. We had a couple of groups of people here in the office yesterday for tours who also had some really interesting ideas. Specifically, hearing from people who are managing large orgs, or several people that I talked to manage or are involved with multi-game orgs and guilds, some of whom are enormous. We definitely want to get that kind of feedback. That's definitely one of the reasons: all throughout the development of Star Citizen we want to release information early to get feedback, so this is no exception, of course. We'd love to hear that, and I'd love to solicit feedback from people.

I think one of my favorite comments after the Spectrum discussion was, this is Spacebook, right?

I want to call it Spacebook. It is definitely a very specific kind of social network for Star Citizen, if you look at it that way. Because we're able to leverage really cool technology that's in the open source domain now, we're not trying to see things and then invent our own fundamental technology to do it. We're able in many cases to actually leverage the same frameworks that are driving popular platforms. So it's exactly that.