Star Citizen Live: Meet the Devs - Publishing and Corporate Technology

I'm John Erskine. I'm the head of the publishing organization for CIG. I've been with the project since essentially the very beginning of the project. I joined in the beginning of 2013. At CIG, publishing encompasses everything technical that's not related to the game engine itself. So essentially, after a developer checks their code into our source control, Perforce, then it goes into our build system, and the build system is the start of the publishing pipeline. So when a build gets generated, it goes through a series of automated tests, and it eventually goes to a QA department. It eventually gets published to some kind of environment, whether it's an internal development environment, a staging environment, or maybe a test environment. And that's all part of the publishing technology group. Then we have also our live QA team and ultimately our customer support team that are supporting the game once it's either on PTU or on live. People have questions about the game afterwards, problems with their purchases or all that sort of stuff. So all of those variety of things at CIG we call publishing in one form or another. So that's my group. There's a whole team of us. We have people all over the world. Principally we're based in Austin, where I'm based, or at least most of the leadership team is based there. We've also had a really long and fruitful relationship with the platform team at Turbulent, which is based in Montreal. So I head up a lot of the functions there related to the e-commerce and the sort of technology behind the platform there. So we're dispersed all over the place and do like a dozen different things, and it's pretty exciting.

I always saw publishing, at least for CIG, as everything that was outside, underneath, and around the games itself. Basically, we've got our developers and our five studios around the world, and you and your team basically make it possible for them to do their stuff. Whether that's providing the IT infrastructure for them to do their work, to the publishing infrastructure to, okay, now you've done your work, let's get it into the hands of the backers, to the platform stuff, like you said, to track pledges and entitlements and stuff. Really, you run this entire cloud that just sits around the game itself.

That's a good description, because it's sort of like the game. We have entire technology teams devoted to the game itself, a lot of the services that run the game, the game engine and everything that happens there. But I like your description that we sort of surround all of that, maybe in a warm fuzzy blanket, and make it possible for people outside the company ultimately to play the game.

I mean, it is pretty close. The fact is we do not just own the infrastructure that the game runs on. We take custody of the game from the moment it comes to us. So we pride ourselves on being experts in how the game runs. Sometimes the developers who work in the code themselves come sit with us and get amazed by how we get to know where the issue might be before we even get to look at logs or look at graphs or look at any kind of data to figure out what's going on. The game is a living creature. It's in its own high fidelity, and the dedication that we see from the community. We also get to see that our community does not deal with it as a rigid system. They try to always interact with it in a very, very unique way. We saw a lot of that, especially after Persistent Entity Streaming was released. And we like the challenge. We like what they bring to us. They try to break our systems. They try to break the game. So there's always a lot of fun going on. And again, it's not like you're hosting any kind of very established static piece of software, making sure it has connectivity and IO and compute. No, this is an ongoing thing that's always evolving. And a lot of the time we learn about the software we're running as it ages. So as a lot of the backers remember, we have publishes that could live for a few days. We have publishes that could live for 90 days. And every single day there is a new thing to learn. So yeah, there's a lot of holes to fill and a lot of plates to spin.

Okay, so when we say infrastructure, we mean the things that it takes for overall operations. So that could be anything from the network that we use to connect to the internet, or the secure network that we use to connect studios to each other so that people can collaborate. The servers that we have our software running on internally to help developers do their work. Storage systems. All the security that goes around that. In corporate technology, we also have a pretty large operation that is our global licensing. This is a pretty massive undertaking, considering we have over 10,000 unique licenses that we need to maintain for every different development operation, every accounting function, any number of things. And my team is the one that not only orchestrates all of that, works those deals, makes sure that everybody has what they need, and also monitors for compliance, so that we're not ever out of compliance on a license or behind on a firmware update. Infrastructure means a lot of things that are behind the scenes. So it's the stuff that makes it work, if that helps.

You cannot say enough about these teams. In the earliest days, we just had a couple of us that would go to a show. I think my first one was in 2013. And I think we had four demo machines and a server. Not much. We even had to string our own cabling just to get internet. But it's really grown since then. And so now, it's lots of people. But basically what we do these days, we learned over time that the needs of this game, and really the vision that we're trying to present at these shows, usually exceeds the expectation of any venue. So we have had to invent ways of getting around that. And a good example of this is last year's CitizenCon, where we had certain constraints that meant that we couldn't hit the vision that we wanted to achieve. So we basically, well, I say we, my team of super expert specialists built out pretty much a whole studio, condensed it down into a traveling package so that we could take with us everything we needed, so that we didn't have to lean on anybody else. And by the time we were done, I felt really proud of what the team put together, with 150 PCs, a stack of servers. We had a lot of help though.

You're talking about, and I don't want this to get lost in the conversation, you and Ahmed and all your team, you built a microcosm of the entire Star Citizen development environment and brought it on site to LACC so that we could download right from the data store. So check-ins are replicated, and so all those 150 computers didn't have to hit the internet, a convention center internet; they could all run locally on that.

Essentially, you could think about a CitizenCon as, like you're saying, in a week, we build a sixth studio on site with networking, security, build system, distribution. This year, we had private cloud for game hosting, Wi-Fi, everything you would expect a studio to have. And then, like you said, at the end of the show, we got eight hours to tear it down and get it in a box and get it out of the building. So that's essentially what the week looks like for us.

Interesting. So, moving to the US: I'm from Egypt, and during my stay in Egypt, a group of friends and I had our own little shop where we used to support startups around the world with all their technology needs when it comes to infrastructure work, DevOps work, even predating the word DevOps, predating cloud. And one of my specialties was building private clouds before the word cloud was also pretty cool. I've been in this field for quite some time. And moving to a new place, I wanted to find a cool story to attach to. And job hunting, I'm out in Texas trying to find something really unique, and I stumbled upon one of the DevOps positions in Star Citizen. I still remember, I met Francesca a few days ago, and I was remembering when I first sent the cover letter. Back when I was in Egypt, I had one of my clients that was a gaming studio, not anywhere near the size of CIG, but it was one of my favorite clients to visit, because you get to see so many aspects when it comes to computer science. And you also get to see the artistic aspect. That is not really something you find in a lot of other clients when it comes to people that would use infrastructure. So I applied for many jobs and I hoped that this would be the one. And I think a few hours later, I got a phone call. I think the cover letter made an effect. And I think a week later, I was meeting with the team for our first interview, and the journey started from there. In between, I got to consume a lot of video content. I mean, at that time, and still till now, we like to share a lot about our vision. And it was fascinating to see that there is a vision to create an MMO with high fidelity, cloud based, with the latest and greatest technology. And I like being on the bleeding edge. I like challenges. So it kept going from there.

No, I just had one of my clients back in Egypt that was a gaming studio, and it was like one out of 35 clients. But again, there has always been that fascination with gaming and how games are being made. I still remember being very young and sitting after the end of every game to read the credits. And there has always been the aspect that you do not see as many guys with a name like mine in the credits. So it was also another interesting bet: how can I navigate that world? How can I make it there? And still till now, one of the most rewarding areas of working, especially now that I was here attending a summit, is you get to work with brilliant minds. You were relating our field to the medical field. I think it's a very, very good analogy. So we have brain surgeons, we have heart surgeons, we have cancer research guys, and it's a great privilege to work in a shop that really cares about computer science and gets to use the magic of that science in so many areas, and then you get to have the art on top of it. And in our case, the lore of the game is fascinating, to see how it relates to human civilization in different areas, how it relates to some World War II style, how it relates to old civilizations that clashed. So I don't want to sound cheesy, but it's very hard not to sell our project to anybody else. It's one of a kind. So it was not hard for me: once I saw the job posting, I said, this is the one that I'm going to put all my effort in the cover letter, and here we are.

My journey in the game industry started... I was living in Austin and I was managing a retail store that specialized in outdoor gear and travel and sort of exciting stuff like that. My roommate was one of my lifelong friends, who is also a CIG employee now. His name is Paul Vaden. He's in our Corp Tech department with Mike, and he's a critical member of our team. But anyway, Paul and I have been friends since we were in almost kindergarten. We have birthday party pictures together from when we were kids and stuff. To make a long story short, he worked at Origin Systems in Austin and was working on an early version of Ultima Online. And we were talking one day at the apartment, and he was telling me... I guess it started before that, because I would go hang out there at Origin with the guys and play Quake on the LAN because they had a great connection and stuff. And so at some point, I was probably spending too much time up there after hours playing games and goofing off. And they sort of said, well, hey, we have an open position working on Ultima Online. And I sort of said, well, what's that? And they gave me a beta key, and I went home and I just played the beta version of UO nonstop for a few days. And I said, well, yeah, sign me up. So I went through an interview and got a job as a game master on the beta version of Ultima Online. And that was the start of it, really: working at Origin, getting to know the team. I was particularly fascinated with online games as a vehicle for connecting people. These were the really early days of what started to become massively multiplayer online games. There's a long history before this, starting with MUDs and bulletin board games, and I'd played those games since I was a little kid, early MUDs and stuff like that. That was really cool. And so I ended up working at Origin and Electronic Arts for a few years and advanced my career internally. It was sort of funny, because one of the things that was attractive to me about going to work there was that I felt like I had a lot of responsibility when I was managing this store, and I felt like I'd have less responsibility just basically being a customer service person. And in about six months I was managing the department again and had a lot of responsibility, and sometimes wished I was back at the store, to be fair. But fast forward, I left there in 2001 to work with Richard Garriott and a small team to start a new company called Destination Games. And pretty soon after we started that company, we actually joined forces with NCSoft and became NCSoft's Western division, brilliantly called NCSoft West. And we built a publishing organization for NCSoft to service their rest-of-the-world venues. And I ended up staying with that project for nine years and left with Richard and a small team to form a company called Portalarium. And we focused on social and mobile games, and we had built some really, really cool tech, but never really had a product that hit the market super strongly. And so in 2012 we had been talking with Chris about some of his ideas about getting back into the game industry, and ultimately, when he launched the initial crowdfunding campaign at the Game Developers Conference in Austin in 2012, he was hanging out in our office space in downtown Austin and using that as kind of a base of operations. And so we sort of chatted a little bit about some of his ideas and what he wanted to do with Star Citizen and the crowdfunding campaign. And a few months later, I was working with Chris instead of Richard. And here I am. It's been about 11 years since then, and we've come a long way since those early days. Who would have thought, right? To be honest, and I don't want to make too big of a thing out of it, but I remember I worked with this guy as a game master in the very beginning, and he sort of had this attitude of, this is stupid, we're doing grunt work, if they fix these bugs, you wouldn't need us, and whatever. And I distinctly remember having this conversation with him and saying, this is the future of video games that we're doing here today. This is really going to mean something, and you should take it seriously. And so I sort of had that attitude from the beginning of, this is really where this is going. And I think it's interesting to think about the aspects of those early games that we thought were really revolutionary. We had stories of people who had military family members that were deployed, and they would use Ultima Online as a way to talk to one another after hours and stuff. Now we take that for granted as a super mainstream activity. 20 years ago, that was kind of fringe. And so I think a lot of the things that we consider mainstream today, social networking and all this kind of stuff, really have their roots in these kinds of games. So I thought that was exciting, even when I was first getting into the game. Not that I've got perfect future vision, but I was at least smart enough to realize this is really meaningful when it comes to how people interact virtually, and it's going to mean something in the future.

So first of all, I had just stumbled into this. I had no aspirations. In fact, I didn't even realize that games were made. It didn't occur to me. I played them, but I never thought about what it takes to make a game and all of that. This is really a long time ago. You said where did the journey start, right? I got an opportunity to tour a video game development company, Sierra Online. That's way back. And at the end of the tour, it was really kind of blowing me away how cool this was. And I got to meet this guy, and it was a job offer, and he said, I have two openings: it's customer service and quality assurance. And I remember thinking, well, I don't even know. Is this for real? Am I actually getting a job here? I didn't even come to get one. But I guess they were desperate. And so I asked, which one pays better? And he said, well, customer service starts out better, but it's pretty hard. So then I remembered, here's what you're supposed to do, at least it's what they taught us: how do I get your job? And he said, well, if that's what you want to do, if you want to be in leadership, you need to start out in QA. So I did. And it went from there. I ran QA teams, and then I found that I had this real strong talent and interest in technology. And so I just kept focusing on that and learning as much as I could, and always going to the right conventions and shows to learn more about tech and get ahead of the curve. And then, I always enjoyed MMOs, so that was part of the journey. But what really brought me to CIG... because I had known about Wing Commander, I knew about Chris, but being on the West Coast, I wasn't as connected. And so this video came up on YouTube, and it was Chris at, I think, the first CitizenCon. And he was introducing his team and talking about what they were doing, and I was just blown away. How is this so transparent, so different than everything that I have learned in my career leading up to this? But the live stream started cutting out. They were having bandwidth problems. And I thought, good lord, whoever's in charge of IT there is probably having a really bad day. And you know what? I think it was two days later, John called me up and said, hey, do you want to come down here? We may have something we want to talk to you about. And so that was the real beginning of my journey. And I couldn't be happier, because of anything that I've done in technology or IT, it felt like it was always on rails. You didn't have a lot of creativity. Here, we're doing something no one else would try. And that means for me, I get to design solutions and work on things where we're actually encouraged to solve problems that other companies don't have. That's candy land for us guys.

So talking about what John was talking about, about how games push boundaries, that is so true. And it's one of the areas that we're always chatting about; it always comes up. Like right now, everybody's obsessed with AI. When you think about it, AI is powered by GPUs, and GPUs came to exist because of video games. So kind of a similar analogy: when 3.18 got published from PTU to Live, a lot of players did not know that the typical ratio between PTU and Live is usually 25x. The problem with 3.18, though, is we decided...

One of the problems.

One of the problems, that is true. But it starts from the fact that our universe, with all of our moons and planets and players and ships, we decided to persist everything we can as often as we can. And that is a recurring theme in Star Citizen. A lot of players might not notice that we challenge ourselves as early as possible, instead of waiting till we reach the million or the two million or the three million concurrent players. So we try to amplify the challenge all the time by avoiding any kind of hacks or shortcuts. And you see it in every single aspect, but I get to see it myself as well in my field. So I still remember when we first published, our public cloud providers reached out to us and said, what is wrong with you guys? Like, what is wrong with us? And they said that we're moving an insane amount of traffic between zones. Some of our backers are technologists, and they know when you deploy resources in the cloud, you have to split your resources across zones. You cannot sit in a single zone because zones can go down, and they do go down quite often. So the first few days, and the first few days, by the way, we were not peaking concurrency at all. A lot of backers chose not to play the first few days. They like to wait, because a lot of them learned that we hotfix a lot of issues behind the scenes. We were pushing 1.2 petabytes of hot traffic every single day, and that's between zones. That's not between every single server. We still have traffic inside the zone.

You said 1.2 petabytes?

Petabytes every day, between zones, in one region, not in all of our regions. We exist in four regions.

And that's just unheard of.

Yeah, the Amazon guys were like, we've never seen this before. Not like you've gone up from what you're used to. They were like, we don't have any other customers that are pushing this kind of traffic.

A day. And when you think about it, like social media: a human being has to create content, the content has to be uploaded, and it has to be interacted with. And most of that is happening in eventual consistency, running through compression. The fact is that every single simulator that we have... and right now, while we have instances with a reasonable amount of players, we have copies of our universe. All these copies are producing data as fast as the system can handle. And we are a competitive group of people, so none of us is telling everybody else, hey, please slow down a little bit. No, everybody's saying, give me more. And that goes all the way down to the wire, to the database. And I know a lot of players, whenever they hear the word database, they get really upset. The bit that I would like for some of them to understand is we really don't want to hold our innovation behind the scenes for too long. They get to see right now we're testing Server Meshing and server crash recovery in Tech Preview. Each and every one of us cannot wait for that to go to Live. So 3.18.0 is one of these stories where we could have stayed longer till we figured out every single little problem that could happen. But they were hungry and thirsty and really very eager to play that version of the game. So thanks to all of them for staying with us till we figured some of the kinks out. We still have a lot more to go, but we will keep pushing, because that's what we do. That's why we're all here.

Well, my first firefight came about five minutes after Chris announced the crowdfunding project and the website went down. I got involved peripherally with that project because, like I mentioned, Chris's team was camping out in our office and we were comparing notes and so forth. At that time, we didn't have an office. Chris was still getting things off the ground. There were a few members of our team that had already been working with Chris on the prototype, and I think everybody was working remotely or from around the world. So in January of 2013 was when we got our first office, and it was in Austin. There are these historic homes that are like three or four stories tall, and it's funny, because this place where CIG had our first office, my wife had actually worked in that building before, in an environmental firm that had offices there on the first and second and third floor. And when I went to meet the team and really talk about switching from Portalarium to early CIG, they gave me directions. And I said, well, I know right where that is. That's perfect. And so I go up and I knock on the door of the first floor, and the nice lady that met me there said, oh, no, you've got to go downstairs. And she directed me to the basement. So it was funny: earlier I told my wife, hey, they're officing in your old building. And I had to go home and say, actually, they're in the basement of the building. And she said, oh, that's where the archaeology department was. And that was kind of how it was for us in the early days. When I joined, I think there were about 12 people that worked there. And there were some people in Los Angeles; Chris and a few other people were in Los Angeles at the time. In the early, early days, we had an art outsource team that was in Monterrey, Mexico, CGBot, with Sergio's team. We had another outsource team that was in Montreal at Behaviour. And that's actually how we met Benoit and the team at Turbulent, by a referral from Behaviour, to help us work out these website issues that we had because we overwhelmed our initial platform. So anyway, that first office was a hoot. It was a perfectly functional office. We did have a few windows, even though it was mostly subterranean. For those veteran backers, I think there's a couple of walk-around studio tours of that office that may be still hanging out on YouTube. And we used to broadcast the original Wingman's Hangar weekly show from there, with the green screen and all kinds of stuff. That was an interesting experience. We stayed there for about a year. We grew to the point where I think we had about 35, maybe almost 40 people in that space. And it was very crowded. We literally had, almost to the point of people sitting on each other's laps and so forth. So it got out of hand before we moved. But we moved into a nice new office in Austin in January of 2014. And in fact, we're still in that same office in Austin to this day.

Oh, I started in the basement, and it wasn't a hoot. Because by the time I started, it was closer to the end of the basement. And so it was overcrowded. And if I had to deploy a PC, you put it up over, because otherwise there's not enough room. You can't carry things around in there. Oh, the stern looks I'd get from certain devs if I bumped their chair leg or something. I came to the company right around the time that we launched the hangar module for the very first time. And we didn't really have much in the way of infrastructure. We had, I think, two tower servers. That was it. That's what ran the company. And there was no server room. I was introduced to my office, which was really a Harry Potter-style under-the-staircase office, or closet, and this is also the server room. And I have to tell you this funny part, because you're fresh in a new job, you want to make a good impression, but it's not making a very good impression on me, because they have this black contractor tarp over the top of the servers. And I wondered, why is this? Well, they pointed out that that's because the sewer pipes go over your head, and if they leak, we don't want that getting on the servers. But I didn't get a tarp. Anyway, that office, we got out of there pretty quick. I'd like to say that I was instrumental in moving the company to the new office, and literally the physical act of moving. So one of my first hires, a man named Chris, I had worked with him before, and I brought him down from Canada to help me with this move. And he did such a great job, the company asked for him to stay, and he moved his whole family down. But he and I carried the core of the company, like all of our intellectual property, source code, everything. And I'll never forget, because he would always say, slow down, we're coming up on a curve. You don't want to lose Perforce. That's the company back there. Be careful.

The first one was that when I went to my interview, I wore a suit, and I'm standing outside trying to get in, and one of our co-workers comes out in baggy shorts that have paint on them and just runs right by me, and right behind him I see one of our co-workers in a tank top, and I was like, I made an awful mistake, I'm wearing a full suit. And then I got in, and you see a dark space with a lot of artists and a lot of cool-looking people, and you're just walking around with a leather notebook and wearing a suit. And I got to have my interview in what we call the war room. And at that time, the AC used to have a very interesting vibration, and I have a very sensitive ear. So yeah, that was my very first impression. And I came the first day to work, and I'm not sure if Mike will like the story or not, and I found a Windows PC. And I had not been a Windows user for quite some time. It's quite common in my field that people use Linux. And I was like, what is this? They're like, it's your computer. And I think that's how Mike and I got to be friends and know each other. He came to my office and knocked on my desk, and he said, my name is Mike Jones, and that's all you need to know, and walked away from me. And the next night we were hanging out assembling my Fedora Linux machine, and that's how things went. But if you're talking about memorable moments, I still remember our concurrency. We had times when we used to cheer because we had 200 players online, and we're a big magnitude above that right now. I still remember when we first launched Area 18 and the whole QA team had their buggies with them by the elevator for every player to come in and ram into them.

I have a confession on that. So we're about to put Area 18 in for the first time, and in one of my very first conversations with Jake Ross... Now, I'm going to preface this by saying I didn't know what my limits were or what I should be involved in or what I shouldn't be involved in. I was kind of all over the place in 2015. Anytime somebody did something we didn't have a person to do, I'm like, I'm going to do it. I was the events manager for a while. I was the platform guy before Eric Green. Eventually, like, wait, Jared's doing the platform? Oh, we need to hire somebody. Keep Jared away from the platform. But we're a day away from launching Area 18, and I go to Jake Ross and I say, you know what would be fun? Put some Greycat buggies in there. And he goes, oh, yeah, we should put some Greycat buggies in, let people drive around. And it was just this innocent little suggestion, and it took 24 hours to turn around. And then when Area 18 launched for the first time, it was carnage. It was absolute carnage. People were just mowing everybody down with the Greycats. The Greycats, you could look at them intently and they would explode and whatnot. But I remember Jake Ross did not throw me under the bus. When Chris goes, where did the Greycats come from? he was just like, it was a decision made internally, we'll take them out in the next build. But he had a chance to throw me under the bus that day, and he didn't.

Yeah, so many great moments. 2.0, I still remember, I think it's one of the longest times I stayed awake. Not because I was able to do much; we were crashing, and then watching the streamers just counting the crashes every single time, and they keep going back in again. 3.0, I think one of our coworkers, I'm not going to name him, was about to propose. And he was literally on his laptop right in front of their in-laws saying, hey, guys, let's launch the game so I can go inside and meet these people for the first time and prove that I'm a responsible man. So we have a lot of amazing moments, and I think right now, we're talking a lot about the future because we're here to have a summit, laying out the plans for the next two years, and we just keep dreaming of what 1.0 will look like and how we get to see the game get into the shape that we all dreamt of. So yeah, it's very difficult to pick a certain moment, but there's a lot of memories, a lot of sleeping under the desk. I think you and I one time were on a call at 2 a.m. in the morning, and I was leaning back on my desk, and I flipped backward. I'm not sure if you remember or not. And we were like, that's it, Ahmed's gone.

I do have quite a few people, and I don't want to put the whole list. I would need some time for that. The way we're set up, we do have key functions and the teams that support them. There's IT managers at each of our studio locations. Our largest location has two. Each of those are supported by a really cool team. And these are people that are really passionate about what they do, and you can tell when you interact with them. And then in Austin, we have additional specialists, from network engineering to systems engineering, administration, security, cybersecurity, even specialists. And so rather than naming them all out, I'd like to call them out in that way.

The people who manage all the licenses and stuff.

Well, yeah, I already said his name. That's Chris.

You mentioned the licenses earlier. We build tools to let these softwares talk to each other. But I imagine what that must be like when someone's like, hey, we need to add another piece of software.

We're a really interesting company even in the game space, because we are equal parts game company, technology company, and a production facility. I mean, we're in a beautiful set here that you operate in. You have very specialized needs that are very different from Ahmed's team or the average game designer or the finance department or whatever. So as a corp tech publishing organization, we're supporting everything from mocap to legal document needs to cybersecurity to all kinds of stuff. Like we've said, it makes it really interesting because no two days are ever the same.

Yeah, well, we've talked about quite a few things, and I can't think of everybody, but I'd like to call out for sure Eric Green and our publishing operations team. They handle a lot of our web platform: everything that you see on the site when there's a major sale or a major event, their team has had a hand in coordinating and preparing that for public consumption. Our marketing teams and art teams and stuff, they come up with the ideas and the content and so forth, but Eric's team really makes it ready to consume. His teams also include our release management team, which is critical for the day-to-day operations of Star Citizen. In a given day, that team is probably considering three to five different builds that may go out to the PTU environment or to one of our special testing channels. And so Jeff and Marissa and Scott, they're up early talking with the team here, Francis and Matt, who have been coordinating the development activities throughout the day here. They wake up in Austin, we have a handoff, they take the baton, and then by the time the evening rolls around, we've got probably a new build that's out there for people to try or not, as the case may be. We have Will Leverett, who heads up our player experience and our live QA teams. When we do have a build candidate that we're thinking about putting out on PTU, the Live QA team, with Bayor and Waka and some of these names that the community will know very well, they're instrumental. We have a whole team of people that will test this build internally, make sure that hopefully there's no spoiler content in there, that it runs as expected, that there's not problems with all the different backend systems that have to be checked. And then Will's got other leadership team, John and John. There's two different people named John. We have a joke amongst us; we've got a chat room called the Three Johns. So we all need to come up with snappy nicknames, I guess. Anyway, they handle customer support and e-commerce support and all the different pieces that go along with that. We've got our whole team in Montreal that's handling all sorts of different platform technology development and web development and e-commerce development, and that's headed up by Doreen. So we've got our build department that handles a variety of things. It's headed up by Miles. Our development process is very rapidly evolving, and so they're always providing build system support and making sure developers can have access to the latest builds. I was just looking at a report: in the fourth quarter, we've generated thousands of builds of various flavors; every team has their own custom stream that they're working in, for the animation team and the audio team and whatever. So his department really keeps that all running smoothly, or as smoothly as possible. We have our data solutions team, and the list goes on and on. So it's exciting to see how much it takes to make it all work.