Reverse the Verse: December 18th, 2015

Well, you get to fly it around. Basically, flying around, and I know Erin Roberts was talking about the cargo system is coming up soon. No ETA, but we've got enough ships that can utilize it now. They're starting to...

They want to do some missions around it.

Yeah, they want to do some missions around the cargo system. So that's one of the things they're talking about right now. When that means it'll come, I don't know.

By the way, if there's one thing those guys at Foundry UK can do... So the team at Foundry UK, they previously worked on Lego games, and before that they worked on Battlestar Galactica game, Starlancer, Privateer 2, all these classic space games. And it's a lot of the same guys who carried through it. If you ever look at the number of missions in Wing Commander games, it's between 16 in an add-on, 20 or 30 in kind of a mid-range game, to 50, 60 at most in one of Chris' base games. Privateer 2: The Darkening, by the guys over there, it's more than 300 missions. And they're all really weird and elaborate. There's one that makes you do math. Everyone hated that. They are the world's foremost space combat game mission designers.

I am Sean "Criminal" Tracy. I'm the technical director for content now on Star Citizen.

Where'd you come from? How'd you come to Star Citizen?

Oh, wow. It's sort of a long story, I guess. I was one of the very originals. Back in 2010, I think it was, Chris came into my office with Ortwin, and my office being the GDC CryEngine booth. And we started talking about how to do a space game on the CryEngine. And this was back in 2010. And a lot of people, and hell, a lot of people still even now are like, well, Crytek, you can't do that. I don't know how you'd ever do such large environments, how you'd ever carry around so many ships, because, yeah, it was mostly an FPS engine. But what a lot of people don't see is all the other games that are being made on it. And my job at Crytek, I was the U.S. business director. So I was actually doing all the licensing of the engine to all the companies using it. So guys like Turtle Rock for Evolve. I'm trying to think of the ones that have been all announced because I don't want to break some NDA from Crytek or anything. But there's been a bunch of CryEngine games since then. ArcheAge, that's a big MMO. So I already knew that there was lots of ways to actually get around this stuff.

So Chris being how he is, he wanted kind of some proof, basically. So we gave him the SDK. He started building up the Hornet. So that was the very first ship that he sort of built up. He implemented all these thrusters and things like this. And this was after our first little demo. Then we met up, I think it was about six months later, or it might even have been an entire year later. And he came in with an implementation of his own ship, flying around pretty much. He had asked us how to do a space environment, so I showed him how to get rid of all the terrain, how you can just be flying around, things like this. So he came in and we were super impressed with the fact that he did this himself, because I had dealt with so many different CEOs, even tech directors, that wouldn't write a single line of code themselves. So they would come in, they would just have to take our word for it and then go from there. So he actually did work within the vehicle system. He was super happy with what he was seeing there and just wanted to know how to go a little bit bigger with it.

Now, I got pretty excited about this. And one of the ways that I moved up pretty quick within Crytek is that I can put together a prototype very, very quickly, as you saw with the planet stuff. We'll get to that. So then what I did, I recruited a guy named Hannes Appell, who now works for us as well. He's the Cinematics Director, as well as Paul Reindell. And we were all still at Crytek at the time. And what I had said to them was, listen, we really want to close this license. This will be an amazing game to have on the CryEngine. So what we did was put together that very first crowdfunding video that you guys saw. So that video changed very little. What we did was we put that prototype together and we delivered it to Chris as is.

So what's kind of interesting to know is that when we do these sort of prototype videos, what we called it was a cover demo. Just like bands do cover songs, it's a song from another band. We make games from other studios is the idea. And we do this little prototype so that way the people that we're showing it to get it. They understand that, oh, yeah, actually we can go about this. These guys put this together in a week. What could we do with three years, what could we do with four years, whatever. So we gave it to him. What's kind of interesting about it was we had a soundtrack on it. And usually when we do these cover demos, they never go up publicly. So you can kind of put whatever audio on it you want. You don't have to worry about copyright or anything like this. So we put a track from The Thin Red Line, I think it was. We sent it to Chris as was. He was pretty happy with it, as you guys saw. Pretty much unedited, there wasn't too much extra work that went on top of that. But there was a lot of tech that had to come online to make that possible, because we'd never had to put in a ship that was as expensive as an entire Crysis level and fly it around and then have additional ships flying around and things like this.

So we did that original crowdfunding video, got that out, and I remember the conversations that we were having when we were talking about the business arrangements for it. So we were saying in the original stages that we'd be lucky if we hit 5 million. Things have changed since, so I'm not too worried about talking about this, but we had sort of peaked it at about five million in terms of the royalty arrangements, stuff like this, and then this happened. So that obviously changed things quite a bit. I stayed at Crytek, obviously. Paul came over here shortly after that, I believe, but I was still at Crytek for a number of years post Star Citizen starting up. But I think I'm citizen number five or something like that on the website. So yeah, I've been around, sort of. It was really way back in 2011, I think it was.

So how I got here anyways: I had still been out in Frankfurt when that happened, a couple years out in Frankfurt, then Crytek opened up a studio in Austin, so I came out here to open that, and then, well, it's pretty public, but Crytek had some financial issues. So this was kind of an easy move. And there was a lot of different licensees that were talking to a few of us about moving over to them. And honestly, I'd always wanted to work on Star Citizen. Since making that very first crowdfunding video, I had not had more fun since working on that. And I knew that if I had a choice, that's where I was going to go. And Chris made that choice pretty easy. There were financial troubles and stuff like this. I reached out and here I am. So that was kind of how I came to be with CIG, I guess.

Well, we've actually been working on that for a while. That's the thing. So when I say we put the demo together within two days, that's not a lie. I mean, we absolutely put that together within two or three days. However, the underlying tech of that, a guy by the name of Marco Corbetta, who's one of the best gameplay programmers in the industry. He was the main gameplay programmer at Crytek. He eventually became the CTO at Crytek. As well as Carsten Wenzel, who is our rendering guru. He basically wrote the CryEngine from the get-go. So they were long at Crytek by the time I arrived. So these have always been my idols, and now that I'm working side by side with them on stuff, it's pretty exciting. But they've been working on this, trying to fit it in between the releases as much as you can.

So what's super interesting about it is, it's pretty isolated in terms of technology, where it is in the code, because it is so new. But what's super interesting about it is that Marco had done so much work originally on Far Cry as well as Crysis 1 on the terrain systems. So basically, he's taking a lot of the procedural work that he worked on for those and bringing it up to a scale that's just never been done before. And Marco's always wanted to be working on more procedural work. Between Crysis 2 and Crysis 3, we were working on a procedural building generator, so that never really saw the light of day. The next thing that we worked on, and Marco and I were together actually within the Austin studio when we started it up, we had worked on something for a game called Hunt, and what this was was a completely procedurally generated game, very much like Diablo, where you have tile sets and you sort of move those around. So he worked on that sort of level of procedural generation, and then finally he gets to work on the ultimate, which is procedural worlds, full procedural worlds.

So all the underlying tech sort of came online within the last month. Maybe two months. It's only been usable, really, for about a week. And I mean usable as in you play it and it looks great and it's fun to play and it streams in pretty fast. It's not super laggy, all these kind of things.

Yeah, it's going to be like all the stuff you would expect on a planet. But our big goal, of course, is to... I don't want to use the terms we use internally, but, yeah, of course we want oceans, we want lakes, we want rivers, we want waterfalls, we want trees and animals and pretty much everything. Now, we're not looking to do No Man's Sky. That's not this game. So we're not trying to get you to land and all of a sudden there's a civilization attacking you and things like this. It's not quite that. What it's more about is exploration and procedural mission generation and doing things on this planet. No Man's Sky has a really cool technique of this sort of living story that sort of builds up around you. We're not really going about that. We want the player to create that story, and we want the players together to create that story, and to get some sort of emergent adventure coming out of that, if that makes any sense. So it's really about what the players do with it and less about what the world will do to the player.

Oh, wow. Well, the reason that I put that there is that was my online handle since I was like 12. So I kept that for a long time. But here's the thing. For me in the industry starting out, I started out as a modder. And in the sort of modding scene, you still use your game name. So a lot of people know me from the MechWarrior: Living Legends mod, which me and my brother incepted and ran and released and brought up to be pretty popular, but everybody knew me as Criminal. Nobody knows Sean. So, in wanting to keep some of the people that played MechWarrior: Living Legends having the idea that those guys had moved over to work on CIG and things like that, I like to keep my Criminal sort of handle.

Where did the Criminal come from? I had this little line when I was like 12. It says, "Some criminals take your car, but I take your life." It was awful, but it is what it is. At least I'm true to myself. It's just Criminal, so that's where that comes from. There's nothing deeper, unfortunately. I wish there was a cooler story. I was 12, come on.

Here's what was super strange. As a modding team, what happens with modding teams, and it's a management nightmare, because people are motivated for about a month, and then they disappear. You get all these guys to come on your TeamSpeak, and on your TeamSpeak, you're all your usernames, but you're not Sean Tracy, you're not Dan Tracy, you're not your actual name. So for a long, long time, a lot of people were just calling me Criminal directly. When they're talking to me, they're saying, hey Criminal, hey Criminal, and it's on the headset. What got really weird is when I started working for real with these people. So a lot of the guys that worked on the mod team with me, we all kind of went to Crytek together. I ended up there, then I pulled my brother there, then I pulled a guy named Morgan Kita, who I only knew as Toth for the longest time. And even still, when I spoke to him in person, on occasion, I would go, Toth. And it was just so strange to him, but it was the same for me as well. So even since then, we brought on Caleb Essex, who was at Crytek, he's now at CIG as well, but he was Shadow Killer. It was so weird to meet the guys in person, though we had spoken so many times. So that was the weirdest transition.

So that's why I kind of leave the Criminal there. One, I want people to know that, yeah, I worked on the MechWarrior: Living Legends stuff. This was my baby. This was my brother and I's sort of pride and joy. It was the first game that we had done together, the first game that we'd really managed to finish, and it was a big deal. It took a lot of my time, but it also made me realize what I wanted to do as my career. I was in the military before. So it was kind of like, what do you want to do? Do you want to do the military or do you want to make games? And well, it was a pretty easy decision. It was Canadian military.

One of the big things in 2.1 is the refactored EVA. It's going to feel a little bit weird to people at first, though. I don't want to go too deep on it. So how it worked before was animation-driven motion, which means basically we're just saying entity move here, entity move here, entity move here. And then we do animation to get to that sort of position. Now this is obviously very fake. This is how most characters move around in the world though, when we go animation-driven motion for the characters walking around, things like this. In EVA it's a totally different beast. What we wanted was a pure physical EVA. So now you are a physical representation of a human. We've always had this for on foot and for ragdolls and things like this. What it required was a mix of animation-driven as well as ragdoll. So at the same time, and blending between different ratios, we are ragdolling you or animating you sort of at the same time. This is kind of a weird concept, but it works so much better for EVA than the animation-driven one.

One of the biggest differences, and I don't know if you had ever noticed this, but in our old EVA, this is a person, this is the feet at the bottom. When you would rotate, you would rotate from the feet. Which is ridiculous, and it looks super stupid when you're watching somebody else sort of do this rotation. You need to rotate from center of mass is the whole deal. And if somebody shoots you in the foot, for example, you've got to go end over end, not from the feet. So what we do with the physical representation now, we don't just say, oh, rotate from the hips. We say, rotate from whatever your center of mass is. So if I had a massive head, big set of armor, and then tiny little legs, then I'm going to rotate way up here. If I'm carrying a weapon and I'm generally human, it's going to be near the hips. It's right in this sort of center of mass kind of area. So now you rotate from that when you're controlling.

Additionally, all the thrusters are actually physical. This isn't just a fake effect anymore. So we don't just turn on these little effects and then say, oh, animation-driven, go forward. This is actually, okay, we apply an impulse to the ragdoll. The ragdoll reacts. So this is really cool when you see it happen. You'll go forward and your arms will kind of go like this, because they're sort of flapping out with the ragdoll. And then as we actually get a little bit more normalized, the animation will come back and take over and the pose will go back in. Where this is really cool, and you guys can try this out, and we're going to work on it even more: when you go up close to physics, and I'm going to pretend these two guys are walls, so I've got my hands out sort of like this, and I'm floating over, you're going to see the arm will sort of move in like this. And then it's going to come back out as you move out. Same thing with the feet. So it's pretty cool, actually. The system is similar to something called Morpheme, if people know that. It's used in the FIFA games. EA does something very similar in the FIFA games, which are pretty cool. But obviously they don't do zero-g.

The other problem is all the transitioning. So you've got to go from that sort of entity-driven movement, which could be a whole game in itself, easily, to going down to FPS combat and all this kind of stuff. Where it gets really tricky is actually doing the aim posing. So the aim posing is the hardest part about this. So actually what you guys were seeing in 2.0 is actually a pretty big bug. It might have seemed okay for everyone, but the weapon was way out in front. You could see the back of it. And it's because his hands were completely messed up. It was just sort of hovering out in front. This is not at all what we want EVA combat to be. We actually want EVA combat to be holding the weapon. And you're going to be a little sprawled out like you're swimming, but yeah, you need to be able to aim the weapon like normal. The aim posing is tricky because you need an actual root joint to do all your calculations off of. If you can imagine, you're moving in all these different directions and ways, for us to actually find a root joint that isn't moving is very tricky. So actually what we do is we do not animate the hip joint. So the hip joint is the reference joint for everything. So when we do our polar to rectangular conversions, we can use the actual hip joint for all the angles. So in that way, you have a proper pose that you are holding all the time that's very, very smooth feeling. It feels like an FPS now. So that was a super important part, and it took weeks to get that right, just the math, getting that right.

It's almost like this game isn't finished and it's still in development.

We're always in R&D land, which is awesome. And I would want nothing less. But it makes production hard, because you end up in a weird state where you're supporting live releases, so you kind of have to support some of your legacy stuff. And then when you're going to refactor something, you've got to find a slot to get all that refactored. And guess what? If that's R&D land, you have no idea where the end of that is. That's tricky. So a lot of times what you guys see is a little bit of a half-finished system. And it's not half-finished because we just left it. It's half-finished because that's a state that it's playable that we can give it to you guys.

The physical EVA is the biggest part about 2.1. We've just been working so much on that, just because there's so many bugs around the old EVA, and I can't stand it. There's so many things like the teleporting to zero, zero, zero, that's from EVA. The head being backwards sometimes. There's just so many. There were so many things that I just didn't want to fix because it's like maintaining an old system. And that's the most frustrating part. Already a guy by the name of Jens Lind in the UK helped me with one of these bugs, and I think everybody in the 2.0 PTU had it at some point or another. When you were transitioning into a gravity volume or a gravity box on one of the landing pads, if you were doing any sort of turning, you would keep turning forever. So you would jump, and then you would end up in gravity, and then all of a sudden you'd be spinning around like this, and there was no way to get out of it unless you could get off the pad, unless you could get back into zero gravity, because then it would normalize itself back out. Most people I saw on the livestreams, they were trying so hard to get back off the pad, but you were running in a damn circle, so you couldn't get out. We had to spend a good day and a half fixing that. But in the end, we've now just replaced it. So that day and a half is kind of wasted. But it's not wasted, because you guys played the 2.0 for a week and a half or whatever. But it was time that, of course, we don't want to spend. But that's the difficulty of supporting live and doing development all at the same time.

And it's important to remember, when you see things and you wonder why haven't they fixed that yet, it's possible that it could be in an older system that's being replaced, and we don't want to put that work into the older system that we know we're replacing.

And a lot of people will just go, oh, well, they just don't care about those bugs. Well, sort of, because we know it's going to be gone, or we know we're going to have a different bug with the new system, and we'd rather be working on those. Unfortunately, that's just not as visible in the live releases, I think, sometimes.

That plays a big part in balancing, because we know when the component system comes online and all these things, the power pipes actually start working and stuff, all this balancing is going to be done after all those, not before.

Tons. There's lots of stuff going on. I don't know what kind of info to give you guys, but we went crazy profiling on the servers in the last two weeks. So it's not really the client side that we're so worried about. It's the server more than anything. So we're getting the servers worked out more than the client right now. We've got to get the optimization on the servers. One of the problems for CryEngine generally is the server. When you're doing a big game like this, what you have to do is you have to allocate a certain number of instances to a certain number of cores. It's a cost thing. So you pay per number of cores per so many hours, or per minute even with some distributors. So what that means is that we've got to optimize first our servers for the number of cores that we will run. What's really hard about that is, again, because we're live development, we don't know yet how many instances per core we want to run. So we started with two. This was the very original Arena Commander sort of stuff, and it seemed to be okay. In the PTU, it got very obvious very quickly that having only two cores... Imagine just trying to run your client on a two-core machine. Forget it. And the server has to do a lot of the same kind of calculation that the client does, just a little bit different. So imagine just trying to run that on two cores. So obviously the server started cratering in terms of performance.

Here's a good example to give you as well about the server and the difference between CryEngine and, say, something like EVE. EVE, their servers, and I've met with them on many occasions before CIG, because they were looking at licensing CryEngine for Valkyrie at some point, which they clearly didn't, which is too bad. They should have. Anyways, I digress. So they have their servers updating about two frames a second. This is not a problem for EVE. Most of it's point and click. It's not very reactive. It absolutely is not the speed of an FPS game. In an FPS game, you need at least a 33 millisecond update. You need to have 30 frames a second going, and that includes the server. So if you think about it, we have to operate at least 15 times the speed on our server, so that way all the things are coming in: when a player shoots, where a player is looking, how he's flying. It's why the experience feels so drastically different, because of the server update. The server's got to run just as fast, but you also have to mitigate all the cost of how many instances per core you have, and it can get very, very expensive.

So what we did during the PTU is actually we increased it from two to three, and we're now taking a look at four. A lot of people are just going to say, well, just increase it to 12 or however many a client's going to get. And yeah, I understand that we're getting a lot of crowdfunding money. It's going well for us, but not that well. And there is a waste at some level. So really what we're trying to optimize right now is the servers on three to four cores. We'll see how those servers actually run for that. So we'll set up our server code in a way that all the threading works out really well for the server. And then we move on to the client, because that's where it's unlimited, so to speak. So you get eight, 12, however many cores the client's going to have to run. But we want to make sure that the engine scales really nicely to all of those cores. Right now it scales super well to about an eight core. So a four core with hyperthreading, it scales pretty well on. So most people that are playing with an i7 or something like this, they're laughing, everything's great. Now, on an i5, it's a very different story. So even this slight step down, of course, makes a pretty big difference, because we have some pretty major job systems going on on one of the cores, and we have to move that onto the main thread for i5s, and it gets a little bit crazy. Now, I don't want to go too deep, because I don't want to say something wrong right now and be called back and quoted and stuff like that.

And so you guys know, one of the big guys that's working on it, and he's going to hate that I use his name, is a guy by the name of Christopher Bolte. He is within the Frankfurt office. His master's thesis was on job systems. And actually, we should make sure it's available in some notes. It's online. It's mostly written in German, though. It's on job parallelization. So he's the guy working on it, and he is the guy you want working on it.