Happy Hour Interview: Sean Tracy, Steve Bender, & Jake Ross

A typical day working on Star Citizen. What are the kind of things that I'm involved with? So the technical content team, and I sort of explained this in a studio, or Eric actually explained it pretty well in a studio update, is actually made out of two separate teams. One is tech art. The other one is tech animation. And basically, the easiest way to describe the guys on my team is a technical animator is an animator that is technical. Basically, it's a guy that has the ability to script, make tools, and can basically go back and forth between animation and code and back to animation and code. And these are the perfect type of guys to support or to develop pipelines with animation. And then the same thing on the technical art side where we have artists that are technical. And these guys, again, are developing the exact same sort of thing, pipelines and processes for the general art teams as well as improving processes. So day to day, it's a pretty hectic day because much of it is spread across the entirety of the company. So I'm not just working on the ship pipeline. I'm not just working on the character pipeline. I'm not just working on the animation pipeline. All of it at the same time. So individuals on the team anyways are embedded within these teams, and then it's a lot of feedback back and forth, and what this game absolutely requires, because the content is to such a scale, is an oversight across all those departments. To ensure that we're all doing things the same way, to ensure that we're not doubling up on work, which is super easy to do, especially when you've got a studio in the UK, a studio in Frankfurt, and then a studio over here, and then another one in Austin. So it's really easy to even get different people that haven't necessarily talked to each other working on identical stuff, so you'll end up getting double work happening. And that's a real frustrating thing for guys, as well as for the teams that are receiving tools and things from us. On top of that, I really enjoy doing a lot of little R&D projects or novel implementations of things that really haven't been done before. I really enjoy working on that, so I try to do as much of that as possible. But of course, depending on the day, there's more or less time for that kind of stuff. And then on top of it is just bug fixing, getting in there and sorting out problems that are actually within the game.

So you're very much the glue that keeps this company together.

I like to think so, but with anything, it's not imperative that you have a tech artist within a feature team or within a process, but it does force multiply that particular team. Anytime you embed one of those guys in there, they're going to make the process faster. They're going to come up with solutions for things and be able to develop content at the scale that we needed for Star Citizen. But it's very much sort of like a client; delivering tools to a client, for example, would be delivering tools to Steve's team. And that can be sometimes easy, sometimes not so easy.

Well, things like, for instance, as we're doing a tremendous amount of work on the story side with cinematics, PCAP, things like this. When they shot the performance capture, they had an editor go in and do edits like you would say in a movie or something like that. And what that creates is multiple FBX files with different time codes that are associated with them. And that can actually be tricky and time consuming to stitch those together properly. In some cases, this took half a day to a day to do on, say, one scene. We talked to Sean's team and they put together an EDL importer that does the job in about 30 seconds.

So, like we were saying, it's a force multiplication thing. It's really to try to make these things faster, because if the cinematics team continued to work that way, it would take until, I think you even scheduled it, it was something like 2025 or some ridiculous amount of time. If that was the way we were going to implement it, this is how long this was going to actually take, pretty much, before we actually shipped Star Citizen. And that's obviously unacceptable on many angles.

Okay, so I've always made a distinction, and this is really to preempt what we are doing for your character creation as a whole. So the first step of character creation is literally just selecting objects for your character, and that to me is customization of your character. And I mean, really, the term doesn't really matter. You can use it generally. The terms tend to get used interchangeably. To me, it's a two-phase kind of approach. Again, we want to give you all the selection of your character items first. So you select a head, a preset head. You select a body, a preset body. You select your clothing. You select your armor. And you put your character together. Now, the next step of that is actually customizing it or changing it, right? And we've shown some of the DNA system, which is something that we've been talking about and developing. We haven't developed the DNA system itself. This is coming from 3Lateral, but we've been developing with that in mind. So all of our heads are implemented with a very specific set of rig logic, so that once that system was online, that is exactly how we would actually be doing that thing. Well, not quite two years ago, but a year and a half ago, that we would be moving this direction, and that would be the end game for character creation and customization. So to me, the creation of your character, again, is just putting the pieces together, and then the customizing of the character is actually pulling it around and making your own version of that particular character and coloring up your clothing for yourself. Customizing your character. So creation, customization, kind of two stages. And the first stage that you guys are going to get is the selection of the preset heads, of the preset bodies, and putting the clothes and armor on. And then the next level will be the actual pulling around of different pieces of the face and getting it all combined.

One thing I did want to address within that, and it was just based on some of the Reddit threads that came out of that, as well as even some YouTube videos from 3Lateral's presentation at GDC, and a lot of people are like, oh, well, on the face customization, they're just combining blend shapes between a bunch of different faces. And it is far more sophisticated than that. There's no way that you could do this blend shape sort of combination and still be able to animate those faces. So the DNA system, and again, 3Lateral does a pretty good job explaining this within their presentation, but what it is, is there's a whole set of rig logic constants that are sitting in this face. Your face is actually created by these constants, this rig logic code, and that's what we call DNA. So when you're customizing it, there's like a dynamic DNA. This DNA is changing, so you're grabbing a bit of rig logic from this head, a bit of rig logic from this head, a bit of rig logic from this head, depending on where it is, and it's actually adjusting all the constants to suit. But all of those heads themselves have 444 blend shapes already. So you're not having a base head that just has a bunch of blend shapes of all the other heads. You have this base head that already has 444 corrective blend shapes within it. And when it animates, those blend shapes work. But you combine those rig logic constants onto the different areas of the face, and it will still animate correctly.

Maybe to take it even higher level, if you were ever to just animate and export an animation for a particular face, the rig is in a certain configuration. So if we took Gary Oldman's animation and we exported that on Gary Oldman's skeleton, and then we played that back on, let's say, Captain White or Liam Cunningham, his face would strangely contort to the shape of Gary Oldman's face, because all those joint positions are all coming from Gary Oldman's rig. Now, rig logic puts a generic, not really generic, but a generic control set. But the rig logic constants are all unique per head. So if you can combine pieces of those rig logic constants, you never end up contorting that face. So Liam Cunningham's face will retain its shape and still do the exact same animation that you exported off of Gary Oldman's head, rather than just assuming that shape. So hopefully that wasn't too, maybe I got a little too deep with it. And it's really better if I have some visual sort of guides. And we're going to have, I'm sure, a whole ATV to talk about this. And it'd be awesome if we could get Vlad out from 3Lateral. But again, we've implemented this right into the engine. And yeah, it's going to be really, really cool to see. I can assure you that there hasn't been a customization system before, at least not that I know of, and I don't know everything. I've never seen a customization system being able to do that. So basically be able to play normal animation at pretty true-life fidelity and be able to change exactly how that particular face looks and still be able to play those same animations.

The helmet HUDs never really went away. Oh, for ships though, right? Well, the main thing with our helmet HUD is that that was meant to give you a bunch of FPS feedback. So how much ammo you have, what's your health. It's kind of your character HUD, really. And that's the one that goes on the helmet. I don't know what the plan is for ship HUDs and the helmet. Maybe Steve has an idea, but I'm not so sure. We're not really the guys to answer that. And really, it's a design question more than anything. There's no technical limitations going one way or the other, but I'm pretty sure we're going to go with the MFDs in almost every single ship. But you're still going to have some sort of FPS HUD up there, because you have to know the status of your character. I mean, what's your health? Because we already have them now, the oxygen levels and all these other things, you're going to want that sort of feedback whilst you're in the ship and still be able to get the feedback from the ship. So MFDs make more sense, I think.

Okay, again, there's no first and third person though, right? It's first slash third. So it's unified, and I mean, that's a great question for Steve.

Star Citizen works off a unified system, unlike, say, a traditional FPS where you may be using the same skeleton, but the animations that are being played on third person and the ones that are being played, at least on the hands, on first person are slightly different. On Star Citizen, they're exactly the same. So what we've started doing is we've started building these goal teams or feature groups. And what they're designed to do is to take sections of a feature and put together an animator with a coder, with a designer, to be able to start executing those things out. Short-term sort of things that we're working on right now, for instance, the shouldered weapon: really getting the railgun to feel like it's a ship-downing weapon. You should be able to just pull this thing out and fire it and just feel like you have just let loose all hell. We're working on that from an effect side, from an animation side, from a code standpoint. It's about feel, right? Getting everything to feel correct. We're working on, as well, things like, Sean mentioned oxygen and stuff like that. We're working on oxygen and stamina. What does it look like when I've got full stamina and I'm running up against something like a wall that I'm going to mantle up onto? Okay, well, what happens if my stamina goes down and down and down? There's a visual difference there, right? Same thing with when my stamina is basically gone and I'm just too exhausted to jump. I'm too exhausted to climb. What does that look like? As well, continuing with, we have new weapons that are constantly coming into our pipeline, and ultimately making an FPS is not just the animation that goes into it. It's the code that goes into it and all this. But ultimately, it's the feel. If it doesn't feel right, if it doesn't play well, you need to be iterating on this until it does.

You have that with the FPS side, but you also have that with the player animations for other things like getting in and out of a bed and stuff like that as well. I know that we've been doing some polish passes on the first-person animations for things like exiting a bed, sitting in a chair, and making sure that the camera's all situated and all that.

Exactly. We've also been working on things like, for instance, usables. If I'm an AI character and I'm going to be sitting here talking with you guys drinking a cup of something, we need the AI to be able to walk in, sit down in the chair properly, to be able to grab the glass, to drink from it properly, to get back out. And this is from just sitting down at a table like this, with a tray table, with food, with drink, at a bar, say in the nightclub and things like that. And also enabling the player to do those sorts of things. So we can have the player sit down at the mess hall and pick the glass up and use the glass, things like this. And we're also moving forward on things for player interaction. Picking up, aside from, say, a glass, but looting things, stealing things from the world and stuff like that.

The player interaction is kind of an interesting thing. There's a lot of little challenges in there that you wouldn't really expect. And one of the big ones is just mesh formats, for example. So a character generally is made out of skin files. So these are the ones that deform and stuff. And an interesting thing with the player interaction system is that it requires... And it's all these little niggling things that take the time. But again, once the system is in place, everything opens up, clouds open, angels sing. And I mean, it sounds like a small thing. Okay, AI, pick up a cup, great, big deal. Not going to make my game much more fun. But what it does do is it opens up the ability for us to do everything else that those AIs need to do. That just happens to be the very controlled environment that we can do this particular tech in. But then once that is in, AIs can then pick up weapons. They can do whatever and sort of inspect and these kind of things. AIs can pick up pieces of the player even, or other AIs. I mean, there's all kinds of crazy things.

Yeah, it's about limiting the delta between systems when we're first working on them. The same piece of functionality and all the tech that gets my hand, whether I'm a player or an AI, to go to this object, this object is oriented properly, for me to pick it up and use it, is the same piece of tech that's required when you go to, say, a shop, if a shopkeeper of some sort pulls a weapon out and gives it to you. It's the same tech. And that's the other thing. In the goal teams, we're also trying to get them to, instead of, say, a typical task which would be have an AI pick up a cup, it is a broader vision. So we want them to be able to pick up this and this weapon and to give you this and to put this away and stuff like that. Now that we know what that broad vision is for that, what is a singular test case that we can do to pound on the issues?

Yeah, exactly. And I think it's important always to bring that up, because I think a lot of times we do talk about these little pieces of tech and there's one implementation. Everybody's going, well, what's the big deal? AI, pick up a cup. Who cares? I mean, you could get away with that, and a lot of other games, go play Morrowind or Oblivion to that effect, and look, very little interaction between AI and the world. Hell, they even got the same voices when you start talking to different AIs. It's like, hello there. Hello there. I think it's important to realize that there's bigger needs from those systems. Though we talk about the one specific example, it's a bigger thing that we're trying to do.

So, melee animations. Some of this is typical sort of things that you would expect to see, where you're going to attempt to come across and, say, punch somebody in the face. If we had some sort of sharp object on the end of this, you might stab them as well, coming in with a butt strike to the face, something like that. One of the things that we're really excited about is (Sean, would you come with me? And just look at the camera there) stealth kills. So we have various things where we're going to come up behind a guy and we're going to grab him and stab him and drop him. We actually have one where a character hauls off on another character. We've got left side ones where a character will come in; it's built into quadrants. So you have a rear, a left side, a right side and a front. So things like on the front side coming in, and again here through the back. I think for the front, we also have just a straight on. You're here, and it's just like that.

Perhaps, but I know what definitely will be, and it's the Sean Tracy haircut. Every YouTube video, I at least see three or four comments: somebody tell Sean that he's balding behind. I went home after one of these ATVs, and I read the YouTube thing, and it said, somebody tell Sean. I said, what? So despite all you guys, I'm going to put one in there. And if I had to do it myself, character art was like, no way. I'm like, I'll do it myself. I don't care.

It's going to be the lowest-poly hair we have in the game.

But the shiniest. Of course we want nice full beards, and we want some fun ones too. But actually, beards are a tricky thing, and that's something that we definitely need to have a conversation with Vlad about, in terms of how do you adapt the beard to different facial shapes and all this kind of stuff. You can get away with a lot of clipping and stuff, but yeah, that's a very, very nice full beard. And doing beard and hair is just a tricky shading problem too.

Animals and plants. It depends. I think it's kind of a difficult question to answer. Can you clarify that a bit more? The animals, for instance, any animals that we have in the game will have a proper animation set for them. So typically for us, design drives a lot of what it is that the creature can do. So that way you don't have a space deer doing jiu-jitsu.

I can mention the tree stuff, like how we actually get away with not animating trees, because of course trees don't need to mantle and take cover and do locomotion. But trees do need to rustle in the wind. They need to bend with the wind. So back at Crytek, this was a huge thing, right? I mean, we had palm trees, we had huge, huge forest trees, and they all needed to rustle in the wind. The palm trees actually needed to bend and move around. So we have a lot of the same technology that was being used for that. It had to be re-implemented in kind of a new way for the planets, and because of the sheer amount of them. But we developed some tools to help the artists mark up the trees; in such a way, we actually use vertex colors. Every single channel does something a little bit different. It's interesting that we keep talking about this, because someone calling used vertex colors and all these other things. I'm trying to remember what exactly each of the channels does, but one of them is like the intensity, one of them is the frequency, and the other one is like a masking, so basically where it goes. Now, the guys have developed some tools so that the artists can actually mark this up a lot easier, so that they're not just grabbing individual verts and saying, okay, this one needs to move this much, this one needs to move this much. And we did develop a pretty good process back at Crytek to make sure that, again, the trees could bend in the wind. They could rustle in the wind. And even better is when you walk through things like grass and that kind of thing, they bend away. We had touch bending and detail bending is what we call it. So detail bending is the sort of stuff that happens with the wind. And then the touch bending is something that you walk through, grass and things like that.

It's very visual effects or physics oriented, depending upon how it works. It is animation in the sense that it moves, but it's not skeletal character animation or something like that.

But it's vertex animation.

There are currently design discussions on full hand-to-hand combat. So a couple months ago, I did some just rough blockouts for punching, blocking, and things like that. Counters, these are all the... Quick answer to this is probably. There are design docs floating around for things like this. We have talked about things like what happens when you get into a bar fight or something like that. The piece of technology that actually creates the stealth kill result would be in part some of that same technology that creates a counter. So if you're just going to do a block, you don't really need anything from the other character, right? Your hand is just going to go up and you're going to block or something like that. But if you're going to counter, like he's going to come barreling down on top of you with something and you're going to go into, I don't know, a shomen-uchi iriminage or something like that, you need the two characters to be in sync. And that in-sync connection then is the same piece of technology that would be a takedown.

Yeah, and maybe to manage expectations here a tiny bit too, we're not talking Dishonored-level type melee combat, and we're not talking For Honor or anything quite as involved, right? This isn't the core of the game we're going to have. We hope to have pieces of that, of course. I can't even say where it's supposed to happen, but just to maybe manage the expectations there a tiny bit. We're not making Dishonored, is the thing.

I mean, a lot of the further-down-the-line melee system stuff is scheduled for later on this year.

Every day is chock full of challenges. The difficulty, I think, in talking about a couple of them is, I don't know that we're going to overcome it, or I don't know that I'm going to solve it in a particular way or whatever. But actually, skeleton extensions are messing up right now. That's a pretty fun thing. This is pretty frustrating, actually. We talked a lot in the character customization stuff that we showed in terms of the port setups and attaching different armors and the helpers moving and all this kind of stuff. It works fairly well, except we've run into a tiny bit of an issue with similarly named joints, and the joint ID is getting mangled, basically. So a joint thinks it's in one place in the animation, but actually it's in a different place in the character. So I've got an awesome video just from one of my senior tech artists this morning, with a guy that we've reattached a torso piece to, and all of a sudden his head detached, so his head is now on his knuckle, because the ID, like, it's on the middle finger, but that's his root now for his head. It's cool, because he can still talk. But you've got the legs, the chest is over here, and I don't know if we'll share the video, it looks pretty awful. But you've got the legs, the chest is right beside it, and then the head is out on the hand. And then we play animation on it and it kind of looks pretty reasonable. The chest does the thing, the legs do the thing, but they're all kind of weirdly attached and separate. So that's an issue we run into right now, and working with Paul Reindel and trying to work that out, and Ivo Herzeg back in Frankfurt.

Well, I think the answer from my side would be that a lot of the stuff that we're doing, because it involves multiple departments and multiple different people's expectations, is that, I use this example from time to time, pick an object, let's say a cup, right? And I say to you, I need a cup. Likely, you're not going to model this exact cup. You're going to give me something else. But that's not what I'm expecting. I'm expecting this cup. So setting the expectations, being able to say, okay, we're going to work on breathing and stamina, and not only the final result that the player should be able to feel or play within the product. The way that we tend to solve this is by removing as much sort of outside requirements as we can. So we take code out of the equation at first. We take animators, we put them together, obviously, with the goal team, where the programmers are, the designers are, the animators are. We talk about what we're trying to achieve. And we create the animations as a previs, basically, to say this is ultimately what this needs to look like, what this needs to feel like. Then we're able to take that result and we're able to put it in front of people and say, this is what we're trying to achieve. How do we get there? It's a hell of a lot easier for me to get a model of this mug if I send you the picture of the mug, and then you guys go, okay, well, we have to model this and look at the specularity of that, and oh, this is kind of a matte finish here, than it is for me to go, give me a mug. Not that mug. Not that either. Can you make it black, but not that black? So I think that's super, super important for things being successful, communicating between different departments and also within the goal team, and being able to, you don't want to get deep into these things and then someone go, oh, yeah, that's not what I meant.

And these two things that we just mentioned here, you were talking about struggles, or you have bugs that are introduced that you have to work through and solve, and these R&D-type phases of these features that we do, are huge things you have to take into consideration when you're creating schedules. Because you don't know what bugs are going to come up. You don't know how long certain R&D tasks are going to take. So these are things in production that we have to take into account. And sometimes we're extremely accurate, depending on how much experience you have with the thing you're working on. But sometimes we're way off, just because we don't know what we don't know. And so when we're creating schedules that we're putting out to you guys, that's something that we're absolutely keeping in mind: okay, what kind of bug-fixing time do we need to allot for this? And what kind of R&D time do we need to allot for this as well?

Sure. And a lot of times, another thing that comes up too is reviews of particular assets or particular things that we've done, or particular features even. A lot of times, in the past anyways, the schedule had been such that we didn't allot for a lot of review time. So at the end of it, the feature came in, whatever was online, and maybe I reviewed it or Steve would have reviewed it. Let's say it's an animation, for example, and actually Steve's not fully happy with that, but in such a way that actually we're out of time. So sorry, Steve, I know you don't like it, but guess what? It has to go. So a lot of that just comes out of, I think, organizational maturity, in terms of the organization maturing and producers getting better.

Yeah, so we created those feature teams and things like that. We said, well, we want to have characters going in, let's say, a mess hall. We want to have characters going in, sitting down, having a proper AI schedule where they go through their days. They get up, they go, they sit down, they eat something, they go to the bathroom, they do X, Y, and Z. So we could create animations for it. They could create code for it, whatnot and whatnot. But when those things were split apart, what you were getting is you were getting animations and you were getting code, but the two things didn't always mesh. So what we did is we said, okay, stop. We're going to create, as we talked about, this simple delta and put together an animator, a coder, a designer, a tech guy, and said, this is what we're trying to achieve. And we would start, let's say, on a Monday. And two Fridays after, we have an official review. So every two weeks, they're putting out, in a way, a little bit of a mini product, where we can say, great, show us that in the game. Show us that in the level. And during that time period, I'm able to talk with the animators involved or the leads involved with that and say, let's keep this in mind. Let's adjust this. I like what you're doing here. Let's do more of this. Let's do more of that. But what it also does is it gets the animators to work with the designers. And together, they come up with solutions to things that we may not have foreseen from the get-go. They're like, oh, wait a minute. If we have all the cups sitting on the table, right, and we say the character goes and just reaches for that cup, the guy next to him has the cup in the same spot. Okay, so what if they both go to reach the cup at the same time, right? Now, you've got five or six doing that, it looks like the Rockettes, right? So instead, they said, well, what if we just did something simple like this? And then this one's over here. Okay. So now what they're doing is, within the goal and the vision for what it's supposed to look like, they're actually going in and driving the feature themselves as well. Making decisions that head towards that goal. And because they're doing that on a daily basis, we're able to head off a lot of these issues that we may have had previously, where, say, something would fester for a couple weeks, or someone would do what was on the task list, but not necessarily what was fully intended, right? And because those people are now working together as a group, they're like, hey, this works, but that thing doesn't. Oh, okay, I didn't realize that was a secondary portion of this. Hold on, let me work on this for a little bit. I can get back to you with it, right? And it makes it more exciting, and more where the teams are able to almost self-manage themselves, and we're able to actually direct in a more efficient way.

Yeah, it's so much more efficient. And a lot of this might have been obvious to some people, but the company again has matured and grown in a pretty organic way. So we have to make these things very clear, and like, this is something that needs to happen this way. This is more efficient. And just like we iterate on technology, we have to iterate on our production process. And that's really what we're doing. And even right now, we're sort of half and half, basically. We've got very linear pipelines, like the ship art pipeline, for example. That is like an assembly line right now. That is perfect. And you're not going to make a feature team or a goal team for the ship art pipeline, because things are very, very solid.

But they are in their own goal team in a way. Like, for instance, the animators in ATX worked very closely with the ship art team to create a system that would allow the ship art team to have a lot of flexibility in the visual look of the ships. But for instance, if I put a stair here, and then on the other ship I put a stair here, did you notice the difference? If we can keep the places that the player connects to a ship in similar metrics, I think there's something like 29 separate metrics for just entering at the moment, right? We are able to spend time creating solid animations for those. And the ship team is able to go and create as many ships as they want. They utilize portions of those templates. And it enables us to be able to get ships out sooner to you guys than we did before. And now that team moves like this. They know how to do this stuff inside and out. And it's like, hey, we got a new ship coming in. Cool. Which metrics do you want to use? We're going to use this for the enter. We're going to use that for the exit. We're going to use this cockpit thing. Not a problem. We can set this up. And it's so much smoother, and it's a perfect example of a team that initially had all these disparate parts that, as they started working through this process, now they've got a solid process.

And don't forget, this is all people in different studios, too.

Yeah, absolutely. Across different time zones, so it's incredibly complex. LA, Austin, and UK.