Andreas Rohr

Yes, so all the heads that have been in the PU previously are going to be in the new one. There are actually going to be some in the background that you don't see, but you can access through one of the customization modes. They're partially activated because we're still polishing their face rigs, so we haven't fully activated them. And obviously, I think at CitizenCon we've mentioned, like Forrest mentioned, that we're working on adding 60 more heads total to the customizer and we're still working on that. But they will come in the future when they're ready.

And then just to kind of follow up on that, not only do we have all the heads that previously existed, but what we've also done is we've given them a visual facelift as well. We've calibrated their skin tones, we've increased skin detail, we've balanced maps, specular maps, gloss maps. And so not only are we kind of improving the pool and all this stuff, but we're also giving everything that existed a great facelift. So I think it'll be awesome.

Yeah, that's been really great. It's been cool to see everything that Forrest has been able to achieve with the old skins and see the difference that it really makes to give these updates to getting the correct tones so that we can use this new color picker system to allow the players to pick just from there, not making it overly complex.

And this means that we can also now apply any skin tone on top of any set of skin textures. So if you don't like the positions and stuff of the wrinkles in one skin texture from a previous character customizer but you like the skin tone, then you can mix and match now a bit there.

Yeah, so as part of the new blending UI, we have made it so that you have this full list of heads now. Instead of just blending from one to the other, you get a list where you can easily just click anywhere. So you can click on, for example, someone's nose and just drag on the nose, and that turns into this slider where you immediately get more and more of the DNA from that nose. And for that full head blending, we have made it so that when you hover just on the outside of the head, then you will see that the whole thing gets hovered and then you can get all of the DNA from the full head.

So all of the existing functionality plus all the new sculpting and stuff and then makeup and all that stuff.

Yeah, but I guess some of the functionality will be different in that, as you hopefully heard Jacob say last week, it's doing the kind of same things, but in a nicer way, whereas we have color pickers, et cetera. It's just way easier. Hopefully it covers everything, but it's better.

Can I quickly reiterate on the blending mode versus sculpting mode? Previously, we've had a version of blending, but that was actually rather tedious to use. You always had to go from the slider to select one of the regions, go back to the slider, change the head, your source head. And right now, you have the list of heads. And you can, as Gabriel mentioned, click right on the feature, right on the nose, and start dragging it in. And you see it being blended in. You can drag it back. You have undo, redo functionality in there. So even the existing mode is going to be much more convenient to use. And it has its own strengths. And I guess Gabriel can talk more on the new sculpting mode. Don't take anything away from that.

I'll start off with a disclaimer: no promises. Is it technically doable? Yes, it's technically doable. One thing that we do like to do internally, especially with the physically based rendering engine, is we like to look at our assets in multiple lighting conditions just to even review them. Because whenever you do something physically accurate, it should theoretically work in any lighting condition that you put it in. And that is a way that we do reviews, and we are using the tool, the character creator, internally to also create characters and stuff. So I could see how that could be something that we would also want internally, and again, technically possible, but no promises. I just want to throw that disclaimer out there.

And for the facial animations, we will keep the FOIP functionality that we've seen in the old character customizer as well. So that's one way of seeing other facial expressions than just the default animation. But we also have, just when you finish making your character and move on to the review page, you will see that they do a little animation. So you will see a little bit of personality on there as well. But as Forrest said, we do have some dreams of what could be done here for sure. I remember me and Forrest discussing some things back in the day. I don't even want to mention them because I don't want to sow any seeds of hope to anyone here. We had some stupid ideas.

If I may, I'd like to add one thing regarding facial animation, or the face breaking afterwards or looking different afterwards. So one of the, I think, quite important things and the things we are quite proud of regarding our face rigs is that they animate properly. In certain single-player games we make, they need to show up right next to AAA actors. So every face can animate in a way that will allow you to lip sync. Now with FOIP, obviously, that doesn't always come through that way. It can produce less than optimal results. So at some point, that's got to be revisited as well. We're quite aware of that, that results are sometimes a bit involuntarily funny. But theoretically, the face can emote just fine outside of the customizer. Obviously, the lighting quality in the levels is not as well designed as it is in the customizer. We've had lighting artists really perfect the lighting in there. And in some parts of the level, it may look super great as well. In others, it may look more average. But yeah, that's expected.

No, they have physics for sure. I'm probably not the best guy to talk about the physics of it. I'm sure Andreas and Forrest can tell a bit more about this. But yeah, we currently use the pendulum system. Star Hair is what we're aiming for. And right now we're using an intermediary way of basically simulating the hair. But yes, in short, the beards and the facial hairs will have the same type of physics applied as the hairs do. So, yeah, definitely.

Yeah, especially the full beards, the large ones, will have some degree of motion and collision with the chest, I guess. Using, like Andre said, the kind of legacy pendulum system that still gives good results. Ultimately, we're aiming to implement StarCloth and everything because it's going to look better. But that is still in R&D phase. We still need to flesh it out together with... Yeah, StarCloth, Star Hair. It's pretty much the same group of physical solvers. The backend, the solver itself, is already quite powerful. We need to do a lot of tooling around those so we can efficiently implement dozens and dozens of assets with them at some point. And that's always a longer process. So initially you make a prototype, you get the core solver, for instance, up to a certain state that you can show, I can reach this and that quality, but then it's always an extra step to develop tools, pipelines, to allow artists and technical artists to work with that efficiently and achieve good results in a reasonable amount of time.

Yeah, and I can add to that by saying that even the construction of the hair itself needs to be done in a different manner if you're looking into using the Star Hair and StarCloth that we're aiming for. So it works well with the pendulum right now. It is made that way. It's optimized that way. But if we wanted to really push the simulation, the hairs themselves need to be constructed artistically in a different manner, basically.

Can I just say how cool it is when you rotate your character and you see the heads in the sidebar with even those hairs all moving? So you have so many different heads with so many different simulations happening.

It's something we're actually pretty proud of. Something we do that other customizers don't is that we update the small heads on the side to match your own appearance. So the heads that show the hairstyles should all match the appearance of the main head that you've customized, so you already get a preview. The real-time thumbnail, right?

Yeah, exactly. It's not just a picture. It's a real head fully simulating. I mean, it's impressive.

I can get that. So basically, as a game, we would have to limit the hair at some point. The limitation that we currently impose on ourselves was to do it up to the shoulders. And this is an agreement that we've done in order to minimize clipping and in order to function with all the things that the character needs to work with, of course. Now, it's not that we technically could not do the hair longer or that we could not simulate it longer or whatever, but at some point we would have to limit it. I mean, how long would it go? If we had it up to the shoulders or a bit past the shoulders, would you like it up to the waist? And if we do it to the waist, would you like it up to the feet? So at some point we would have to draw the line, and right now that's where the line is drawn, basically. And that's it for the size that we are achieving right now, that we want to deliver right now. Though, hat hairs and mask hairs right now, I think I can safely say this, that by 3.23, you're going to be able to basically choose a long hair, put a hat on, and the hair will just adapt to it. It's not actually an adaptation of the hair to it. It's actually replacing the asset. That's what we did. We created a specific asset for the hats and for the masks. So basically, to answer your question, yes, from 3.23 onwards, if you choose a long hair, you basically put on your mask or your hat, and you will see your hairstyle, not just something else. You see your hairstyle deformed to that specific hairstyle, and with physics and all. And so everything still works. It's just compressed by whatever the object is that's tightening you, basically.

Yeah, I saw this the other week and I was blown away. It looks so good.

Might be worth adding that even just the variation that Andre described kind of multiplies the number of assets that each department has to deal with. So Andre's department and Tech Art, maybe they start out with, I don't know, 30 hairs or 40 hairs. Then they need to conform them; let's say they groom them on what we call the male standard head. They need to conform that to the female head. That doubles the number of assets. Then they need to create the hat hair variants that, again, doubles that number of assets. And so you can imagine how that number would grow exponentially if we didn't apply very strict rule sets to where the hats are allowed to sit and where the volumes are, what the restrictions are. We have to limit those. Otherwise, it's an unbounded amount of work.

Yeah, Andreas, just to confirm what you're saying, basically each hair asset that we create, you can multiply it by four, or no, by more. It needs to go into the male, the female, the head hair... Because you have female head hair and male head hair, so each hairstyle just gets multiplied by a lot. This is what is actually taking a bit more time. We wanted to include this stuff. So we can create hairstyles really fast now, but now each hairstyle needs to just go with all this stuff in order to be implemented correctly and to deliver everything. Because we want you guys to use your hairstyle and then you put your hat on and suddenly you see your hairstyle with the hat, and you remove it and you get back to your hairstyle. So this is a very cool achievement that we just got.

Yeah, it is. It's amazing. I know the technical team worked with Andre and basically we have templates, and within those templates, it will represent a hat, and then the hair needs to be kind of adjusted based on these templates. But every time you have a new template for a new type of hat or whatever, that's an additional asset. So they could definitely stack up quite a bit. So kudos for supporting all of these variations of outfits. It's cool.

Well, E-Tech, in a company of 1,300 people across five studios across four countries, there's literally Andre and the people here who are working on this right now. So it's a pretty small, dedicated team who are spending 20 minutes out of one afternoon talking about it. So I think when you put it in perspective, it's not so bad.

I'd also like to emphasize on that. This tool was developed, obviously, for the backers primarily, but was also developed in a way with some extra debug UI things you can activate as a developer, on top of the UI the backers get. So we as devs can customize NPCs with the system for both the PU and Squadron. And that's why it's an important tool for us. And the fact that it's so powerful is enabling us to make NPCs quickly that look unique and look special and high quality.

It's a point that was somewhat glossed over in the show, but this is literally the same tool that we're using internally to make NPCs for both Squadron 42 and Star Citizen. So this is work not just for the players, it's work for us, for the game, that we can share with the players.

And developers are way, way more demanding than the players as well.

Yeah, for Squadron, that's the other thing: we've got hair and the hair needs to look great, and Andre's making some great hair and that hair goes into Squadron. And then we also, like we said, wanted to get everything propagated to Star Citizen. So it's like, why not put it in Star Citizen? We already need it for Squadron. Let's double our wins and get it over to SC.

I guess I'll take that one. Regarding body customization, it's simply a super hard problem to solve. It consists mainly of two problems that I think we can differentiate. One is body shape customization, so that's more about being able to blend between a skinny type and a heavy type and maybe a super, on-steroids bodybuilder type, these kinds of shapes. We could technically do this already with the way the DNA system works currently. The question is, what happens with the attachments? And for the head customization we've developed runtime wrap deformers, which can scale helmets, which can conform beards to changing face shapes in a way that keeps the art intact and keeps the art directors happy, no distortions and stuff like that. That works on the face. The same deformation tech applied to bodies would break in many cases. So while we can already morph the body shape, we cannot conform the attachments in a good way right now. So that's one of the two big problems we need to solve for that. And the other thing is regarding body height changes and limb length changes. This is another super hard thing to do because that affects all animations in the game. If you imagine a character holding a pistol or even holding a mug and moving that to his mouth to drink something: currently in the game, we have tens of thousands of animations that have been authored with so-called forward kinematics. Every bone in the skeleton defines its rotation over time, and that's how the arm moves towards the mouth, and then you get the impression that they drink something without spilling anything. Imagine we just scaled the length of the neck, for instance, or the length of the arm; your mug would end up somewhere else. So you need to retarget that animation at runtime, and while we already use inverse kinematics in many places, characters pushing buttons, reaching for things, there's a lot of procedural elements to our animation already, doing that for the entire body and for all animations requires massive changes to the animation backend. But yeah, we have all intentions to tackle those problems. Personally, I have some good ideas for how we can fix the first problem about the conforming of the attachments. And then we have smart animation programmers, Ivo, that I've been on stage with in 2018 in Austin. He's somebody that has the knowledge and the brains to fix that hard retargeting problem. But it's going to be a lot of work. But it'll be worth it because everybody having the same height is not great. But I think as a first step, we could, for instance, tackle just the shape changes, which would already give us some big gains.

Because we kind of already do it with the head, right? When we're in the DNA customizer and we make the head larger, in the game you're going to put a helmet on. Well, that helmet's not going to fit anymore. So the technical artists go in there and they mark up these assets to be able to deform based on the deformation of the DNA head. But then obviously that's quite limited here, limited movement. Introduce a whole body set, and it's a lot of data that you're going to have to mark up and stuff like that. So that's another one of the challenges. But I think the head was kind of the first step a little bit. And now we're moving into the difficult part, right?

Yeah, and body customization immediately has implications on gameplay. When you have a larger character, that's going to need a larger hit volume, right? I hope not all players will then shrink themselves. So maybe you want to balance that with, yeah, larger characters can run faster, carry more, stuff like that. But it will affect a lot of systems once it comes online. But one more thing I'd like to add regarding what the state of this is. So initially, our DNA system, which is the system that allows us to morph the heads and the faces and conform the attachments, was developed specifically for heads. That was around 2018, 2019. We've then used it on heads, and there have been systems that have been built around that system. So the code making use of that backend grew and grew and grew. So at some point, we already knew we wanted to tackle bodies. But we have a system that is very bespoke to faces. So I've worked for almost two years, not full time, but half time, on a refactor of that DNA system with the goal of getting it into a position where we can actually start tackling the body customization problem. So that was finished last year. And there are other priorities, like the gene pool head extension giving us 60 more heads, and the rigging pipeline that needs to be set up and the tools around that in order to even be able to create fully rigged, nicely animating heads. That also takes a lot of time. So yeah, it's a resource and priority issue.

So it sounds like the 10-cent summary version of that is that different body shapes are something that we want to do, something that we're pursuing part-time on the side in between a whole bunch of other higher priorities and stuff. Nobody should expect it soon, but it's definitely something that is progressing, that we're still investigating, and something that we very much want to do.

We want it, it's hard.

I would say the first thing that I think of when I hear that is, first of all, awesome. Second of all, it's solving the issues of what a camera could create. So, for example, when we do our skin textures and stuff like that, there's a lot of requirements, especially again with physically based rendering; we can't have shadows in the diffuse map, right? We have to extrapolate the normal information or height information, essentially spec cavity, specular occlusion, stuff like that. So the real question is, let's say the camera can take a picture of your face to eventually bring in as a texture, not going into any additional data that could potentially actually shape a head or something. But assuming we're just talking about textures, I think that those would be the biggest issues to solve: how do we clean up that data in order to get it game ready by processing the textures, which currently our artists do, and Claire Engle has done for us. So that's just the first thing that pops in my head when I hear that.

Okay, so I think there are some ways we could go about this. I think once we have all the 60 heads from our gene pool extension, once they're in and we have their textures, together with the skin tone tinting that we can already do, I think we have enough variety in terms of the textures, I would hope so. In terms of the shapes, it's going to be a problem of finding a way to recognize the features. There are already some SDKs out there that we could use; face detection itself does that to a degree. It shows you where the eyes are, where the mouth contours are. So you could leverage that to a degree, recognize those features, and then try to find samples in our gene pool that can approximate this. So we have a vague idea, or a pretty good idea, of what needs doing. It would be super cool for the players to use that. Even for NPC creation, you just show a photo, and the software would create that face or approximate it reasonably closely. But again, priorities. It's not very high up on the list, I guess.

I keep thinking about it now. And then even on a simpler turn, just with the skin tone itself, if you knew what the offset is, because what we do with the skin tones now is we calibrate them, we balance them to make sure they're accurate to the real-life talent. So if you did have mid-gray or whatever in the shot with you when your image got taken, or some kind of Macbeth chart, obviously that would increase the... whatever. But theoretically, if you did have some kind of mid-gray or something, we would know some offsets on what would actually be happening in your environment to bring you back to a neutral skin tone color that could then be applied using our current palette, at least matching the closest possible. So I don't know. It's interesting. It's a fun one to think about for sure.

Yeah, Andreas mentioned the thing I was thinking about: using Faceware, not for the textures, but to determine the location, the spacing of certain things, and just to give you a base shape of the face that would be somewhat closer to you that you could start with. There seemed like there's potential there to me.

It's a good idea. It's a great idea. I mean, you've got to start somewhere. You've got to start trying things.

I mean, one thing we already do is with face sculpting, we take a position of a point and we use that to decide what heads to blend between. So once you've got the positions of those points, through however we want to detect it through the camera, then it's actually solving the same problem.

I mean, they go as far as the processed head scans will go. And already I'm thinking, because right now we just have the group of heads, 9 to 15 or whatever, that pool is going to explode. It is going to get so big and so awesome so soon. I can already think of a head that looks very close to Danny DeVito. So I think you're going to be able to represent yourself in the near future.

Yeah, it's funny that this question is asked, because when we started out with this gene pool initiative, which is closely related to the DNA system, having more variety in there, we made a big reference matrix of heads and different head types that we wanted represented. And most of them have obviously been celebrities, public figures, politicians, all kinds of people that pretty much everybody will know and recognize. And one of them happened to be Danny DeVito. And yeah, like Forrest said, we've scanned all kinds of different shapes. I wouldn't want to say extreme, but the DNA system is the most efficient when it covers all the edge cases and then can interpolate between those. So we have very, very slim and skinny and narrow faces. We have super wide and heavy faces. We have very angular faces, super masculine. We have all the types you can imagine for various ethnicities. Yeah, I just need to process them all. Once that comes in, you'll be able to approximate most people pretty well, I would say.

Yeah, I mean, it's amazing what you can already do now. And this is nothing compared to what you will be able to do in the near future, for sure, most definitely. And Andreas has actually spent a long time, before we even did the scan, going through all the heads to ensure that we have the best range possible. So massive work went into ensuring that the pool is as diverse as possible, because we want to be able to create everyone.

I feel the need to clarify that Danny DeVito is not part of the project. He was not scanned. It's simply like you see on any kind of inspo board. We want faces like Danny DeVito's.

I think it's also not a CIG stance that Danny DeVito would be cursed in any way. We'd love Danny DeVito.

I mean, obviously that's your head. That's what I would like to add to the character customizer. Your head, Jared.

I want to add more randomization, right? So we've got some pretty basic randomization being rolled out with 3.23, but stuff like hair color and things: right now we just sort of randomize a simple natural color. It'd be super cool to have some advanced stuff with dye and wear and stuff. Yeah, I like randomization. I like hitting that button. And I want to see more cool results.

I think just for the head scan library, it would be awesome to be able to scan your own face, obviously, and then maybe have it in the bank, adding to that bank so someone else could get my disgusting nose on their face. That would be something.

But you bring up a good point. Andreas was talking about this in the pre-chat. Scanning the face is only one part of the process of getting a face in here. There's still a lot of processing work that gets done after you sit in the booth and get your stuff done.

I'd just like to flesh out the preview page a bit more, the one where you can see a full character, like the stuff that was mentioned: playing different animations, different lighting scenarios, even swapping different outfits or something so you can see how you end up looking, like a fully fleshed-out character, together with your facial appearance and everything else that you customized. I think that'll be really cool to have more options there. But that can be near-endless spending of resources there.