Star Citizen Live: Audio Team Roundtable

I think they're referring definitely to the ship computer voice, the ship computer kind of telling you this state's happening.

The ship computer voice, or just a "red alert, red alert"?

Yeah, I think a lot of it has to do with getting that situation to really occur and give us enough game data to make that happen. I think right now a ship's life is rather binary. You're alive or dead. And there's a lot of plans in there to have a lot of components take damage and failure, leading you to have an imminent death where you have X amount of time to really have this drama unfold. And in order for us to really gain on that kind of ship computer use case, we really need to have that drama happen for us. But also, especially for a ship-wide announcement like that, the bigger side of it is to have, and we've talked about this before too, that system within your ship so that you can hear these ship-wide announcements from anywhere in the ship. And we're getting closer to that. Actually, we have that tech at its base core working right now and we're using it in [...]. So it is something that's happening, but to get it to the point of having it in all of our ships, that's going to take a lot of backlog work and things like that. Plus, the tech is also in its early stages. Early days, so we are getting there. That is a vision. And I do like the idea this kind of question poses, where the captain can kind of trigger that use case as well, just like battle stations. I mean, how would you really determine that other than the captain saying, "We think this is a battle stations scenario, so I'm going to trigger that as the captain"? And I think that's a pretty cool idea. I'd like to make that happen as a user myself. I think that's pretty cool.

Yeah, I was just going to say, because everything we do is systemic. So from a tech side, we have a systemic VoIP system where people communicate with each other. We have a tannoy system that, although we haven't marked it up in all the ships, is proven to be working. And we have a ship-wide audio channel for the VoIP for communication. So it's not a massive leap to then say, let's stick a microphone on the captain's seat and then give you a button you can press and then you can talk to the tannoy system. Because everything's systemic, there's not a great deal of work involved in doing that from a tech perspective. There is a huge amount of markup required in terms of making sure that the tannoy reaches every extent of the ship and things like that. So yeah, it's a great idea. I love the idea. And if you're talking about having a player drive it, a player sets the alert state and a player talks to the rest of the ship, then yeah, it doesn't sound like a huge leap from where we are right now in terms of tech, but obviously we'd have a huge amount of design and work to follow up with.

And I do want to follow up. When Bob said that some part of it's coming in the next patch, he's not talking about the player-implemented part. It's a part that we're using that we're not going to tell you about yet. Discover it yourself.

I certainly can, because I'm currently working on it. So right now I'm working on some additions to the room system in the game. Those additions are not specific to audio. They're going to underpin how this audio propagation is going to work, but they're going to be used by other systems too. The AI team's interested in what I'm doing, and there are other systems that would definitely benefit from this thing. But I'm in the process of making it possible to efficiently traverse through our room system. And by efficiently, I mean I'm talking thousands of queries per game frame, so that we can make sure all the audio knows which path it should be taking. Then also we can help to inform the direction that AI should be taking to find a particular room or place. So it's got a long way to go. I'm not going to say anything about when it's going to finally make it into the game, but I am actively working on it, and it is my number one priority right now. So obviously your priorities shift all the time, but yeah, this is very much in progress. Personally, I think this is one of the biggest things I want to see in the game from my own perspective. So yeah, I'm really pushing hard to get this working.

Yeah, so spatialization is something we're currently exploring. So we're looking at enabling spatialization in Wwise, which will allow backers to make use of Dolby Atmos, DTS, Windows spatialization and stuff. The current version of Wwise that we're on doesn't permit that yet, but when we migrate to the new version of Wwise, then it's something we can continue to explore. But it's important to note that when we enable that, the Dolby Atmos, DTS and stuff, that's third-party software. So although we can enable the feature, it will be down to the backers to choose which versions they want to go with. People might just have aesthetic preferences for which spatialization tool they want to go with. But beyond that, we still have a keen ear on surround support. Now, it's been a bit challenging working from home, because we've all been working from home since March last year. And most of us are working in, some have spare bedrooms, I'm currently in my dining room and stuff. And we're doing our best to work and build the acoustic environment, have decent playback and stuff. But the practicality of it is we don't all have room for 5.1, 7.1 systems. So that is a shame, but it's something we still have a keen ear on. But also beyond that, so we've got spatialization we're looking into, keeping an ear on surround sound, but we're also having a focus on fake HRTF, which is a head-related transfer function. A couple of our really talented sound designers, like Fran Delpia and Adi Kelch, are really spearheading looking into attenuating sound based on the position of the emitter relative to the player. So what a head-related transfer function basically is: things sound different depending on where they are around you. If they're in front of you, behind you, above you, below you, the ear attenuates the sound slightly differently. And we're looking to, in simplistic ways, try and telegraph that to the player. So when you're in a dogfight, for instance, you can kind of hear the sound changing and you can learn that, hey, that's above me, that's below me, because we're not just dealing with a flat plane like an FPS, where things are in front of you or behind you or slightly above or slightly below. If you're in 3D space, in a dogfight in space, then everything could be on all axes at the same time. So yeah, we understand it's massively important for a game such as Star Citizen and S42. So it's definitely on our radar and we are looking into it.

Yeah, I mean, I've had chats with Dave about this a few times, and I think this might have even come up in previous Q&As, but the idea is definitely that we want to have that, like a pilot has in a typical aircraft, where they get other chatter on an open channel. And that concept is something that we really would like to have for Star Citizen as well. I mean, the main thing that really holds up something like that is just pure content. To have conversations like that back and forth with many different actors, enough to make a feature like that very interesting, takes a lot of recording time. So it's something that's going to take a bit of time to get into, but it is on our roadmap. It is something that we really want to achieve, especially for interactions with the ATC and having people kind of overhear other people parking and things like that. We want that kind of feeling of life going on around the landing zones. And that's really important to us. That life element is, in general, really something that we're putting a lot of focus on as of late.

One cool little thing I was speaking to Darren, the audio director, about is, in the future, looking to go through jump gates and wormholes and that kind of travel. It would be really cool if we could, with permission, record the VoIP input of any player and then play that back whilst they're in these altered states traveling through wormholes, as if it's warping the reality of time and space. There's many hoops to jump through before we even start exploring that, but I think it'd be really cool to hear back some comms, all processed, reversed and stuff, just to give that ethereal feeling as you're traveling interstellar through this medium.

That would be very cool. I can feel my backlog growing as we speak.

Well, I'll take that one then. So what we'll do is we'll look at the manufacturer and see what the brief is about those. Are they aligned with military? Are they more aligned with alien tech? Are they cheaply made? Are they well made? Then we'll look at the ship specifically. Is it a fighter, a hauler? Is it meant to be focused on mining? Anything in particular to that? Then we'll have a creative kickoff and pull in references from other ships that we have and see where there's any parity and stuff. And we'll generate a style guide and then start generating source, look at references, and then start to implement them and then tune them in game. So we'll get them in Wwise, because a lot of the sound effects only really come together at runtime in the ship. So when you're flying a ship, it's kind of like you're playing a musical instrument in a way, because we'll implement all these different layers and one-shots, but it's actually the ship's movement and input that triggers all these. And we have different RTPCs that will attenuate sounds in pitch and volume, high pass, low pass and effects based on the action, the input from the ship. So yeah, at a high level, we go: this manufacturer typically does military ships, so everything sounds really robust and, say, aggressive. And this ship is a fighter, so everything needs to be sleek and fast and not too bulky. And then start tailoring the sounds to that. And that doesn't just pertain to thrusters. That could be anything like moving parts, ambiences, any kind of component work and stuff like that. So yeah, we want to try and establish a brand identity for each manufacturer, and then with each category of ship, make sure it sounds appropriate. So you've not got a fighter that sounds like a giant whale of a cargo ship, and vice versa.

Yeah. Well, the indiscernible dialogue that that kind of question really alludes to is walla. But in general, liveliness kind of comes from having lots of layers. You have a background and a foreground and a middle ground, or even more than just those three layers, for dialogue. And it's a little bit more complicated than you might think. In general, the walla, as it would sit, is a very static looping bed. And it doesn't really do much to help sell when the population of a room changes, if it grows or decreases, or if the mood changes, or if the player has an impact on the mood and pulls out a gun in a crowd and starts firing. Don't do that. But if that were to happen, the crowd would change, and you would want to hear that. It's quite a more impactful system. And we actually have a real goal to achieve this. We want to do a dynamic walla system that actually accounts for players coming and going, the size of the population, even the economic status of what the population would be, or their mood, and how things change like that. And that's something that's really heavy on our radar. But that's just the walla, that's just the background. We have the foreground, which I think they're also alluding to in this question, of actually seeing players having a conversation. And a lot of that is very dependent on behavior, AI behavior. And again, alluding back to what I said before about conversation trees and that chatter, that's a lot of content with a lot of actors. But mainly it depends on behavior and acknowledging those two characters, acknowledging that they're in a shared conversation. That one person said this topic, and now I need to comment on something that makes sense from that topic, and then they want to rebut after that, and you want to have at least a three-way back and forth of a conversation. And that takes a lot of effort, and it's a lot more complex than you might think. And then that again is just the foreground. Then you have the background needing its system and the foreground needing its system, and then, well, what do you do about gluing those two together? And this is something that we've been really trying to achieve more recently, because it's like a filmic kind of thing where you're trying to showcase sound unseen. So we don't really want to allude that there are people actually talking right there with you or whatnot, but we want to hear that there are conversations going on about you that aren't part of that static bed. And so we kind of have this: it doesn't call out to you, but it doesn't wash away. It just adds some pepper and some spice to what would otherwise be a static, bland bed. And that's what you've seen in the cargo and refinery decks. You're going to see it in our new location for Orison. We're trying to achieve a bit more of that. And that's going to help glue some of the stuff together while we still really need to put a lot of effort into these systems that are a bit more complicated than you might think. But it's definitely heavy on our minds right now, because that's a big thing for us, feeling that life in a space. Even if we don't have the tech or ability yet to do that, we want to try to achieve it in any way that we can, even in the interim. So we're doing our best.

Yeah, just to add on the end of that, the reason these things take time is because we try to do everything systemically. We try to solve the problem for the whole picture rather than solving specific cases by adding beds that suit this particular area. And that would mean that when the tech's not ready yet, certain areas, like you mentioned in the question, can feel a little bit like they're lacking some life and some colour. But what we're looking to do to solve that is not to just artificially throw that life and colour in, but throw in a system that allows us to colour whatever space it might be. So it's not just about walla, it's not even just about characters. It might be that you can walk into a room full of, say, hospital beds, and we could have a system that allows us to recognise that there's a lot of hospital beds in this room and gives it that kind of calm audio background that a hospital room has, because it's found all these objects in it. This kind of idea. So we're looking to make a system that basically allows us to colour every part of the universe.

And again, looking at that big picture of content, we try to get a little bit at a time. We're getting this content that's going to take a long time to get. We are getting a little bit at a time, and it just takes a lot of time to get those kind of big dialogue trees going.

So that's me. We've got early concepts that we're looking at, but we've not really dug into it yet. I think the first one is going to be the space whale in and around Orison, right? So that's going to be really interesting, because it will be a big collaboration between sound design and dialogue design and code, because we're like, we need this to make a sound, can you build this implementation so we can hook into it? And then speaking with Bob, it's like, what is our baseline for creature sounds? How big are they going to be? What are they going to sound like? And then just start building some mock-ups and then refining it. So yeah, early days, but nothing concrete yet.

It's also important to acknowledge the behaviors of what these creatures can do, because then you have a mixture of different kinds of things: can they eat? Can they attack you? Are they going to be sleeping or quiet? It's really about capturing that essence in their emotion, the different things that they can do. I'm not sure what the Orison whales can do other than looking really pretty. I don't know. But as we expand, especially expand on the fauna, that is kind of the biggest thing: looking at all the different things they can do and making them really feel like a real living creature.

It's a really good point about the behavior, because you almost have to establish a language, right? Can it sound angry? Can it sound sad? Can it sound happy? Can it sound aggressive? Are there any gameplay cues that we need to support? Like, it's about to attack you. What's that gameplay loop? Will they make certain attack calls before they come after you or something? Or if it's a hunting mechanic, I'm not saying there is one, but how do we respond to that with sound? We really need to understand what the whole gameplay loop requirement is of said fauna.

Sonic boom. It does come up every show, and it's tricky. We understand why people want it. So we've made an active decision to not go along with the speed of sound in the game. That's primarily because as soon as something happens in the game, you want the audio feedback there for gameplay reasons. So as soon as something happens, you want to hear it. And what a sonic boom is: if you've got, say, an observer and an aircraft moving at or faster than the speed of sound, the sound builds up at the front of the aircraft. So it's basically a wall of sound that persists. And as the aircraft passes by you, the observer, you perceive that as a transient boom, but it's just a wall of sound passing your singular location. So what we do with ships specifically is we have NPC proxy passby loops, which, in atmosphere specifically, is the kind of rippage you normally associate with jet engines and whatnot. And then we have bespoke flyby sounds that are triggered when certain conditions are met: the ship is moving at a certain speed, which will vary which flyby sound it plays, and then how close it is to the player. Now, to scratch that itch for sonic booms, we currently have quite heavily designed flyby sound effects that are quite sci-fi. For atmospheric flight, what we could do is look to introduce more typical sonic boom elements in these passbys. So there'd be a bigger delineation between a passby in space versus a passby in an atmosphere, just to give that feel of a boom. But we don't typically honor the speed of sound, because that would be massively expensive to calculate, for one. But that's not saying it's completely off the table, because we might play with it creatively, if there are certain contexts where you want that delay between visuals and sonics just for impact, say if it's in a cutscene or something. So we are aware of the mechanic. It's just something we try and move away from gameplay-wise. But I don't know, Graham, just blue-skying, how expensive do you think it would be to create a system that would accurately map the speed of sound over distance in varied pressurized environments? Because the medium would determine the speed of sound. Sound moves faster in water, the denser the medium, than it does in air. And then you have to account for the different atmospheric pressures of the different planets and how thick the atmosphere is at different altitudes. And then send it out to 100 different clients at various points on the planet.

Yeah, I mean, it would be quite a large undertaking. There would be two possible approaches. One would be to delay the audio itself, so adding a delay step to the playback of the audio, which would be a little bit expensive in terms of CPU, maybe not so much, but it would use a heck of a lot of memory, because you'd have to be storing up all your audio and waiting to play it when the time comes that you should play it. And that would be applied across all sounds, so that would be prohibitive. The other approach would be effectively to delay all the events that affect the audio. So if a weapon fires, you'd store up the fact that it had fired, and then at the point where the delay time is up, you would then fire it. But again, you've got the same problem of having to queue up all these actions that happened and then play them back after a time. And it would just be a hugely complex undertaking. But I think really that's not why we don't do it, because we would be able to do it. We would find a solution. That's what we do. I think the big problem goes back to what Phil mentioned earlier in this call, which is that for gameplay reasons, it doesn't make sense. If somebody's firing at you, you want to know. If audio is the thing that tells you that somebody's firing a weapon at you, you want to know straight away. You want to know you've got time to spin around and see who it is. And if you have real speed of sound delay, you can be dead before you hear the shot. And that doesn't make for a fun game. It might for some people. Some people might be big fans of sniper games and things like this. But our game does a lot of things, and to focus on that style of gameplay, I think, would probably be wrong.

That's a good question. I think we've skipped over the jukebox question, and if you don't mind, Jared, I'd like to step back to it.

No, do it with the jukebox. It's the same thing as the old subscriber jukebox.

Yeah, so we're aware that the jukebox is broken. I think what I certainly wasn't quite so aware of is how popular the idea of resurrecting it is, because it was quite high on the list of questions. That same tech could certainly be used for playback of music in any ship. We would have to suggest the idea to design.

Just the Carrack.

OK, just the Carrack. OK, yeah, that would be easier. Yeah, I mean, the jukeboxes are a flair item, and anyone who knows me knows I'm a big fan of flair. So we should be able to get that working without too much trouble. It's been pushed down the schedules due to priority for quite some time. But I think it does seem that people want it, so we'll certainly review that next time we come to look at the schedules. And I think in terms of adding maybe a little music player that uses the same tech and can be placed in the ship, that's not a huge thing to ask for, because the tech would be the same. But obviously, we would want to talk to people like designers.

But with that, we have to be careful, because if we enable people to effectively stream music to other gamers, then... That's not what this is, let us be clear. This is just playing music locally. Client only, only for you. You wouldn't be able to stream any music to anyone else, because then you infringe on copyright and licenses and stuff like that.

Now, that was actually the specific wording: "let me play music for my crew." So they are talking about streaming music.

Ah, that's a good point. OK, let's rewind everything I said, and I'll just say no.

I can take it from a sound design aesthetic point of view. So Pyro's palette is massively different visually to Stanton. It's a lot more turbulent. And with that are going to come unique challenges. The planets are going to be different, the moons are going to be different, have a completely different vibe. If you've got a Stanton palette and you've got a Pyro palette, they're in themselves their own beasts. And then you establish: what does Stanton feel like? What does it embody? What do you want it to sound like? And then the same with Pyro. And then filter that down, similar to the ship thing, right? You build a style guide of what you want Pyro to sound like, dangerous and foreboding, and then how do we filter that down and realize that on this particular planet or that particular planet, and then take some specifics from each of those. It's going to be a load of fun, because there's quite a lot to work with, and we've learned a lot working on Stanton. So the Pyro stuff is really nice to get our teeth into. But it's new, uncharted territory. The tech and tools have improved and grown in scope since I started on the project and working on Stanton. I think the actual ability to put sounds in and tailor them to the experience is much improved. We have a great deal of fidelity. But I think the main challenge is really having that feel, a shift between Stanton and Pyro. So when you're in Pyro, anywhere in Pyro, space traversal or actually on a planet, you feel like you're in that system versus Stanton. So hopefully we can achieve that.

I wanted to add too that there's a discipline here that's kind of missing some representation. Music, I think, will have a really big impact on selling the emotional aspect of where we are, either Stanton or Pyro. And that's a very critical element, I think, when feeling out a different space. In a lot of cases, the way music affects you in that kind of sense is meant to not be very obvious.

Yeah, so looking at Crusader and Orison, the landing location, the sky city, we've obviously been syncing with art really, really closely. And wind is going to be a big hero character of that location. So yeah, we have wind and weather and stuff in other locations, but we'll be paying particular attention to crafting the wind and the ambience of that location to make it sound good. Wind is essentially just shaped broadband noise, so it's going to be slightly challenging, but obviously there's the actual landing location itself to tie into, and that's going to have a lot of opportunities for nice audio tidbits. But regarding Crusader, the planet, with it being a gas giant, I think one thing we can really play with is the idea of pressure. Now, we don't want audio to be driving gameplay decisions and features and stuff, but what I'd like to explore is the idea of pressure on the ship and then the audio feedback you would get in regards to atmospheric pressure. Because there is no surface to get to, but the further down you go, the more pressure you'll be taking on the hull of your ship. Think of a submarine underwater and how they can creak and groan and all that kind of stuff. So I think it'll provide some opportunities for how the environment of Crusader can affect the likes of the ship and whatnot.

I'm hoping we'll see some contrast between the inside and outside too, right, Phil? I mean, you think about the outside being so windy, but then the inside being nice and calm and serene. You get a bit of a sharper contrast between the indoor and outdoor space.

Yeah, you get that juxtaposition from being inside, being sheltered, and then being out on the open gantries and stuff. With Orison specifically, it's making sure that the different districts, the different areas, have their own kind of feel as well. They're all in the same environment, being up in the sky, but it's making sure that each one has a different feel. And maybe each one deals with the wind slightly differently. If it's a more business-functional area, then it's making sure that everything just sounds functional; they're not trying to make the external traversal pleasing. Whereas the communal area, or where the shops are and stuff like that, it's making sure that they're protecting the people on foot.

Taking all our secrets. Bob, do you want to talk about some dialogue design stuff, or I can approach it from the sound effects side?

I mean, the idea behind it in general is finding sounds. You can take a real Earth animal, or even use your human voice, to create a base of something, and then layering sounds and processes and working with that to create something new is kind of the key there. But you don't have to start with a real animal source to get a real feeling of a being. It's just a matter of how that sound interacts, or how you can play with that sound to make it sound like a living being. And that's what's really special about sound design for creatures, creature design in general. So it can really come from any kind of source. It's just about what's the palette that you're looking for, or what's the sound, or what's their function, that really sets you up with what your source is.

Yeah, you can do a lot. So one technique that's used for dinosaurs is birds and stuff: you slow them right down. So if you record at a high enough sample rate, say 192 kilohertz, you can really slow that down and you don't lose fidelity of the information. So you can take a tweet tweet from a bird and pitch it right down to make it sound like a behemoth, like a stegosaurus or something. But what's on the market now is certain microphones that can record frequencies way above what the human ear can perceive. We can hear up to about 20,000 hertz, but there are microphones that can record up to 90,000, 100,000. So it's audio content that we can't even perceive. Now, if you record at 192,000 and above the 20k range, so you're recording at, say, 100k, and then you pitch all that stuff down, you're moving stuff that is unperceivable to the human ear into the audible range. And it's also high fidelity because of the sample rate. So you could take a bird recording, a walrus, a horse, and you could pitch it down, and you hear all this hidden content that is just out there in the world, but you just can't perceive it. So you can draw upon that for sure. And then the other thing we can do for creature design is use plugins that can morph sounds together. So you can get the sound of one thing and take its impulse, its dynamics, its frequency content, and have that morph with a completely unrelated sound. You can get a dolphin and a chainsaw, so you can make a chainsaw that has all the articulation of a dolphin call, or vice versa. So that's a good technique to employ. Or just, as Bob said, making vowel sounds, sounds of the vocal tract. There's loads of formant filters out there that you can make things sound like "ooh, ah, ee, ah, ooh". That really grounds sounds in some kind of creature, because it's all relatable, right? All these vowel formants that we all make. And that's a trick that we do for wind. If you want wind to sound creepy or ominous, you'd put an element through a formant filter. So as you walk past a door, you go, that's creepy, not going in there. No, thank you. But yeah, there's loads of techniques we can do. Creature design is really fun, and you'll come up with some absolutely ridiculous results, and that's part of the fun. So yeah, looking forward to that.

So the good news is that the bit of love for this important subsystem came in the quarter that just ended, because we've been working on it all the way through since the beginning of the year, trying to improve all the stability and get rid of all the connection issues that we've had. And all that is now going to be in the next release. All that work is already submitted. Obviously, we have to wait until that build goes out to know just how much we've achieved, but I am very confident that you're going to find that the VoIP stability is way better than it ever was. If we find any more issues, we'll be jumping straight on them, because we are aiming for this to be 100% reliable. We want people using this feature. We've put a lot of work into it, and it's so important for us that people use that system that we've put so much work into. So yeah, we've put a lot of work into it, and we would love to hear your feedback.

So with this, I think it would come down to syncing with narrative and with Bob, because if they're writing lyrics, presumably they have to be in lore, right, and appropriate. But what is challenging on the music side, because we have composers, we have Pedro on the PU and we have Geoff on S42, and we do compose some music internally, is really trying to establish what music would sound like in lore, in the 'verse, that far in the future. If you just put some AC/DC kind of rock in the game, it would be out of place. So if we do start to flesh out more diegetic music, which is music from in the world, it's always going to have this sci-fi take on it, to make sure it sits well in the setting. But yeah, I'll defer to Bob about how you'd engage with narrative and kick off that process.

Yeah, I mean, I know of a little bit that we've done, but that's for Squadron, so I can't talk about that. But yeah, it's a great idea. It's a great concept. And it's something that I know Narrative are very interested in, because they've already named these bands. They put them in there so that we can have things to talk about in these conversations that I alluded to earlier, these trees. We want to have things to talk about, about the universe. And a lot of times they talk about music, or their favorite this or that, food places to go, or brands, or things like that. So we're building all of this in our universe, and music makes sense to have as well. But yeah, Phil is definitely right. It's not just about naming them and referencing them, but tying them to real assets and having them have a real sound, and this persistent kind of feeling of these bands, or these entities in general, in the universe. And so it's very much on narrative to push that for us. But we will definitely build a relationship to... I think having more music identified like that is pretty cool, so I'd like to push it.