Star Citizen Live: Cargo & Hangars

Time is relative, no. So a few things on that. First is when I said that, first off, I do regret using that exact terminology. But to be more serious, when I said that, that doesn't mean we're really close to going live, right? So I think that's one thing to consider is that there's still the whole Avocado PTU live flow that we have to go through. And there's going to be a lead time to getting that out to players no matter what. So it wasn't like we were so close to just pushing the button and being live. I probably could have been clearer on that point.

But on top of all that, there's a lot of things that happen even still in between what we would normally go to live and doing for the development. And in particular, some things that ended up being especially problematic going long was things in particular like ATC in transit. ATC and transit were two really critical parts of the whole hangar loop, of course, and the ship loading changes that we did with the ship loading platform. And unfortunately, if you could ask me to pick two systems to not touch at all, it would be ATC in transit because they're two older systems in the game. They're very complicated and they have lots of bugs historically. They're problem children for us as systems.

So we knew coming into this that it was going to be a risk making changes to those systems. And what turned out to be, I think, more of a problem than we anticipated is that once we started getting wider testing, those systems really buckled underneath load in a way that we weren't really seeing internally. So if anybody was on the PTU, for example, whenever we first started going out, it would be rocky. We'd put out some fixes, bring in more players, and then the thing would fall over. And as a developer, to make it more frustrating, fall over in a way where not only could we not reproduce but our own QA oftentimes couldn't easily reproduce it. Or if they could, there wasn't a coherent understanding about what the fundamental cause was.

So for those kinds of issues, we had to get creative as far as understanding what was happening and what to do to address it because again, we couldn't really consistently or fully understand it. So we put in a lot of diagnostics, a lot of telemetry, try things, and then eventually work towards a conclusion on what the actual fundamental problems were. On top of that, I think there was kind of a consideration with those systems about, right, we're gonna go in here, we're gonna make some big changes. We know we have to in order to make all of this hangar stuff work. But these are kind of older systems. Do we rewrite them? Do we largely refactor them or do we try to keep them largely intact but with the smaller changes that we need, hopefully smaller changes that we need to get us out?

We wanted to hit the release date sooner. So we made the call that we were not going to largely refactor or rewrite them for this release. In retrospect, I'm not sure, like if I could do it again, I would be interested to know, would it be longer actually if we just decided to just rewrite or refactor them first? The good news is that we already are planning on doing just that. And in fact, transit is already in progress with the new version, which needed to happen anyways for server mashing and all this stuff. So that's good. And then ATC as well, we already have plans on how to rework it and to make it just fundamentally some of the problems at play here because ATC as a system is so like centralized and it keeps a lot of state internally in a very centralized place.

But these systems are extremely complicated about how they deal with persistence and logging and streaming. And in particular, tracking ship allocations in the hangars. So we would get bugs, for example, with like ships doubling up in hangars, et cetera. Like those things were really challenging the system as well as like the doors not opening. So like a lot of that stuff, I think we have a lot better idea about what to do about it as a system in the future. But this release, we made the call to try to take the system that we had, just kind of like get it working with the new direction and then hopefully give us some more time without so much time pressure to make the right adjustments like more fundamentally.

It's a risk we took. It didn't turn out I think as we hoped, but sometimes that's how it goes. But there was a lot more things than that. Another thing was the mission system. It's a new mission system with a new delivery hookup for the delivery missions, first outing, right? There's a lot of new shaky things that we found with in particular the hookup from the freight elevators to the missions. There's just like a lot of loose ends, a lot of things threads would get lost. We have this whole warehouse system that has to interact with the mission system. And there was just, again, a lot of these like rare low repro issues that in these very fundamental places, actually, this is another point for ATC in transit.

Like when you have a moderate inconsistent bug in a system that's not as critical, it doesn't seem as bad. Not to say that bugs aren't bad. Of course, we want the game to be stable. But whenever you have something that's like, let's say it happens 1% of the time or 0.1% of the time. Unfortunately, if that isn't a part of the game flow, that's critical. It's unplayable, right? So we have to like take even more care with those systems to get them into a better place. So the game is more playable because those bugs have an outsized effect.

But beyond that, there's also in particular, a lot of issues that we had with networking and serialization and the replication layer changes that were coming in preparation for server meshing. And I can just speak from my own personal experience, debugging some of these things. There was some very complicated, hard to understand bugs that we had to work back to eventually being actually corruption in serialized data that was coming from some bugs in the replication layer, which we could not reproduce that again internally. And that was a wild one to figure out. It was very, very confusing as you can imagine, because it's rare and it's random and like the data you get doesn't make sense and you try to make sense of like how these things come together. But we got there.

There were design and iteration changes. Some of them, we were pretty public about like the item bank, but also tweaks to the freight elevator kiosk, changes to how the economy tuning works, for example, auto loading times and stuff. So like, and to be fair, we were wanting the patch to be a complete package as much as we could. So given that we were already going late, we didn't entirely shy away from taking some of the time to like also like clean things up on the design side while we're also grappling with some of these harder engineering problems.

And then like the final thing I'll say on it is we didn't want the whole schedule to shift. So we still have a lot of other goals. One of them obviously was the Atlas, but there's things coming up 4.0. We're obviously still trying to get Squadron finished and out the door. And we didn't wanna just take everything and just like take everyone, focus entirely on this release, get this release out the door and have our whole schedule change for the whole company. So we had to like keep, there was a lot of people who needed to move on to 4.0 goals or whatever. And so it meant that we needed to keep the core people that were working on the main line features on the release, but we didn't have the full force of the company to just close that one specific patch out. We could have, but like I said, it would have caused problems for our other goals otherwise.

So you kind of in total, there's a lot of things that I feel like I've learned from and I hope in the future, it gives us a better like framework for how to deal with something this complex.

Yeah, one last thing I'll say is: I think there was also just a fundamental problem with the whole conceit of having this many fundamental things in one patch, where I think if I would do it again, I would slice it differently and try to do the persistent and instance hangars in their own patch first and then layer on top all this other stuff. We wanted them to come together as an experience. And I understand that might have been actually kind of a weird thing for the players to go through to have like this one particular aspect of that feature set come in without the things that make sense. But from a development perspective, it would have allowed us to harden on the aspects of it that were in particular thorny and more of an engineering challenge and then get that hardened and stable so that we give that to the players and then layer on top of that all this other stuff around it. But doing it all together, I think really challenged us when some of the, like I said, some of the risks didn't break in our favor.

To me, this one, it's not a challenging question. It's gone on a bit of an arc and I don't think that arc is completely finished. What's funny about that arc is we've kind of ended up where we started actually because in the original proposal for the item bank, it works like it does now actually, which is it's the personal inventory UI and really the item bank is more of an interaction point to trigger that UI. And the reason to understand why that makes sense is because to think about what you're doing at the item bank. What you're doing really is we're trying to make it easier to get things that make sense from like a FPS player perspective, right? So if you wanna get meta pins, you wanna get armor, a weapon, these kinds of smaller items, which are often equipment. We wanna make that intuitive and easy and quick, frictionless. The personal inventory UI makes sense for that because it has the paper doll view where you can pull things over and like kind of very more tangibly interact with the player and more quickly equip and un-equop, swap things out, etc.

However, the personal inventory UI, I think everyone that's played the game knows it's not perfect and there's things that we have long wanted to do about it. And in fact, for this release, it was one of the things that we hope that we could get to, but we didn't have the time to. That's still, we're still going to come back to making improvements to that UI. So for that reason, we had some internal discussion on iterating on it and looking at it. And after we had gotten the Freight Elevator Kiosk UI to a pretty good place, I think people saw before that we originally had this more 3D-based UI on the left-hand side of the Freight Elevator Kiosk, which we may come back to at some point with some rethinking. But when we went to the more like kind of straightforward like grid view and then added certain things like search and better filtering, that experience started to develop in a way that had some functionality advantages over the personal inventory UI. So we had some discussions internally and decided that we would try moving it over to use just the Freight Elevator Kiosk UI rather than the personal inventory UI. And there was trade-offs, of course. So like some of the like faster, tangible, easier kind of changes are gone, but you have other things like search and filtering that are better. And after giving it to y'all and collecting a lot of feedback and internally, we had to make a call. It was one of these things where there was very strong voices actually on like kind of both ends of it. So I think no matter what, some people, you know, it actually, I just went to a bar citizen in Seattle. It was funny, like I talked to people and it was like, someone would say, man, I'm so happy you made that change and that somebody else would be like, why don't you bring back the Freight Elevator Kiosk UI and the item vague. But I think the answer is gonna be, we're gonna probably keep it like it is now, but bring over some of the things from the Freight Elevator Kiosk UI like the search functionality to make that a better experience itself. So both of them are, you know, they are better user interfaces and UXs in general.

I'm merging all these [questions] together because the answer to all of these are sort of the same thing. [...] I'm gonna read these notes directly from Ian and Rick here so these are just their words.

"We plan to add garage modules. So we don't intend to expand the pavement but there should be more room for vehicles when you could basically add on modules and expand the hangars to the left and right and to the front and back and there's different sections. The plan is for personal hangars to be able to be customized and expanded in the future with the addition of these hangar modules. The exact list of modules are still being defined at this point so I'm not gonna be able to give you a list of whether it's a firing range or something like that. But you will be able to modify your hangars to certain extents and expand them out in other directions and stuff so that you can bring vehicles out and kind of store them off to the side and you don't have to use the ship elevator for bringing all your ground vehicles and stuff or just your smaller fighters and stuff off to the side. And as far as cleaning up the garbage around the end, we've already done multiple passes for the base hangar OCs. So we're listening to the community and moving in the right direction, that's Ian speaking there. We want to split the data for instance hangars and personal hangars separately so that we can completely remove everything in the personal hangars while still leaving some form of dressing and world building in the instance hangars."

So what this means is that as we've talked about before, you have your home location and then you have the instance hangars that you'll get at every other location when you go there. For those, right now, those are using the same OC. So when you remove the dressing from one, you remove the dressing from all of them. We want to split those into two different object containers so that the instance hangars that you get at every other space station or whatever, those can stay dressed and look good and everything, but your personal hangars, the ones that get assigned to you at your home location when you log in, those will be completely empty and you'll be able to decorate those to your heart's intent. That is our intention there. No timeline on when that's gonna come.

When you get it, I don't think so, but... I'll say this, we plan on, and as I've said before about bed logging, we're definitely gonna work more on bed logging and we're gonna work more on logging in and out. And one of the things in particular that we're gonna work on is logging in and out of hangars specifically. We wanna support that. That's something that we've already been working on to some extent on the technical side, but there's still more work to go there. I don't wanna say that when you get this particular thing, a certain feature is gonna be supported in the next release though. So as far as that goes, I don't plan on that being a supportive thing by then, but logging in and out of hangars and bed logging is gonna be developed further is what I'll say.

[...] You're telling us that at some point, not saying the next release, but at some point you will be able to log in and out from your hangars?

Yes, 100%. That's definitely a thing that we're gonna support.

We actually answered this last week with Benoit. It's here because I wanna refer people back to Benoit. Basically, it is not a feature issue. It is a matter of physics. Physics is one of the largest performance hits to the server. So that is basically it trying to negotiate the physics of all those items placed on other items. As server performance improves, as server meshing provides us the benefits to server performance that we all hope it will and stuff like this, that problem will become less and less severe as it is. And then we'll be able to go in and find the individual edge cases that cause it here and there and there. But basically, it's one of those things. It's literally one of those things where it works on my computer, and then you put it up on a server with 100 people and it starts flopping around, which is why it looks one way on ISC and it looks another way in life. We show the intention of the feature and how it's supposed to work, and it is basically a performance-related error that's causing it in the servers right now. So as we continue to optimize, we continue to improve performance that will resolve most of those issues, and then we can go in and do the edge cases.

For sure. This is another one of those things where we had to make some calls as far as what we were gonna try to deliver on this release, and if people saw the very early, actually, I'm not sure this part of it got to Evocati or not, but at one point we actually had a dropdown where you could pick on the freight elevator kiosk a party member and actually choose their inventory there, so we were trying to really integrate party members into that hangar experience, but on both of these, where we ended up is needing to remove it just, again, get the release done, but also I think we realized that there's some more design considerations around the experience of security and permissions, and party membership isn't always the right bit to answer some of those questions anyways, so there's definitely some design work to do there as far as the experience and how you attribute or revoke certain kind of rights in your hangar or to other players, but we do want to allow that general kind of flow where you can have your friends there that you're playing with, be it in a party or in a group or an org or whatever, and then have a way to spawn your ship in there. Additionally, the ATC did not really support this pattern very well, and it's because the ATC is an old system that was originally written with Port Olisar in mind, and just some landing pads of Port Olisar. That was the original implementation of ATC, so this whole idea of even hangars is kind of alien to that system, let alone that multiple people could be in a hangar, and we've kind of had to build that into it, but it has some fundamental concepts like this thing called a permission token that don't really make sense in that world, which is why we're reworking some of the implementations to essentially actually, this is kind of getting at one of the core things that we're gonna do is separate out from ATC hangar management as a concept, and ATC is gonna become a simpler system that all it deals with is requests to land and exit, and so it's really just governing the entry ways to the hangars, and that's it, and then have a separate system for hangar management for ships and all that, and included in that can be the idea of how do you handle different situations and corner cases around multiple ships, including multiple ships from multiple players.

I don't know about that. It's not a thing that we've talked about or I've heard even feedback on. But one thing I'll say is we are looking at ways to make it faster, that whole loop, because we wanna make getting ships like a quicker experience. Like I think at this point, people have enjoyed the pomp and circumstance of getting these ships. It's a really cool thing when you do it. But of course, with anything, you do it enough times and you kinda just wanna get your ship, right? So we're looking at potentially some ways where you know you're getting, let's say like a Titan or something like out of an XL hangar. It feels kind of absurd, like the giant experience that unfolds for you over the period of time it takes. So we're looking at, for example, maybe we can like preset the loading platform height or maybe adjust some timings to make it faster. So you don't have to wait quite as long, but without changing some of the fundamental elements of that experience. We still want it to feel big and important and have that pomp and circumstance, have it feel somewhat epic. But where it makes sense, have the ability to like have that time, not as long for the smaller requests. I don't know that we'll outright disable the sound there. Again, if there's lots of feedback on that point, maybe they can.

Well, I mean, it's obviously what we have now is I would say a, well, you know, tier zero implementation. We wanted to get something in for the hulls that was unique because it's a unique ship in the game. It can't land with the spindles extended, and it's big, it's epic, and we wanted the cargo loading for this ship and the series beyond that to feel like it made sense for the cargo trading at that scale. And in particular, have that visual be like something that was like compelling. And so we wanted to make that cargo loading in and out, like feel like something that was like physically like building up and reducing on the ship. However, obviously it's, you know, just doing this like popping in thing, and then also it doesn't support any manual loading. So in the nearer term, we are looking at adding support for manual loading for the hull series and any other ship that might need to dock for cargo support. And I'm not gonna say like any particular release or anything, but like, absolutely, we're looking at bolstering the support for those ships, or not just both of it, creating the support for manual loading at those loading areas on the exterior of the stations and actually adding to them and having potentially multiple, we're considering different ways to make the throughput on that system make sense, both to allow for multiplayer and ships using tractor beams to help optimize that loading time. And I think that will be a really big, like meaningful change. Beyond that, obviously we still plan on doing a better implementation of the automatic loading for the whole series, be it NPCs or drones. I think that probably is gonna be later because it's more of like a visual experience and less of like a gameplay experience, whereas like the manual loading would be a really big meaningful change from like the actual in-game experience for the players. So that's what we have now.

Yes, so hover trolleys, another thing that was sacrificed to the release gods. In particular, the hover trolleys have some new physics around their behavior and they were being pretty wonky, I'll put it like that, as far as the actual like attachment onto the hover trolley, not with the attachment, but how the hovering dealt with it. So like you would suddenly have this attachment and like the thing would kind of like sway around and sometimes like violently flip over or something. So there's some physics issues with it that need to get sorted. And also there's some questions about quite how to integrate it into the game loop, as far as how you get them, or is that something that you buy or can you find them in the world, et cetera, et cetera. So we took a step back from that to focus on getting the release out the door, but it's not forgotten. They are gonna come back once we've kind of smoothed out the implementation and like the physics problems, and then also pinned down exactly like how we want players to use them and where they are in the game. So yeah, it's still coming. I will say beyond the hover trolleys, there are other things in addition that we're always looking at as far as like figuring out how to make the game loop for moving cargo better. And certainly I think in mass is certainly like a thing that now that we have manual loading in the game, we have more experience with it. It's clearly like there's like this interesting point where if you're not dealing with the largest boxes in the game, some like combinations of medium to small boxes can be cumbersome. So it's definitely like a thing that we're aware of. And there's, I think gonna be continued design consideration on that.

Isn't that? I'm not sure I understand because like you pay a fee to auto load now.

Yeah, I was gonna say, that's a weird question. I thought that was, that's pretty much the thing. We'll skip ahead.

CM I mean, I can give a, maybe I can answer a question that they didn't ask. Which is, is auto loading always going to work the way that it does now? Sure. We're gonna build on that. Right now, auto loading is built into the commodity trading experience in part because that was the quickest way that we could do it. I think we'll always want to have that because it's convenient in order to just make those decisions when you're doing trading. However, there are times when it would be convenient to auto load things that aren't necessarily part of a transaction or after a transaction.

Like mission cargo can't be auto loaded.

Mission cargo, but also items in general that you have. Maybe you want to load those into the ship. And so a full auto loading experience is a feature that we're currently looking at as far as making it to where you can just have whole sets of stuff. If you just don't want to touch the freight elevator, you just don't like it, that's fine. We're looking at having an experience for you to where you can say like, okay, I want this arrangement of stuff. And as long as it fits into the cargo grids on the ship, then it'll move them around. And now that we've made it to where the cargo grids support so many different items, that means that there's a lot more opportunity to make that work. Of course, there'll be a small fee associated with it, but it will, I think, support, it's kind of a bigger solution to, instead of just like solving it specifically for mission items, it's something that you can do more generally. And then also, we talked about small vehicles, ground vehicles, getting those into your ship. Well, that would be a solution, right? If you didn't want to drive it out yourself.

Now, not just spawning a ground vehicle in the cargo, but spawning a ground vehicle in a cargo crate. That can then-

Well, I'm saying like you could have the ship stowed and then request to have the vehicle loaded into, like the ground vehicle loaded into your vehicle.

Right now, I mean, it's using the locs [localized strings], what we call the loc string. So, I mean, it does, but the problem is there's various kinds of data associated with these items that are in different places. And what we haven't done is really built that out, but it does use the localized item name. But there are other kinds of fields and stuff that it could be built out. In fact, one of the bugs that we had, one of the bugs that we had when we first added it is that it was just using the loc name and not the localized name. And it would make it really confusing because some things are really close to the actual end string and some things just aren't at all. And you would search for certain stuff and it wouldn't find them. And the reason is because it was searching the loc ID, but not the localized name, but that's not the case anymore. But there are other kinds of fields that it's not searching on. And sometimes I think to make that a little bit confusing, sometimes there's different strings that are shown in different UIs. So like, for example, the shop UI might use a different string for the purchasable display from the item ports editing and stuff like that. So more to be done, but it is using the localized name.

Great question. Unfortunately, another thing that got cut. This one's in particular, it hurt me because I made the packing and the attachment behavior and the system, the packing and arrangement works, but there's one bug I'll get to in a second that didn't. And unfortunately, because most of the items work, another thing that was sacrificed to the release gods, which hurts me because I wanted to be good because I like to be proud of my work. But the issue in particular with those, I feel I'm gonna get a little bit technical for a moment, is the pivot. So for things like cargo boxes, the pivot is on the bottom in the center. And whenever I originally wrote the packing system way back for the whole sea, the only thing that we were putting into the cargo grids at that time were the cargo boxes. I knew that more things were coming. So the packing system itself can expect any kinds of dimensions and handles multiple orientations. But the pivot was something that I just didn't have time to generalize. I had to just essentially kind of like hard code it to expect the pivot in a certain place on the item. And the technical challenge there is you might say, well, I'm holding the item. How do you not know where the pivot is? The problem is when you're doing things like calling stuff up from a freight elevator, you don't have the item. The entity isn't spawned. You don't have the geometry for it. You can't directly access the pivot. So you have to have some way to know where it would be without loading the skeleton and deriving it, which would be incredibly slow. So what we have to do is essentially kind of bake that data out into a data set that could be read from an asynchronous persistence query or based on the class data. So that's the work to be done here. It won't take too long, but it's possible it comes in very, very soon, but that's essentially it. The reason is the mining bags and these things, their pivots are in a different place than most of the other items in the game. And the system isn't yet expecting that the pivot might be somewhere else than where most of the other items' pivots are. And for that reason, the way that it rotates around and the way that it orients itself on the cargo grid is just wrong for that particular thing, which particularly for the mining bags suck because they're situated in a way that actually puts them through the freight elevator. And sometimes when they detach, they actually drop down, which is awful. So like definitely aware of it, but yeah, fix is gonna be forthcoming.

Yes. - So yeah, I mean, definitely it's a thing that we're looking at on the design side. And so I don't wanna give like complete answers cause that's something that might change a little bit, but in short, we want it to be to where you can spawn multiple ships, especially in these XL hangars where you have these huge platforms and certainly you have the room. There's just some thought that has to go into all these different permutations. Let's say you have a ship on the platform and you wanna get another ship. Right now, the way the system works is if you get a different ship, we're gonna sell the first one. You don't always wanna do that. How as a player do you express that or how do you understand that what's going to happen? Do you have any control over the arrangement of it? Et cetera, et cetera. So those are things that are getting like pinned down on the design side, but the goal itself that we wanna be able to spawn multiple and give players more control over that is something that we're going to do.

Yes. Another thing that is on the design side, we're still working on, I remember talking about this like a year and a half ago about like how we could handle this because before this release, we added the inventory containers at the end of last year, but we were working on it before then, especially on the design side. So this is something that we've known about for some time and those boxes all look the same. So if you have lots of different boxes, how do you know what's in them? And we've had lots of ideas so far as tenting and labeling and naming, et cetera, et cetera. And then also potentially wanting that to be something that you could maybe search on in the front elevator kiosk, right? I'm not gonna commit to one particular solution there because as I said, there's still some design work that needs to be done to like make the final call about like what that, how that's gonna work and then we would do the actual implementation, but it is a thing that we want to address for sure.