TL;DRBeausejour says 4.8 involved constant firefighting, including two bad check-ins that broke persistence, though the overhauled entitlement system worked well, and Lind says extra protection against such check-ins is needed.
Well, for me, if I take it from the machine room side of things, we've been firefighting since we launched 4.8 from a whole slew of different issues. Some of them are to be expected when we launch a patch that has this many core changes. We've touched on systems that are really central to the traversal again. So we're talking about the entitlement system, which has not changed in years at this point. And we've completely overhauled the entitlement system in 4.8 to power in part the item imprint system, but more importantly, to pull the entitlements out of the patch cycle and persist them across patches. What we call the LTP layer is not just what you have in your inventory, it's also what you own. Your ships and all of the items that you purchase in-game now have an entitlement record that holds the insurance loadouts for it, all of the customizations that you make on it. And that's hosted outside the game. Not outside the game, but outside the game patch. Which means that when we come back from a patch, there's no risk of, oh, I'm not getting my ship back. What you see in ASOP in the first patch will be exactly the same data source in the second patch. And so that risk of losing ships across patches is completely gone. So we're very happy. In fact, that bit actually went very well in 4.8, even though up to the launch, it was always last-minute things to look at. But we have been firefighting across a multitude of different issues that were tangential to that. Some of them we can touch on a little bit, because I've made some notes to talk about gameplay systems here and why they were affected by some minor infrastructure issues that turned out to be massively contributing to the perception of the patch. So I'll talk a bit about that in a bit, but basically we've been firefighting since 4.8.0. I want to touch on one thing because we just had an off-the-cuff moment about the Idris last night. But two weeks prior to that, we had another downtime patch that needed a lot of rectification. So the teams that run the online group were hard at work on this maintenance. And unfortunately, another one of those unwanted check-ins came in during that 4.8.1 cycle, where, in an attempt to fix the VGL flight suit that was missing hands and feet, another check-in came in that removed sub-components of this flight suit, but not just removed them from the flight suit, deleted them from existence from the game. And that's not something that you can do in a minor patch. And so clearly, that's two instances in this patch where transition check-ins should have been validated and prevented and went in, and then after that became an infrastructure problem. Because same issue here: if somebody was wearing the VGL flight suit and referred to these entity classes that were actually representing, I think, the nozzle and the front pack of the suit, then the game server would crash. And so we had, again, to do percussive maintenance to remove those items and fix that. And so that highlights to me a massive issue with how we validate check-ins and how we get changes in the game across persistence, until we have full forward porting compatibility on the persistence layer, which we don't have at the moment.
Live servers, you're always running, right? We're talking about forward patching. If we'd been behind closed doors, you would have done that, right? No one would have ever seen the game until you'd done that piece of technology, but that's not who we are. We're always running, and that's what makes it really, really exciting to work here as well, right? Because we're putting ourselves out there all the time. But if I look at this specific case, you're saying, right, we have checks and we have contracts on what you can and can't do in these sub-streams that we've already released, where the data management is super, super strict. And obviously now we invalidated them twice in a row. So it's time to look at how we really add that extra layer of protection.
TL;DRBeausejour agrees the weapon pricing and the crafting breakdown were validated separately with no check across them, and says human validation is no longer enough, so machine validation is needed for everything.
Transversal to check all of that. And also, we're definitely now, post 4.7, in a realm where just human validation is not enough, right? And so a lot of game design data is typically managed by the team, but I think that we're way past that and we need machine validation for everything, because the rate at which we build it and the amount of technology we're putting in is just too fast for validation to keep up.
TL;DRJens Lind says the problem came from the 4.8.1 database combined with 4.8.2 data, an environment QA doesn't have, so hiring many more QA people wouldn't have caught it.
In this case, it took the situation of how the database on 4.8.1 was, with the new data from 4.8.2, to create the situation. And that's not something that the QA environment has. So whatever we do, we as a technology group would have to come in and at least give them that. And then if you put your 500 QA on it, then probably one of them would have spawned an Idris. But without that environment, you could put as many as you like on it. No one would ever have discovered this situation. This is just too far removed. And I think that sort of comes back to the large dataset, right? As you have a large dataset, you just can't scale in manpower alone. You need something that scales way beyond that.
TL;DRBeausejour says the cause isn't pinned down yet, but some server meshes run below the target frame rate, and the team is tracking six to eight optimizations, including lock contention in the area component, for the hotfix channel.
Well, so what we know about right now is that there are some meshes, and when I say meshes, I mean a dedicated game server inside a shard, that are running way below the FPS spec that we want. And so we don't have a handle just yet on what is actually causing it. We do have performance captures, and we're tracking at least six to eight very specific optimizations that need to happen. One of them is within the area component, which is used all over the engine. Area and zone component, they kind of go hand in hand. Again, because we're pushing more content, the hierarchy tree of the game world is much bigger. So there's more areas, more zones, more subdivision, more, more, more. There are some areas of that where you add a lot of entities that enter or exit these areas, and the engine provides the listeners to track entities entering areas or leaving areas or zones changing, right? And obviously there's concurrency constructs around this to prevent one thread from accessing another thread, right? And so, just keeping it simple without diving too much into the low level, which we will probably do in some of our desync areas, there's lock contention in the area component at the moment in some places. There's lock contention in some areas of the game. Well, I literally mean area: if there's many entities in this spot, because of the way it's configured, we're seeing basically FPS drops in those areas. And so we're tracking, like I said, about six to eight major optimizations that we're trying to get in our hotfix channel, to try to get this FPS back up to the spec that we want, which is above 20, across the...
No, no, no. And it's important to note that when a server is running low FPS, it doesn't mean that it will create desync, but especially when you're interacting with entities that are simulated on that DGS, AI characters, NPCs, then you're going to perceive these things, because the source is slow at creating its output. The traffic goes from your game client to the replication node, replication node to game server. Now, if the game server is running slow: at the beginning of the frame, he's taking the state from the network to the game and applying it. At the end of it, he's applying the changes back to the network. If that loop is slow, then your client's going to predict something, and then that prediction will get corrected the next frame, and that will create a rubber band, right? And so that's the gist of why there's desync in general. Why does the client have to predict? Because if he doesn't predict, then you're going to be at the tick of the server, and you'll see...
TL;DRBeausejour says ship cleanup is one thing being looked at: the system seems to work but may be misconfigured in some areas, letting ships pile up, and more entities make servers do more work.
That's one of the things that we're looking at, how aggressive or not the ship cleanup is actually happening. That's one of the things we're looking at, yes.
Because you can watch a Twitch stream or look at an Issue Council report and you'll see dozens and dozens, I feel like it's even more, of bricked ships everywhere, with the insurance changes bricking these ships. Is this viably part of what it is?
In part, yeah. That for sure doesn't help, because the more entities are replicated, the more the server needs to do every frame, and so that contributes to this. So, yes.
No, a server not performing so great tends to have a slightly larger vehicle population or a very large active population. That can also happen. In general, it's just a lot of everything. We're quite a heavy game, entity-wise. We want to avoid getting to the sort of size that we were in Stanton before we went into meshing, right? Because that was consistently difficult to manage.
I know that there's been some changes related to how we perform entity cleanup. So unattended vehicle cleanup used to be a separate system that ran on its own. That's been merged into what we call the density manager, which is basically an engine functionality that will look at overpopulation in clusters of areas that designers obviously have control over. For example, we want to make sure the exit routes for hangars are cleaned up. And so these have active cleanup zones configured, right? That's designer controlled. Now the engine will then apply the logic on the volume, determine, oh, there's too many of X in this area, and will start popping stuff out, right? Basically going to do the cleanup. Now for ships, the unattended vehicle cleanup system used to be separate. Now it's been brought into the density manager, I think end of last year, something like this. So it's been a while. And this one tracks whether there's player passengers on a vehicle and will, based on a certain timer configuration, and the values are different depending on the areas, either stow the ship back into the originating inventory or destroy it, depending on if it's a player-owned ship or if it's an AI ship. It does look like maybe there's a misconfiguration in key areas that are leaving ships there. So we don't think the system is incorrectly functioning. We think it might be inaccurately configured in some areas. And so that would allow a very popular area to just accumulate ships like crazy. And so that's what we're looking at right now.
TL;DRThe speakers are unsure the density manager has a specific rule for bricked ships, but say it will eventually, and that it works on how crowded an area is more than on timers.
It will. If it doesn't, it will eventually. And it's less about timers. It's more about density. It's trying to say you have overpopulated this area. So it's less about that ship having been there for a long period of time. It's more like, there's a lot of ships here. Let's remove one. Let's figure out which one is best to remove. Unattended vehicle does have a timer as well. So it's sort of like we talked about with validation. You start with a timer, then you start adding density, then you start building up these views of the world to understand which areas are problematic. Like, oh yeah, the entrance to a hangar, we have to be super strict.
There's a place, as we're talking about this, something that pops into my mind that I think is also in our process. Not even the development process, but let's call it live operations of the game. We're very lean in terms of our live operations at CIG, because we're in development. And so even though we do run a live service, we try to run a really lean ship on that side. And so we don't have an army of site reliability engineers that are looking at every aspect of, oh, shard 25 has an overpopulation of ships at Jackson Swap between this time frame and this time frame. We don't go to that depth. And so we track overpopulation across all shards, but we can't go granular into this. But that is an area where we could improve this, detect those things and better tune these values. Typically right now we will tune the density manager values every patch or every hotfix, but that is a slow cadence compared to how you would react if you see, oh my god, on every shard this location is loaded, let's effect the density manager change right now. Right now we don't have the ability to do that live, and that's a problem. That would improve the game quite a bit.
I always say, if it was easy, it would have been done. So by that, you have to assume there are reasons that make it not easy. And it's usually the complexity in the interactions, like Benoit talked about: you sell a gun and then you deconstruct it, and then the material can be sold somewhere else for something else. These kinds of complex interactions between lots and lots of different systems are usually what stops us from doing something that, from a tech point of view, is literally a flag in the database that says that's bricked, I can treat it as garbage.
TL;DRBeausejour suspects behavior changes tied to the Imprint and insurance changes, such as more ships left behind or players favoring loosely configured locations, while Lind can't point to any other gameplay change that explains it.
For me, it's behaviors. So there's one thing about running a game server, which is that it's kind of an organic behavioral system, right? And so you can have a game server with a version of the code and a version of the data running a certain way, and then players start doing something different, and that completely changes the performance profile. And so I feel like in 4.8, most likely, my hunch for these six to eight performance issues that we're looking at is that it's related to some Imprint changes. So the way people use ships, leaving bricked ships behind: are there more claims than before? Are there other egregious issues that showed up that are creating more claims that would affect the amount of entities on the server? Is there a malfunction on, for example, the density manager cleanup that is leaving items in place? Or are people just going to a location that's configured to be very loose on its density, and it's somehow the location that people have decided to go to in this patch? So sometimes the performance profile of the game servers changes just based on behaviors, not on, oh, we changed something.
I mean, it's hard not to just parrot what you just said. It does feel like, when I think of 4.8, there's nothing that I can point to and go, yeah, those gameplay changes we made could have caused it. The only thing I can think of is, we did performance improvements, but there is a large change in how we handle vehicles and the lifetime of vehicles.
I mean, that would be, based on what I know of what that patch put out there. I just don't recall there being any larger changes on the gameplay side that I feel I could explain it with.
TL;DRBeausejour lists several suspected causes without going deep: navmesh tile changes on planets, an entity spawner respawning, area and zone lock overhead, ship cleanup with active MFDs and flight systems, and fire and temperature physics.
I don't want to deep dive into each of them, because it's a bit intense if we go down that route. But there's navigation system dispatching of tile configuration changes on planets, which affects our AI using NavMesh on planets. So there's a performance cost on that that is costing us a chunk of frame in the servers. This is purely based on the servers we know are affected by this problem.
Yeah, exactly. This is cross-linking a lot of things. This is a pet peeve of mine that I keep pushing on, and we're going to definitely resolve that from a software point of view: there's an entity spawner in some of the areas that are affected that is coming in and out of streaming and respawning its spawn set every time. These are the NPCs that just spout out. That's one example; I have multiple others related to this. That's one. The area component we talked about. So what we call register with external systems is a phase when a component or an entity is coming online, when it's streaming in. It needs to register with external systems. That's a notorious area. That is an issue on multiple components. This one, from the area component, is a problem in that place.
Correct. This one is more engine and, let's call it, area configuration related. This one bleeds into the zone lock that we talked about earlier. So those are two, area and zone lock. So this is really tied to the amount of content in the game and the complexity of that content in some areas. Because we're also making more and more complex scenes, like space stations and the content we've put out recently, whether it's TSG or Rockcracker, there's a lot more complex layouts, and those elements contribute a lot to this. And so those are part of that. The ship cleanup is another big one that we're looking at at the moment, which is probably the source of the other two, which is the amount of registered entities that have either MFDs and/or the flight system active.
Well, we don't know specifically. I bet that if I went into that area and I looked, there would be a bunch of MFDs that are somehow visible. So these components have a visibility aspect to them where, if you look at it, it's going to replicate for you. But if you don't look at it, it doesn't. Might be a malfunction with that. We're not sure.
It could be broken ship MFDs. We've seen weird things; everybody knows about the graveyard. There's been other graveyards in other areas, right? So there might be a new one we haven't found yet where somehow MFDs end up there.
And one final one, that is fire. So some ships catch fire. And depending on the size of the voxel grid, we've seen that the area physics properties of how the temperature system interacts with fire actually need a big optimization, and it's costing us frame time. That's a combination thing, because internally we can't reproduce it, but we see the cost on some of the game servers that are running slow.
TL;DRBeausejour and Lind say optimization and crash fixing are a fog of war, where fixing the known issues may help or may just reveal new ones, and different servers can have different problems.
Now the thing with everything in optimization is that it's a fog of war. You fix that, that's what was slowing it down, then you find the next thing. You get rid of the elephant, and behind it is another one. The same properties apply to crashes, right? Typically when we do stability checks on game servers, which, by the way, other than the two big issues that we've had, the baseline stability for 4.8 outside of these two events of the missing entity classes and the Idris problem has been in a really good range overall. And so we've been happy about this, other than now the stats are terrible because of this. So we'll have to take a look. But the same properties apply to crashes. So when we look at top crashes, you're like, oh, we just have to fix this one. It's super high frequency. Then we fix it. And then, oh my god, there's three more. And so it's a fog of war. Optimization and crash stability are fog-of-war activities where, yeah, you've got seven now. You know what those are. We fix them, it might be the end of it, it might not be.
You can get clusters as well, and that's the same on the performance side. We might be looking very, very closely at the servers performing poorly, and we might be fixing things that help those, but that doesn't help at all with the other servers, because they might be running into completely different issues. There might be something that's happening specifically on those servers that's unique to them, that brings them to that low rate, whereas the rest of them don't benefit as much from that. So you always have to look closely at the details and react to those, but proactively try to look at the big picture and where we are going overall. Once we fix that cluster there, how do we get all of it to benefit from a smooth ride with zero desync?
TL;DRThe team is working on hangar bugs such as exterior doors staying closed, players being told to enter someone else's hangar, and repositioning getting stuck after impound, mostly from failed handshakes, missing timeouts or invalid reposition positions.
So hangars. I'm going to start with hangars. Now, I'm going to pick the ones that I think are interesting in the list, in each segment. These aren't in any particular order. This is just some of what people are working on today. So: PU, multi-locations, instanced hangars, "exterior hangar doors remain closed while interior hangar door opens." So that's when your hangar doors open, but the gateway outside is still locked.
So they belong to two different parts, right? You have the exterior and you have the interior. The interior is yours. You can control that in your instance. The exterior needs to be synchronized with your interior door. So what most likely happens there is, for some reason, your interior thinks you've been given the right to this gateway. Or you could have been given a right to a different gateway, and there's an exterior door opening somewhere to nothing, and your door is opening to a closed door. So there's a few things that could happen there, and normally it's that handshake between the exterior and the interior, via the notorious ATC.
But this one goes with its little brother, overlapping hangars: "Player reposition when landing in hangar: ATC given, but seen as trespassing in someone else's hangar, results in being repositioned." Which I think is a sibling.
Yeah, definitely, because you're looking at a situation where you think your interior has been given a certain gateway, you're heading there, but it's actually someone else's casa. So you're correctly trespassing, but you've obviously incorrectly been told that it was yours.
And then the third of the trifecta in this segment, if you put them all together, is "players can get stuck at repositioning player indefinitely after getting impounded." And so if you get impounded, there's a chance that the server will never complete the repositioning that's supposed to take you out of your hangar. Most likely in this case it's one of our charter elements, which is that there's an asynchronous call waiting for a callback that never happens. There should be a timeout so that it can be retried and complete. In this case, it gets stuck there.
It can also be validation. For some reason, the reposition positions that were created are no longer valid. Because I think one of the rules is you will not be repositioned until we know there's floor underneath you. So until there's floor.
That is definitely related to this. It sounds like his repositioning position wasn't amazing in that case. It also does disprove my floor, because he clearly did not have floor.
TL;DRDocking is being worked on, including a docking-collar fix that was resolved but reopened after failing on the network, plus collision loss that ejects players and cargo services not answering hails.
Correct. Let's talk about docking. The very first one: "unable to dock using docking collars." That is problematic, isn't it? We looked at this this morning and it was actually resolved when we were talking. It is now reopened. So we're going to have to dive into that. But basically, docking is a subject that's always coming back. And this patch has also been contributing to the overpopulation of ships, because players would just basically land on pads and leave the ships there, where there's no impound. And so that's a big issue. So the first one is unable to dock using docking collars.
So I can quickly just go on: when you say it's resolved and then reopened, what happens there is the developers, to the best of their abilities, found the issue, they've fixed the issue, they've tested the issue locally, and then gone, it's solid. I know how to cause the error, I put a fix in, I validated it, I've resolved it and solved the issue. But obviously, in this case, where it sounds like the feature is not working at all, clearly there is a situation that either they didn't predict or that doesn't happen on a local machine. Again, we talked about that.
That said, I will say that we have a very, very impressive setup for locally running what is a replica of what we play in live, which is amazing. It's an amazing system.
"After manually docking a vehicle, there is a chance for the docking bridge to lose collision, causing the actor to start to EVA." Most likely close to another issue we've seen on the command module about managing physics skip lists, which is how we skip elements from collision in certain cases from a given perspective.
You don't want the docking collar to collide with your ship and cause lots of problems there, right? There's a period in time when the two are trying to get together.
"Cargo services will not respond to hails, making players unable to request cargo loading zone," related to ATC and docking in this case. That's another one we're looking at.
TL;DRFreight elevator issues are being looked at, including missions with no cargo to collect and a generic overload error that is hard to fix without more granularity, plus a cleanup bug that can lose cargo.
Freight elevators, let's skip to that. So "hauling missions have no cargo to pick up at the collect location." So this is basically when you get to your mission location, pick up this Adonite from this location, and you get there and there's nothing to pick up.
Yes. That's a bridge between missions and freight elevators, which is interesting, and which requires specific people in this case. "Freight elevator is not working and displaying an overload error, preventing cargo pickup." Do you want to speak about this one? Because I think we talked about this today.
Well, I think we talked about the overload error. It's one of those that's a little bit too generic, that can imply many different things. Similar to the ASOP saying your ship is unknown. It becomes a bit of a catch-all.
People pull up and it looks like a small box somewhere, but they're getting an overload error. The system thinks the small box is a much bigger box kind of thing.
And like Philip said in the letter you so eloquently read out, obviously we fixed tons and tons of bugs on freight elevators as part of a long-running initiative. And we've probably fixed lots of overload issues as part of that. But because overload is a catch-all, unless you get more granularity...
Well, it's a variation of that really, because the freight elevator looks at the volume and will make sure that it's got stuff that you own in it and that there's no overload on it. So that's one of the things.
That's an example where there's another bug further down, which is "you have a chance to lose your cargo by activating the freight elevator kiosk." Which is basically a malfunction of that cleanup to prevent the overload. So that needs to be solved.
TL;DRThey agree the cargo deletion violates their principle that player items are sacred, say it comes from a cleanup meant to prevent another issue, and want to find the root cause, possibly removing the cleanup entirely.
But that also highlights, the same as in the inventory, there's this fragility in the transition between the physical world and the virtual kind of UI world that a lot of the inventory lives in. It's a visualization of the back end; it's a virtual item in a way, and the physical item.
Yeah, it could be the cleanup, right? It's something added to prevent another issue, now causing an issue that's worse than what we were trying to prevent in the first place. Again, like Philip said in his letter, this is one of the places where we want to go back and go, right. So it might mean that in the short term you're not seeing lots of individual fixes, but we're going to go, what's actually at the very bottom of this?
Just one last one on freight elevators before moving on: freight elevators going completely inoperable, which is still a thing, which we've had multiple fixes for, but that's one that we definitely need to iron out.
TL;DRThey list quantum travel and star map bugs they are looking into, including route resets, no plotting around obstacles, and tunnel exits landing in the same system, and say the old routing code has changes coming in the future.
So "quantum travel route resets after setting route." It needs to be flawless. Because that's, again, a traversal system, right? The star map becomes the routing system.
And I think, again, from a gameplay point of view, I don't want to sound super apologetic, but the routing is one of those that was written a few technology iterations ago now. We've got some interesting changes coming in the future.
Well, so the first one is it resets. Next is "doesn't plot around obstacles." Next is "while in station, destination on opposite side won't set route." Those are the main ones, I think. There's more obviously being worked on, but those are the main ones I wanted to call out. On QT, "tunnel exit can be in the same system as entrance," where you basically go through the quantum jump successfully, you're piloting it right, you're doing all the things right, you get to the end, and then you're back in Stanton. Obviously, that's not great. So we're looking into that at the moment.
TL;DRYes, they want to avoid teleporting; ship entry should no longer teleport you back if you walk away and should slide you into position, and they hope to walk players over for pickups while letting them break out.
The direct answer is yes. Obviously teleporting is not nice and not something we want to do. It's not something that's very safe to do in a multiplayer environment. I mean, someone else could be there. So it's something we really want to improve if we can. The ideal scenario is you guys are right on the money when you use your enter interactions and everything looks great. But of course, we want to be generous. We don't want you to have to stand on pinpoint accuracy to hit that ship enter or get the soda from the drink machine. So there's various things we're doing, and some of the things should already be coming your way in 3.2. One of them would be the ship enters: it used to be you set it going, you can run away and then it teleports you back. Now it shouldn't teleport you back anymore. If you run away, that's telling us you don't want to get into that ship anymore. And then if you're close enough, we should slide you into position a bit better. But what we're hoping to do is more the things we're doing when you choose to pick something up, where it actually walks the player over if it's close enough. Obviously, at any time, you should be allowed to pull out and say, hey, I actually didn't want to pick that up, I didn't realize you were going to move me. We don't want to take control away from you. You're the player, you're in control. The kind of animation is the length of which we don't want you to have control. Even then, wherever we can, we should allow you to break out.
It's a delicate balance like everything else with Star Citizen. You want it to be as diegetic and realistic as possible, but you have to measure that against time and technology and priorities and whatnot.
Everyone wants different things as well. Some people are very happy with if you click to use a vehicle, it will just teleport you into the seat and off you go. Zero second delay kind of gameplay. And our animators will obviously love you to see the beauty of the things they create.
TL;DREVA between fast-moving ships isn't planned for the immediate future but isn't impossible; it would need a gameplay reason plus ways to attach to or match a ship's speed, with technical concerns.
So the boring part first is obviously this is not something we got planned for the immediate future, but it's not something that's going to be impossible either. From my point of view, I think there's a lot of things that would need to be done to EVA to make this a fun feature. And it usually starts with a reason, like you would need a reason to catch the pilotless ship. Once we have a reason to pursue this feature, then we could jump on it. But there's things like, I think you should be able to attach to the surface of a ship so you could hold on to it somehow, whether that's like a physical attachment or you've got some kind of tractor beam you can grab onto it with, or if you can sort of get into the ship's influence in some way so that your relative speed in EVA is always moving with that. So I think otherwise, small adjustments that ship makes and the power a ship has compared to your little EVA process will always make it very, very difficult to catch up to anyone else. There's always technical concerns, would be things like how well will the animation-driven ragdoll hold up when you're traveling at such insane speed inside the world grid, because while you're inside the ship, there's no problem because your relative speed to the physics grid is essentially zero, right? Even though the grid is traveling full pelt. So that would be a technical concern. Another thing is currently the EVA thrusters are like safe. So they say, this is how fast you're allowed to go. This is the maximum rate of acceleration you're allowed to achieve. So the moment you leave the ship, they would start to decelerate you to the safe operating speeds. So these are considerations we would have to get on. But like I said, I don't feel this is something that would be impossible. I always remember the Gamescom floor demo when the guys jumped out of a ship and then had someone free fall and then tried to catch him with the ship again, which is really, really cool, which makes me think of this feature.
No, it's all 3.1 bugs to be honest. It's nothing spectacular. Trying to get some of the nice-to-haves in as well as fixing a lot of the serious crashes and things that are stopping us from releasing. So yeah, there's a lot of that.