Jared Huckaby

Well, it started with an attempt at a bug fix a month ago. So with this 4.8 release and the way we've been doing patches this year, we have two streams when we work off from the working set. We have a base stream that we use for the base release, so 4.8.0. And we have a hotfix stream where we bring all the continuous changes along. So there was an attempt at bug fixing something on the Idris that was actually creating some performance issue somewhere else that was checked into that stream and nobody saw it and it went through. So we've been on the hotfix stream now for 4.8.1, now going to 4.8.2. In 4.8.2, we went back to the base stream, which are supposed to be at parity at this point, except additional content because we've added additional content in the patch. That change was included. And so what's supposed to be a routine update became a massive, wide-sprawling issue because basically the Idris was no longer compatible with its persisted copy. And so what that would do is players have in their ASOP terminal a version of the Idris that's from 4.8.1. This change, which was not supposed to be in the stream at all, gets in there. And so now the game server is expecting another shape for this. The game server has a series of checks and balances for this, but he's the last banker in the chain. And so for him, if the schema is incompatible, he's like, there's data corruption risk, I'm out.

Who's he in this analogy?

Well, he's the game server, basically. So the game server is trying to spawn the Idris, wherever it is, right? So that's called the async spawn chain. And so as he's trying to resolve the object containers for this and to materialize the ship in game, he comes around an incompatibility. And then there's a hard check for this. Because if we let those through, if we disarm this check, for example, then the game server might corrupt ships or might be twisting data and then we would go wider than that. And so the game server would crash.

And so every time an Idris would be attempted to spawn, that server in that region would crash and then restart and then try to load the Idris again and then crash again. And so as players were unstowing more Idrises in more shards, more servers got affected, until you reach a point where now the ops team is in the background and they're lighting all the fires, calling the Rohan. And so we're looking at this and basically we start cordoning off shards, because at the beginning we're not sure what the issue is. And what happens in this case is the more shards are getting offline, the less capacity we have for players that are either entering the game or shard hopping. We have a lot of people shard hopping. And so what happens is they hit the dreaded 60012 error, which means, hey, there's no capacity in your zone. There is capacity, but we've had to restrict it in the meantime. And so we took a hard decision later in the night to prevent players from spawning the Idris temporarily, just to stop breaking other shards. We brought more shards online to keep people playing. And then we basically went through and manually rectified. That's what we call percussive maintenance. When you see that in the status, it means we went in and we did a drastic measure to basically fix the Idrises that were either spawned in shards or still left in inventories, to make sure that they had the proper schema. And so now we know that all the Idrises are fine. We've lifted that temporary stoppage for that ship. And so now we're back. That's what happened.

This is cabaret trays being spawned by some subcomponent in the Idris cargo room.

And it's not even anything specific. In this case, it's the Idris, but it's not because it's a capital ship or anything.

No, it could have been any ship in that case. So normally, there's a check-in discipline that we apply, especially this year, because we have a different publishing model this year. And so ships that have been distributed already in a given major-minor patch cannot be modified. Now, for some reason, this check-in passed all the validation checks and ended up in the build.

Well, for me, if I take it from the machine room side of things, we've been firefighting since we launched 4.8 from a whole slew of different issues. Some of them are to be expected when we launch a patch that has this many core changes. We've touched on systems that are really central to the traversal again. So we're talking about the entitlement system, which has not changed in years at this point. And we've completely overhauled the entitlement system in 4.8 to power in part the item imprint system, but more importantly, to pull the entitlements out of the patch cycle and persist them across patches. What we call the LTP layer is not just what you have in your inventory, it's also what you own. Your ships and all of the items that you purchase in-game now have an entitlement record that holds the insurance loadouts for it, all of the customizations that you make on it. And that's hosted outside the game. Not outside the game, but outside the game patch. Which means that when we come back from a patch, there's no risk of, oh, I'm not getting my ship back. What you see in ASOP in the first patch will be exactly the same data source in the second patch. And so that risk of losing ships across patches is completely gone. So we're very happy. In fact, that bit actually went very well in 4.8, even though up to the launch, it was always last-minute things to look at. But we have been firefighting across a multitude of different issues that were tangential to that. Some of them we can touch on a little bit, because I've made some notes to talk about gameplay systems here and why they were affected by some minor infrastructure issues that turned out to be massively contributing to the perception of the patch. So I'll talk a bit about that in a bit, but basically we've been firefighting since 4.8.0. I want to touch on one thing because we just had an off-the-cuff moment about the Idris last night. But two weeks prior to that, we had another downtime patch that needed a lot of rectification. So the teams that run the online group were hard at work on this maintenance. And unfortunately, another one of those unwanted check-ins came in during that 4.8.1 cycle, where, in an attempt to fix the VGL flight suit that was missing hands and feet, another check-in came in that removed sub-components of this flight suit, but not just removed them from the flight suit, deleted them from existence from the game. And that's not something that you can do in a minor patch. And so clearly, that's two instances in this patch where transition check-ins should have been validated and prevented and went in, and then after that became an infrastructure problem. Because same issue here: if somebody was wearing the VGL flight suit and referred to these entity classes that were actually representing, I think, the nozzle and the front pack of the suit, then the game server would crash. And so we had, again, to do percussive maintenance to remove those items and fix that. And so that highlights to me a massive issue with how we validate check-ins and how we get changes in the game across persistence, until we have full forward porting compatibility on the persistence layer, which we don't have at the moment.

What's your perspective, Jens?

Live servers, you're always running, right? We're talking about forward patching. If we'd been behind closed doors, you would have done that, right? No one would have ever seen the game until you'd done that piece of technology, but that's not who we are. We're always running, and that's what makes it really, really exciting to work here as well, right? Because we're putting ourselves out there all the time. But if I look at this specific case, you're saying, right, we have checks and we have contracts on what you can and can't do in these sub-streams that we've already released, where the data management is super, super strict. And obviously now we invalidated them twice in a row. So it's time to look at how we really add that extra layer of protection.

I think there's a part of that which is that we're still transforming the game into an MMO. We're still in the process of taking a game that used to run on a single server and now obviously a server mesh, which is working, but the game systems need to expand as well. If you add the content pressure on top of that, of getting content into the game, releasing it at the same time, stabilizing it, hardening it, and features that come in and do the inverse. New features come in and they destabilize the systems. Without stronger guardrails that also need to evolve with this, that's where we ended up. And so, to be honest, I think right now we're outpacing our ability to verify everything that's being added to the game. And this is where we need to put a lot of our energy at the moment, to put in those guardrails, so that the check-in that would cause this crash issue, right? Those are identified at this point. And so it's all about adding these validation frameworks to prevent that from happening and keeping the data stable and fully validated. Now, that sounds... well, why didn't you do that before, right?

You're not saying that there were no guardrails in place.

No, no. There are guardrails in place, but they're not keeping up with the pace. We create more content and more features than we have time to add guardrails and to prevent these cases. An example I could give on something unrelated to persistence: we used to make a weapon, we make a ship, that's it. Now we have a team making weapons. They balance the weapons. They design the weapons. They work in that realm. They have guardrails in there. They know what to check for, outliers, data sets, how power is distributed, et cetera. You go to the next team. The next team is making ship components. They have their guardrails. They have their approaches. They balance it out. You have another team that's making recipes and dismantling. They have their own guardrails, their own systems. You have another team that's making economy. They add those things to shops. They add pricing. They make sure that that's balanced. And then it moves on to another team that's adding this content to loot boxes. Now, we didn't have that a while ago. And then mission rewards are added in there. All of these have validation, but there's no holistic validation across. We're at the point now where all these systems communicate with each other all the time. And crafting has changed the game quite a bit in that area. And so that translates into more weapons being created, more content being created, more missions giving more rewards, and all these things need to be validated together, not in silos.

And you're saying that's because the weapon and the pricing and everything for the weapon was validated by one team, in one silo, if you will.

Right, yes.

The validation of the crafting and the breakdown was done in a different one. And there wasn't something that...

Transversal to check all of that. And also, we're definitely now, post 4.7, in a realm where just human validation is not enough, right? And so a lot of game design data is typically managed by the team, but I think that we're way past that and we need machine validation for everything, because the rate at which we build it and the amount of technology we're putting in is just too fast for validation to keep up.

And we can't just rely on endpoint validation. Because in this case, it was machine validation that caught the error and stopped it from corruption.

Yes, exactly.

In this case, it took the situation of how the database on 4.8.1 was, with the new data from 4.8.2, to create the situation. And that's not something that the QA environment has. So whatever we do, we as a technology group would have to come in and at least give them that. And then if you put your 500 QA on it, then probably one of them would have spawned an Idris. But without that environment, you could put as many as you like on it. No one would ever have discovered this situation. This is just too far removed. And I think that sort of comes back to the large dataset, right? As you have a large dataset, you just can't scale in manpower alone. You need something that scales way beyond that.

I suppose if you just hired 500 QA people, that doesn't mean there's 500 requisite programmers that can all take the stuff and fix it either.

I think that would meet some frustration with their output, and our ability to tackle it would certainly create a separate set of challenges.

Also, what's frustrating with this situation is that we've done multiple patches where this just worked fine.

Well, so what we know about right now is that there are some meshes, and when I say meshes, I mean a dedicated game server inside a shard, that are running way below the FPS spec that we want. And so we don't have a handle just yet on what is actually causing it. We do have performance captures, and we're tracking at least six to eight very specific optimizations that need to happen. One of them is within the area component, which is used all over the engine. Area and zone component, they kind of go hand in hand. Again, because we're pushing more content, the hierarchy tree of the game world is much bigger. So there's more areas, more zones, more subdivision, more, more, more. There are some areas of that where you add a lot of entities that enter or exit these areas, and the engine provides the listeners to track entities entering areas or leaving areas or zones changing, right? And obviously there's concurrency constructs around this to prevent one thread from accessing another thread, right? And so, just keeping it simple without diving too much into the low level, which we will probably do in some of our desync areas, there's lock contention in the area component at the moment in some places. There's lock contention in some areas of the game. Well, I literally mean area: if there's many entities in this spot, because of the way it's configured, we're seeing basically FPS drops in those areas. And so we're tracking, like I said, about six to eight major optimizations that we're trying to get in our hotfix channel, to try to get this FPS back up to the spec that we want, which is above 20, across the...

Server FPS.

Server FPS, correct.

Somebody's going to take that clip and say Star Citizen's only looking for 20 FPS in gameplay.

No, no, no. And it's important to note that when a server is running low FPS, it doesn't mean that it will create desync, but especially when you're interacting with entities that are simulated on that DGS, AI characters, NPCs, then you're going to perceive these things, because the source is slow at creating its output. The traffic goes from your game client to the replication node, replication node to game server. Now, if the game server is running slow: at the beginning of the frame, he's taking the state from the network to the game and applying it. At the end of it, he's applying the changes back to the network. If that loop is slow, then your client's going to predict something, and then that prediction will get corrected the next frame, and that will create a rubber band, right? And so that's the gist of why there's desync in general. Why does the client have to predict? Because if he doesn't predict, then you're going to be at the tick of the server, and you'll see...

Even if it was running perfect, you'd see it in an FPS. Because your need for it to be so, so responsive is too high.

That's one of the things that we're looking at, how aggressive or not the ship cleanup is actually happening. That's one of the things we're looking at, yes.

Because you can watch a Twitch stream or look at an Issue Council report and you'll see dozens and dozens, I feel like it's even more, of bricked ships everywhere, with the insurance changes bricking these ships. Is this viably part of what it is?

In part, yeah. That for sure doesn't help, because the more entities are replicated, the more the server needs to do every frame, and so that contributes to this. So, yes.

You can usually find it when you look at how servers are performing. It usually has a larger vehicle population, certainly.

A server that is performing great has a larger...?

No, a server not performing so great tends to have a slightly larger vehicle population or a very large active population. That can also happen. In general, it's just a lot of everything. We're quite a heavy game, entity-wise. We want to avoid getting to the sort of size that we were in Stanton before we went into meshing, right? Because that was consistently difficult to manage.

I know that there's been some changes related to how we perform entity cleanup. So unattended vehicle cleanup used to be a separate system that ran on its own. That's been merged into what we call the density manager, which is basically an engine functionality that will look at overpopulation in clusters of areas that designers obviously have control over. For example, we want to make sure the exit routes for hangars are cleaned up. And so these have active cleanup zones configured, right? That's designer controlled. Now the engine will then apply the logic on the volume, determine, oh, there's too many of X in this area, and will start popping stuff out, right? Basically going to do the cleanup. Now for ships, the unattended vehicle cleanup system used to be separate. Now it's been brought into the density manager, I think end of last year, something like this. So it's been a while. And this one tracks whether there's player passengers on a vehicle and will, based on a certain timer configuration, and the values are different depending on the areas, either stow the ship back into the originating inventory or destroy it, depending on if it's a player-owned ship or if it's an AI ship. It does look like maybe there's a misconfiguration in key areas that are leaving ships there. So we don't think the system is incorrectly functioning. We think it might be inaccurately configured in some areas. And so that would allow a very popular area to just accumulate ships like crazy. And so that's what we're looking at right now.

I'm not sure if the density manager will have a modifier specifically for bricked or not. But it can see it.

It does or it will?

It will. If it doesn't, it will eventually. And it's less about timers. It's more about density. It's trying to say you have overpopulated this area. So it's less about that ship having been there for a long period of time. It's more like, there's a lot of ships here. Let's remove one. Let's figure out which one is best to remove. Unattended vehicle does have a timer as well. So it's sort of like we talked about with validation. You start with a timer, then you start adding density, then you start building up these views of the world to understand which areas are problematic. Like, oh yeah, the entrance to a hangar, we have to be super strict.

There's a place, as we're talking about this, something that pops into my mind that I think is also in our process. Not even the development process, but let's call it live operations of the game. We're very lean in terms of our live operations at CIG, because we're in development. And so even though we do run a live service, we try to run a really lean ship on that side. And so we don't have an army of site reliability engineers that are looking at every aspect of, oh, shard 25 has an overpopulation of ships at Jackson Swap between this time frame and this time frame. We don't go to that depth. And so we track overpopulation across all shards, but we can't go granular into this. But that is an area where we could improve this, detect those things and better tune these values. Typically right now we will tune the density manager values every patch or every hotfix, but that is a slow cadence compared to how you would react if you see, oh my god, on every shard this location is loaded, let's effect the density manager change right now. Right now we don't have the ability to do that live, and that's a problem. That would improve the game quite a bit.

For me, it's behaviors. So there's one thing about running a game server, which is that it's kind of an organic behavioral system, right? And so you can have a game server with a version of the code and a version of the data running a certain way, and then players start doing something different, and that completely changes the performance profile. And so I feel like in 4.8, most likely, my hunch for these six to eight performance issues that we're looking at is that it's related to some Imprint changes. So the way people use ships, leaving bricked ships behind: are there more claims than before? Are there other egregious issues that showed up that are creating more claims that would affect the amount of entities on the server? Is there a malfunction on, for example, the density manager cleanup that is leaving items in place? Or are people just going to a location that's configured to be very loose on its density, and it's somehow the location that people have decided to go to in this patch? So sometimes the performance profile of the game servers changes just based on behaviors, not on, oh, we changed something.

You're blaming the backers.

No, I'm blaming the features that cause behavioral changes. And so for me, that's what comes out of it. You're talking raw server performance here.

What are you thinking, Jens?

I mean, it's hard not to just parrot what you just said. It does feel like, when I think of 4.8, there's nothing that I can point to and go, yeah, those gameplay changes we made could have caused it. The only thing I can think of is, we did performance improvements, but there is a large change in how we handle vehicles and the lifetime of vehicles.

You mean the changes with the imprinting and the insurance system?

Insurance, yeah.

You're bringing a big chunk of it, not everything, back to the number of ships.

I mean, that would be, based on what I know of what that patch put out there. I just don't recall there being any larger changes on the gameplay side that I feel I could explain it with.

I don't want to deep dive into each of them, because it's a bit intense if we go down that route. But there's navigation system dispatching of tile configuration changes on planets, which affects our AI using NavMesh on planets. So there's a performance cost on that that is costing us a chunk of frame in the servers. This is purely based on the servers we know are affected by this problem.

There's a lot of digging through a lot of data to find this common culprit that they know is going to make a meaningful impact.

Yeah, exactly. This is cross-linking a lot of things. This is a pet peeve of mine that I keep pushing on, and we're going to definitely resolve that from a software point of view: there's an entity spawner in some of the areas that are affected that is coming in and out of streaming and respawning its spawn set every time. These are the NPCs that just spout out. That's one example; I have multiple others related to this. That's one. The area component we talked about. So what we call register with external systems is a phase when a component or an entity is coming online, when it's streaming in. It needs to register with external systems. That's a notorious area. That is an issue on multiple components. This one, from the area component, is a problem in that place.

So the first two are NPC related?

Correct. This one is more engine and, let's call it, area configuration related. This one bleeds into the zone lock that we talked about earlier. So those are two, area and zone lock. So this is really tied to the amount of content in the game and the complexity of that content in some areas. Because we're also making more and more complex scenes, like space stations and the content we've put out recently, whether it's TSG or Rockcracker, there's a lot more complex layouts, and those elements contribute a lot to this. And so those are part of that. The ship cleanup is another big one that we're looking at at the moment, which is probably the source of the other two, which is the amount of registered entities that have either MFDs and/or the flight system active.

Entities that specifically have MFDs? Why MFDs?

Well, we don't know specifically. I bet that if I went into that area and I looked, there would be a bunch of MFDs that are somehow visible. So these components have a visibility aspect to them where, if you look at it, it's going to replicate for you. But if you don't look at it, it doesn't. Might be a malfunction with that. We're not sure.

So the ship is bricked, but the MFDs are still visible.

It could be broken ship MFDs. We've seen weird things; everybody knows about the graveyard. There's been other graveyards in other areas, right? So there might be a new one we haven't found yet where somehow MFDs end up there.

The locations are getting more complex and the area system is not properly calling them?

Yeah, it's an optimization at this point. The functionality is correct, it's just a lot of it.

And one final one, that is fire. So some ships catch fire. And depending on the size of the voxel grid, we've seen that the area physics properties of how the temperature system interacts with fire actually need a big optimization, and it's costing us frame time. That's a combination thing, because internally we can't reproduce it, but we see the cost on some of the game servers that are running slow.

"Player loses access to persistent hangar at primary residence." It's a classic one. My character is actually in this state right now.

Where is your hangar?

Area 18. The hangar is actually there, but I never spawned it, in my case. But in these cases, with most players I've discussed with, the hangar is actually stowed in another inventory somewhere else for some reason. So that's the hunch so far. We'll see what the team actually comes up with, but that's one that we definitely want to solve. Now, I've got more on the hangar side, but I figure in the interest of time I would keep going with the other categories. We've got login flow related issues. So we talked a bit about shard lock. That's one of them. So basically being a bit more communicative about shard lock, and no longer having shard lock at all for players that stay in the shard. 64008 spawn resolution error and its little brother, 64006 disconnection. Again, resolve player location. I can speak to this one a little bit. This is typically a symptom. If we fix this issue specifically, the display will change. But what actually happens is that when you connect to a game shard, the replication layer, so the hybrid that's in front of it, is going to coordinate bringing your character into the game world. But it doesn't know at the start which of the 10 servers it's running your character should be on. And so it needs one of those servers to help resolve your location, so that we can unstow you in the right area in the tree and then hand off authority to the server that's managing this region. The way this is done is that the replication layer will hand this work off to any of the game servers on the shard. Now, when we have Idrises blowing up servers left and right, there's a very high chance that the server this is handed to is dead, or is in the process of dying and is rebooting. And so what happens in this case is that the player will get to the end of the context establisher tasks, the CETs, and he will get a 64008 or 12 error. We need to make this a bit more robust on that side, so that we know which servers are completely alive, and if a server is doing a reboot, we don't assign them a resolution task. That would be one thing. But also trying to streamline that so that it's a bit more efficient. A small one I wanted to touch on here was "friends list showing 0 out of 1 instead of the player counts per shard." So we're rejigging this bit. It's going to come back. This is a temporary thing.

Is that a priority fix though?

In the login flow section, it is.

Correct. Let's talk about docking. The very first one: "unable to dock using docking collars." That is problematic, isn't it? We looked at this this morning and it was actually resolved when we were talking. It is now reopened. So we're going to have to dive into that. But basically, docking is a subject that's always coming back. And this patch has also been contributing to the overpopulation of ships, because players would just basically land on pads and leave the ships there, where there's no impound. And so that's a big issue. So the first one is unable to dock using docking collars.

So I can quickly just go on: when you say it's resolved and then reopened, what happens there is the developers, to the best of their abilities, found the issue, they've fixed the issue, they've tested the issue locally, and then gone, it's solid. I know how to cause the error, I put a fix in, I validated it, I've resolved it and solved the issue. But obviously, in this case, where it sounds like the feature is not working at all, clearly there is a situation that either they didn't predict or that doesn't happen on a local machine. Again, we talked about that.

It works on my machine, but the second it got up to network validation, it fell down.

That said, I will say that we have a very, very impressive setup for locally running what is a replica of what we play in live, which is amazing. It's an amazing system.

We call them the CIG dev kits.

But yeah, unfortunately, in this case, clearly.

"After manually docking a vehicle, there is a chance for the docking bridge to lose collision, causing the actor to start to EVA." Most likely close to another issue we've seen on the command module about managing physics skip lists, which is how we skip elements from collision in certain cases from a given perspective.

Is this when you're trying to dock and it ends up just ejecting you out into space?

Yes.

You don't want the docking collar to collide with your ship and cause lots of problems there, right? There's a period in time when the two are trying to get together.

"Cargo services will not respond to hails, making players unable to request cargo loading zone," related to ATC and docking in this case. That's another one we're looking at.

Freight elevators, let's skip to that. So "hauling missions have no cargo to pick up at the collect location." So this is basically when you get to your mission location, pick up this Adonite from this location, and you get there and there's nothing to pick up.

There's somebody on that right now?

Yes. That's a bridge between missions and freight elevators, which is interesting, and which requires specific people in this case. "Freight elevator is not working and displaying an overload error, preventing cargo pickup." Do you want to speak about this one? Because I think we talked about this today.

Well, I think we talked about the overload error. It's one of those that's a little bit too generic, that can imply many different things. Similar to the ASOP saying your ship is unknown. It becomes a bit of a catch-all.

People pull up and it looks like a small box somewhere, but they're getting an overload error. The system thinks the small box is a much bigger box kind of thing.

And like Philip said in the letter you so eloquently read out, obviously we fixed tons and tons of bugs on freight elevators as part of a long-running initiative. And we've probably fixed lots of overload issues as part of that. But because overload is a catch-all, unless you get more granularity...

This will be affected by people throwing water bottles.

I was going to say, because that was that thing a year ago where people would put the bottle on top and just kill the entire...

Well, it's a variation of that really, because the freight elevator looks at the volume and will make sure that it's got stuff that you own in it and that there's no overload on it. So that's one of the things.

It does have good cleanup, to the point where it was cleaning up players recently.

That's an example where there's another bug further down, which is "you have a chance to lose your cargo by activating the freight elevator kiosk." Which is basically a malfunction of that cleanup to prevent the overload. So that needs to be solved.

Yes, this violates the first principle in the charter we're putting in place, which is that player items are sacred.

But that also highlights, the same as in the inventory, there's this fragility in the transition between the physical world and the virtual kind of UI world that a lot of the inventory lives in. It's a visualization of the back end; it's a virtual item in a way, and the physical item.

It sounds like delete's just happening too early.

Well, in this case, it's a function of the cleanup, yeah.

Yeah, it could be the cleanup, right? It's something added to prevent another issue, now causing an issue that's worse than what we were trying to prevent in the first place. Again, like Philip said in his letter, this is one of the places where we want to go back and go, right. So it might mean that in the short term you're not seeing lots of individual fixes, but we're going to go, what's actually at the very bottom of this?

So it looks like an issue with Wikelo, but it's actually an issue with the freight elevator. It's actually an issue with the cleanup system.

Or maybe we can find a way that we don't need the cleanup at all.

Just one last one on freight elevators before moving on: freight elevators going completely inoperable, which is still a thing, which we've had multiple fixes for, but that's one that we definitely need to iron out.

That's worse than an exploit. That's legal and legit in this case, because we put those prices in, right? It's not an exploit in this case, but we are tracking several potential exploits. Basically, we're not going to go into details on exploits, because that's kind of an area that we don't want to be too public about. You don't want people to be using those exploits if they are confirmed.

Well, we benefit from those people not knowing we know what we know. It lets us track, it lets us learn more about it, it lets us find not just that cause of it, but that cause of it and that cause of it. It's very easy to be frustrated when you see an exploit go on and continue. It's like, why the hell isn't CIG doing something about it? Well, no, we are. We're baiting the hook. If we smack this cause of it now, then we never get to see this one and we never get to see this one. So there are reasons why we leave these things going for perhaps longer than some people would prefer, because it's how we find out how it's happening, it's how we find out who is doing it. That's one of those ones where I could see why it's frustrating, but that's actually the process working.

So we're tracking exploits across a range of different types, whether they're environment exploits or economy exploits or duping or adjacent duping or mission bypass, things like that. So we're tracking a bunch of those. We have a great team doing this work, by the way, that supports the developers with all our crazy questions about what, who, when and how, and so these guys are doing great supporting us.

I've got a cheat sheet, because I think the only person who would be able to memorize it off the top of their heads is Lee, and Lee's currently not in the country. Otherwise he would be here and doing this. So, starting from the top, we have two size 12 triple ballistic cannons that are manned turrets.

And for reference, what's the biggest gun on the Javelin?

It is 9s on the big remote turrets at the back.

And the railgun on the Idris is a 10. So these are triple size 12s here. Is there two of these? One on the top and one on the bottom?

Well, there's two triples. So the Bengal, the large lower turret on the Bengal that people would have seen in the Squadron videos, that's size 12. So that's got four.

We have three single size 10 manned turrets. So that is, instead of having dual mounts, it's a single mount. We have eight, once again, size 10 single ballistic cannon remote turrets. We then have... I know, it's still a lot to get through. We have eight size 8 double cannons that are manned, and then there is two quadruple laser cannons that are also piloted. And those are size 8.

Essentially it's a size 32?

No. Think of it more like the one on the bottom of the Bengal, where it's multiple barrels. It's worth noting that the weapons for this ship, just because of the sheer size and scale and complexity of everything to do with it, are all specific. They're not just using our generic PU weapons. All of them are built in and designed to specifically work with each of the turrets. Because you can't just go down to your local shop and pick up a size 12.

Oh, yeah. Well, I mean, we did see it had some pilot weapons. It has a beam weapon of, yes, size... It is the entire front of the ship.

Indeterminable size.

It's a beam weapon with 12 barrels. It's six on each side.

Yeah, but it does combine to one giant beam. It's a beam on a capital ship of a scale... well, the size of the ship, as a random data point, is over 750 meters, and then we just have a giant beam that is the front of the ship. So... yes. I mean, we can give it a size, but fine. I'll also cover: it has a significant suite of PDTs, flak, and other automated defense systems. There is still hot debate about the specific number of this. It's either going to be 42 plus 1 or 43. We had an interesting problem when we were actually doing the PDT and flak distribution across the ship. Because they were so small, you couldn't actually see them. So what we ended up having to do was make clustered banks of them, just so that they were actually readable on the exterior. And there's just so many, it meant that we had nice little defensive clusters so they could cover each other as well quite nicely.

Well, the reason we need them to be readable on the exterior is because part of the gameplay of tackling a ship of this size is going to be peeling back the layers of defense. You're not just going to rock up with a couple torpedoes and do the job, you're going to have to take out the subcomponents, take out the shield emitters, take out the PDCs and defense before you can even begin dreaming of trying to either board it or disable it. If we just had the tiny little dots distributed everywhere, people would struggle to actually be able to engage with that gameplay. We want people to visually know, without owning the ship, where to target.

For as powerful as this thing is going to be, it still has to be killable. It still has to be assaultable. It's going to end up being its own TSG just attacking one of these.