There were two
Destiny 2 has been having a rough few weeks. It’s not like Bungie didn’t bounce back, but it’s been rough nonetheless.
Several weeks ago, a potentially catastrophic bug wiped out a lot of data following a very casual afternoon reset-time hotfix. It caused a game-wide panic and Bungie had to take down Destiny 2 for the entire afternoon and evening to fix it. After they identified the problem, accounts were rolled back before the bug. Crisis averted.
Then last week, a potentially catastrophic bug wiped out a lot of data following a very casual afternoon reset-time hotfix. It caused a game-wide panic and Bungie had to take down Destiny 2 for the entire afternoon and evening to fix it. After they identified the problem, accounts were rolled back before the bug. Crisis averted.
Déjà vu, right? Well, it looks like Bungie has identified the issue and is aiming to not have it happen again.
In a huge blog detailing the incidents, Bungie breaks down how inventory management works server-side, and how back in October the publisher “spun up additional servers” to help with the CPU load of the Shadowkeep expansion. What happened is as follows: “one issue was that a small percentage (less than 1 percent) of these servers would crash on start-up due to the volume of servers overwhelming one of the backing databases. Our workaround for this was to simply manually restart the crashed servers each time we detected this issue, and this appeared to address the problem without any discernible side effects for players.”
Going forward in time to the hotfix deployment, the servers crashed again, and according to Bungie, the initial crash “resulted in those WorldServers not applying the previous character data corruption fix.” When actually testing that hotfix their entire team was “lucky enough” to just experience the “good servers,” which resulted in an “all clear” decision to pull the trigger on the hotfix.
So how are they going to make sure it doesn’t happen again? Well servers are going to be given preventative maintenance of sorts, WorldServers should crash less, a “permanent fix” for the corruption issue is coming in the new update, they are working on new rollback and recovery systems, and are cleaning up their code in general (a long-listed complaint for Destiny, dating back to the original game).
Well, live and learn. Not being able to play a few nights is a small price to pay for having all of this addressed and fixed. Although if it wasn’t, Bungie would have had an even bigger problem on their hands when it comes to player retention.
Destiny 2 Outage and Rollback [Bungie.net]
