Google with a late night announcement

Ah late Friday announcements.

Google have explained how they deleted Unisupers account - their $125billion pension fund customer in Australia causing two weeks of downtime

“Unprecedented” Google Cloud event wipes out customer account and its backups
UniSuper, a $135 billion pension account, details its cloud compute nightmare.

Now first. Just my opinion. I know Google still want to seem all, whimsical. With their bean bags and whatnot. But if I’d been writing this, after being the first major cloud provider EVER to accidentally delete their customers account. Not an SMEs account, but a $125billon pension fund that made the news worldwide…I might have used ‘Executive Summary’ rather than ‘TL;DR’

It’s a bit of a serious situation, not a Reddit post.

I will also translate into simple language for the non-technical - when setting this up they failed to enter a value in a box, so it defaulted to one year, therefore in one year the whole thing deleted itself. The end.

No ‘Are you sure this should be blank?’. No ‘Can’t submit this until this is filled in’ warning. No checks at 30, 60 or 90 days out. No checks by the Account Team (pft, only a $125billion business). No alerts to the account team along the lines of ‘Your very large customer is about to be deleted’…

Update risk registers accordingly.

Details of Google Cloud GCVE incident | Google Cloud Blog
Google Cloud shares details of an incident impacting one Australian customer’s use of Google Cloud VMware Engine. Learn what happened and how we’re preventing it from happening again.
TL;DR
"During the initial deployment of a Google Cloud VMware Engine (GCVE) Private Cloud for the customer using an internal tool, there was an inadvertent misconfiguration of the GCVE service by Google operators due to leaving a parameter blank. This had the unintended and then unknown consequence of defaulting the customer’s GCVE Private Cloud to a fixed term, with automatic deletion at the end of that period. The incident trigger and the downstream system behavior have both been corrected to ensure that this cannot happen again.”

Subscribe to Gary P Shewan

Don’t miss out on the latest issues. Sign up now to get access to the library of members-only issues.
jamie@example.com
Subscribe