Google Cloud čÆÆå 大ååŗéåŗē”č®¾ę½åę°ę®äŗ件
Source: Pragmatic Engineer
A $124B fund in Australia would have lost all data stored with Google Cloud, had they not relied on a third-party backup. A rare blunder from GCP, where regional replication did not stop the deletion ā and a just as rare statement from Google Cloudās CEO taking the blame.
The below is an excerpt from The Pulse #93: OpenAI makes Google dance, originally published on 16 May, 2024. I am republishing it because on 20 May 2026 Google Cloud has done it again: they took offline cloud infra provider Railway by blocking Railway's Google Cloud account. The below is a warning: if you are on GCP, have a plan B, should GCP delete your account and data, or block your account.
Following on from Google Cloud being a distant third among cloud providers, a recent event could cement its reputation as the least reliable of the top three.Ā
UniSuper is one of the largest retirement savings accounts in Australia, used by 615,000 citizens, thatās also known as a āsuperannuation fund.ā UniSuper has $124B of assets under management and is one of the biggest in the country.
On 29 April, the service suffered an outage. Members could not log into their online accounts, or manage their funds until two weeks later, on 15 May.
The reason was that Google Cloud accidentally deleted UniSuperās subscription, which also deleted all data associated with the subscription. UniSuper had set up replication across two regions in Google Cloud to protect from a regional failure, but Google Cloud deleted the replica as well!
UniSuper could only avoid data loss thanks to having a backup on another service provider outside Google.Ā In a surprising admission, UniSuper would have lost all data with Google thanks to the failure of the cloud provider. The only reason UniSuper could restore services was by having another provider with whom theyād backed up the data. Basically, UniSuper not trusting Googleās replication across two regions turned out to be a 100% correct assumption. Whoever pushed through the decision to spend additional resources in āa backup in case Google failsā saved the day at the retirement fund.
The incident is incredibly embarrassing for Google. UniSuper seems to have forced Google Cloudās hand by issuing a joint statement with Google Cloud CEO Thomas Kurian, in which Google CloudĀ takes all the blameĀ for this failure. In my experience, the situation is rarely this black-and-white, as it usually takes two parties to cause such a major outage. I would not be shocked if it turned out UniSuperās staff played a role in this failure, but Google Cloud made enough mistakes that the press release could dump all blame on it.Ā I asked Google Cloud if the press release really was a joint release, and if they had more to add. The company confirmed the press release is correct and added nothing else.
Whoever was at fault, two weeks of downtime is still very long for a major fund. As I understand, the damage to UniSuper is mainly reputational because the funds are safe and secure.Ā
Users could not see their balances for a few weeks, and were told Google Cloud had messed things up. This means there are up to 615,000 Australians in whose minds UniSuper and Google Cloud are indelibly linked with unreliability.
I keep seeing that Google Cloud has no apparent strategy for what it wants its cloud to offer. A few months ago, we dived into howĀ AWS, Azure and GCP respond to regional outages, and I concluded itās hard to see a strategy at GCP beyond following processes, while doing the least impressive job of all three cloud providers. Itās hard to gain market share if you remain the slowest to respond to regional outages, and the provider for whom a zone outage takes down a region, or which loses all customersā data, despite regional replication, by deleting it.
This incident is a reminder you shouldnāt fully trust your cloud provider.Ā UniSuper was smart to have backups elsewhere for its data in Google Cloud. And while itās tempting to point fingers at Google Cloud: there are no definite assurances that another vendor would not make a similarly unprecedented mistake in the future!
The learning is that if you have really valuable data, keep a backup somewhere else. If you use any cloud provider, useĀ anotherĀ cloud, on-prem backups, or something else.