← All incidents
Incident dossier · Rank #15
Google Cloud Accidentally Deletes UniSuper's Entire GCVE Private Cloud
Google Cloud (customer: UniSuper)
2024-05-02
216h 0m core impact
Human errorSoftware
During the initial provisioning of UniSuper's Google Cloud VMware Engine (GCVE) Private Cloud, one input parameter was left blank in an internal Google deployment tool. Per Google's disclosure the blank value caused the system to assign a then-unknown default fixed one-year term to the Private Cloud, which silently armed an automatic-deletion action. Roughly a year later, at the end of that system-assigned term, the Private Cloud was deleted across both of its copies. The A$135B fund's 647,000+ members lost access to online services for nine days. The deletion did not reach data backups stored in Google Cloud Storage in the same region, which — together with third-party backup software — enabled a rebuild rather than permanent data loss. Members' account balances and investments were not lost, and the incident was operational, not a cyberattack.
Failure cascade
Trigger → primary fault → downstream blast radius, derived from the sourced root cause and affected-services record.
Facility & location
- Operator
- Google Cloud (customer: UniSuper)
- Data center
- Google Cloud VMware Engine (GCVE) Private Cloud — UniSuper tenant
- Location
- Australia, GCVE Private Cloud replicated across two copies (joint statement: 'two geographies'; Google technical detail/itnews: 'two zones')
- Date
- 2024-05-02
Impact & scale
- Users affected
- 647,000+ members
- Financial
- Not quantified in accessible sources; A$135B+ funds under management exposed but no member-dollar loss or compensation figure disclosed (itnews and the 8 May joint statement are both silent on a figure)
- Scope
- Tier-1 (fund-wide member-facing outage; full private-cloud loss across both surviving copies)
Services / systems down
- UniSuper member-facing online portal
- UniSuper mobile app
- Back-end estate: hundreds of VMs, databases and applications
Impact data & metrics
| Members affected | 647,000+ (as of 31 Mar 2024) |
| Funds under management exposed | A$135 billion+ |
| Online services blackout | 9 days inaccessible / rebuilt from backups |
| End-to-end disruption window | ~12 days (2 May – 13 May 2024) |
| Copies destroyed | Both copies deleted — joint statement: 'two geographies'; Google technical detail / itnews: 'two zones' |
| Latent term before deletion | 1 year (system-assigned default fixed term) |
| Misconfigured inputs | 1 blank input parameter |
| Backups lost | 0 — GCS same-region backups + third-party backup software survived |
| Recovery cadence | 24x7 over several days (network/security config, apps, data) |
| Financial loss / compensation | Not disclosed in any accessible source |
Magnitude profile
Magnitude 8.2 = blast 10×0.35 + users 9×0.25 + financial 6×0.20 + duration 6×0.20 (sub-scores 0–10 · weighted composite)
Blast radius maxed: both surviving copies of the GCVE Private Cloud were deleted, an outcome geographic redundancy is specifically meant to prevent. Sources vary on terminology — the 8 May joint statement says the copies spanned 'two geographies' and that deletion hit 'both of these geographies', while Google's technical write-up and itnews describe 'two zones' of one Private Cloud — but either way both copies were destroyed together and redundancy gave no protection. Users score high (fund-wide, 647k+ members locked out). Duration mid (9-day member blackout, ~12-day end-to-end window). Financial mid and uncertain — no dollar-loss or compensation figure was ever disclosed and surviving backups prevented data loss, so exposure is bounded by service disruption not asset loss.
Sequence of events (SOE)
- TRIGGER Google operators provision UniSuper's GCVE Private Cloud using a deprecated internal capacity-management tool; one required input parameter is left BLANK. The system silently substitutes 'a then unknown default fixed 1 year term value' — the software analog of a mis-set timer arming a destructive action.
- TRIGGER The defaulted fixed term binds automatic-deletion behaviour to the Private Cloud lifecycle. No validation rejects the blank parameter and no confirmation step flags that a destructive 1-year term has been set — the latent defect is now armed.
- CASCADE Latent dwell period: the misconfiguration sits undetected for roughly a year, past any provisioning-review window. No telemetry, alarm, or scheduled check surfaces the impending destructive term expiry to Google or UniSuper.
- TRIGGER The system-assigned 1-year term expires. The automated lifecycle process interprets expiry as a deletion instruction and DELETES UniSuper's GCVE Private Cloud — 'After the end of the system-assigned one year period, the customer's GCVE private cloud was deleted' — an unattended automatic actuation with no human in the loop.
- CASCADE Because the deletion acts on the whole subscription, the redundant copy in the second geography is destroyed with it: the deletion 'caused deletion across both of these geographies.' Geographic redundancy — the primary designed 'suppression' — fails to contain the event because both locations shared fate with the deleted object.
- TRIGGER NO customer notification is generated: 'the deletion was triggered as a result of a parameter being left blank by Google operators ... and not due a customer deletion request.' The final opportunity to intervene at the moment of destruction is lost.
- DETECTION UniSuper's online services go offline; members of a major Australian superannuation fund lose access to their accounts. Detection occurs here — at the point of TOTAL loss, as a full outage — because no automated alarm or safeguard caught the deletion earlier.
- IMPACT Member-facing systems are dark. Both zones of the Private Cloud are confirmed deleted; the primary disaster-recovery mechanism (cross-region/zone duplication) is already known to have failed because it shared fate with the deleted subscription.
- MITIGATION UniSuper and Google Cloud mobilise a joint incident team — the emergency-response analog — to rebuild the destroyed Private Cloud rather than merely restart it.
- MITIGATION The safeguard that DID work is identified: out-of-band backups. GCS backups held in-region 'were not impacted by the deletion, and, along with third party backup software, were instrumental in aiding the rapid restoration' — surviving because they were a separate object, not a dependent of the deleted Private Cloud.
- MITIGATION A further out-of-band layer is confirmed: 'UniSuper had backups at another cloud provider,' preventing complete data loss — independent blast radius, not merely independent geography.
- RECOVERY Joint teams rebuild the GCVE Private Cloud, restore network and security configurations, restore applications, and recover data from in-region GCS backups, third-party backup software, and the second-provider backups. UniSuper's online services were inaccessible for 'nine days.'
- CASCADE Blast radius is confirmed contained to a single tenant: the event did not impact any other Google Cloud customer or service, nor UniSuper's other GCVE Private Clouds/Account/Orgs/Folders/Projects. UniSuper ran 'more than one private cloud' and only one was destroyed.
- RECOVERY UniSuper CEO Peter Chun and Google Cloud CEO Thomas Kurian issue a joint statement attributing the loss to an 'inadvertent misconfiguration,' calling it 'an isolated, one-of-a-kind occurrence' and stating 'This should not have happened.'
- RESTORED Services are progressively restored. No member funds and no account data are ultimately lost — recovery achieved entirely from out-of-band backups.
- RESTORED Google executes systemic remediation: deprecates the internal tool ('now fully automated and controlled by customers via the user interface'), corrects the deletion-on-blank-parameter behaviour, and scrubs the system database plus manually reviews all GCVE Private Clouds — concluding 'It is not a systemic issue.'
Root cause
There was no ignition source, no equipment failure, and no fire — this was an accidental, silent deletion of UniSuper's entire Google Cloud VMware Engine (GCVE) Private Cloud. The fire-forensic schema fields are mapped to software/technical analogs, and every gap is disclosed rather than invented.\n\nThe immediate ('ignition') mechanism was a deprecated internal Google Cloud capacity-management/provisioning tool operated by Google staff during the INITIAL provisioning of UniSuper's GCVE Private Cloud — roughly one year before the deletion (no source pins the exact provisioning date; the ~1-year gap is inferred from the term length and the early-May-2024 deletion). Per Google's official RCA: 'one input parameter was left blank when using an internal tool to provision the customer's Private Cloud. As a result of the blank parameter, the system assigned a then unknown default fixed 1 year term value for this parameter.' That defaulted term silently carried automatic-deletion behaviour: 'After the end of the system-assigned 1 year period, the customer's GCVE Private Cloud was deleted.' Causal chain: deprecated tool -> required input left blank by operators -> system substitutes an unknown, undocumented default 1-year fixed term -> the term expires one year later -> automated, unnotified deletion of the Private Cloud.\n\nThe latent root causes are design and process, all on the Google Cloud operator side. (1) Unsafe silent default: a blank required parameter resolved to a destructive 1-year auto-deletion term instead of failing the operation — a validation/guardrail gap Google later fixed ('We corrected the system behavior that sets GCVE Private Clouds for deletion for such deployment workflows'). (2) Deprecated tooling in a live provisioning path — Google 'deprecated the internal tool that triggered this sequence of events.' (3) A notification lapse: 'No customer notification was sent because the deletion was triggered as a result of a parameter being left blank by Google operators using the internal tool, and not due a customer deletion request' — removing the last chance for either party to intervene. (4) An in-band redundancy design that could not contain a subscription-level delete: UniSuper had duplicated the deployment across two geographies for disaster protection, yet, per The Register, 'When the deletion of UniSuper's Private Cloud subscription occurred, it caused deletion across both of these geographies.'\n\nNo maintenance, inspection, or testing lapse is attributable to UniSuper. UniSuper's out-of-band backup regime — Google Cloud Storage backups held in-region, third-party backup software, and (per The Register) backups held with a second cloud provider — is credited as decisive to recovery. Google framed the event as an 'inadvertent misconfiguration' during provisioning; the joint UniSuper/Google statement reported by The Register called it 'an isolated, one-of-a-kind occurrence that has never before occurred with any of Google Cloud's clients globally' and stated 'This should not have happened.' Google's own RCA concludes 'It is not a systemic issue.'
Contributing factors
- Unsafe silent default (Design/Process): a required provisioning parameter left blank did not fail the operation but resolved to 'a then unknown default fixed 1 year term value' carrying automatic deletion — a validation/guardrail gap Google later 'corrected' (Google RCA).
- Deprecated tooling in a live provisioning path: the Private Cloud was provisioned with an internal capacity-management tool Google subsequently 'deprecated,' replacing it with customer-controlled UI automation (Google RCA).
- Operator procedure lapse: a Google operator left the required input parameter blank during provisioning, with no field-level validation or peer confirmation catching it before it committed (Google RCA).
- Notification/telemetry gap: 'No customer notification was sent because the deletion was triggered as a result of a parameter being left blank by Google operators ... and not due a customer deletion request' — removing the last chance to intervene (Google RCA).
- Redundancy design shared fate with the deleted object: duplication 'in two geographies as a protection against outages' could not contain a subscription-level delete, which 'caused deletion across both of these geographies' (The Register).
- Latent-defect dwell time: roughly a one-year gap between the ~2023 misconfiguration and the early-May-2024 deletion meant the fault sat undetected past any provisioning review, with no alarm on the impending destructive expiry (Google RCA).
Correction of errors (COE)
- Deprecate the internal provisioning tool; automate capacity management via customer-controlled UI
- Correct the deletion-on-blank-parameter behaviour (fail-closed on destructive deployment workflows)
- Scrub the system database and manually review all GCVE Private Clouds for at-risk deployments
- Ensure operator-triggered deletions are preceded by customer notification, as customer-initiated deletions are
- Maintain out-of-band backups (in-region GCS, third-party backup software, and a second cloud provider)
Lessons learnt
- Defaults must fail safe. A blank field defaulting to an irreversible, delayed self-destruct is the antithesis of safe design; unsafe defaults on destructive actions are as dangerous as an explicit destructive command because no human ever consciously chose them.
- Redundancy is not backup. Duplicate copies protect against regional failure, not against a valid control-plane instruction to delete the resource itself — such an instruction hits every copy at once. True recoverability came only from backups held outside the deletion's blast radius.
- Latent misconfigurations with long fuses are uniquely dangerous. A destructive timer that stays invisible for a year defeats normal monitoring, change review, and incident-response reflexes because nothing appears wrong until the moment of loss.
- Out-of-estate backups saved the fund. GCS same-region backups plus third-party backup software turned a potential permanent-data-loss catastrophe into a nine-day service outage — the single control that most changed the outcome.
- Transparent, joint vendor-customer communication limited reputational damage: clearly stating this was an operational misconfiguration, not a cyberattack or member-data breach, and that member balances were not lost, was essential for a 647,000-member fund.
- Even hyperscaler-managed platforms carry residual operational risk; customers of managed private clouds should assume misconfiguration is possible and retain independent, tested recovery paths they control.
Improvements & remediation
- Safety/guardrail (fail-closed validation): reject blank required parameters on destructive workflows instead of substituting a default term — Google 'corrected the system behavior that sets GCVE Private Clouds for deletion for such deployment workflows.'
- Maintenance (retire deprecated tooling from production paths): Google 'deprecated the internal tool that triggered this sequence of events,' now 'fully automated and controlled by customers via the user interface.'
- Detection/telemetry: add standing alarms and human-in-the-loop confirmation on impending destructive lifecycle actions (auto-deletion timers), so a latent destructive expiry surfaces long before it fires rather than at total loss.
- Notification integrity: require customer notification before ANY deletion regardless of trigger source — operator-initiated deletions must not bypass the notice that customer-initiated deletions receive.
- Fleet-wide corrective audit (Maintenance): Google 'scrubbed the system database and manually reviewed all GCVE Private Clouds to ensure that no other GCVE deployments are at risk.'
- Out-of-band backup discipline (customer side): maintain backups on an independent blast radius — a separate service object AND a second cloud provider — validated here as the decisive control that redundancy could not provide.
Comprehensive analysis
Control plane vs data plane: where the failure actually lived
Nothing failed in UniSuper's running workloads. The defect was in Google's provisioning control plane: a blank required parameter resolved to 'a then unknown default fixed 1 year term value' that bound an automated deletion to the Private Cloud lifecycle. A metadata default became a destructive action a full year later, with no running-system fault ever involved — the software analog of an armed, unattended timer rather than an equipment failure or fire.
Redundancy vs backup: shared fate is the whole story
UniSuper duplicated the deployment across two geographies for disaster protection, yet the deletion 'caused deletion across both of these geographies' because both were dependents of the single deleted subscription. Redundancy defends against physical zone/region loss; it cannot defend against a control-plane action naming the parent. Recovery came only from out-of-band copies — in-region GCS backups, third-party backup software, and 'backups at another cloud provider' — objects with independent blast radius.
Detection failure and latent dwell
The misconfiguration sat undetected for roughly a year, past any provisioning-review window, with no telemetry on the impending destructive expiry. No alarm, anomaly guard, or human confirmation gated the deletion, and 'no customer notification was sent.' Detection therefore occurred at the point of total loss — the outage itself — after which services were inaccessible for nine days. The single most actionable gap is standing telemetry plus notification on destructive lifecycle actions.
Blast radius containment and remediation adequacy
Impact was contained to one tenant: the RCA states no other customer, service, or UniSuper GCVE Private Cloud/Account/Org/Folder/Project was affected, and only one of UniSuper's 'more than one' private clouds was destroyed. Google's remediation targeted the mechanism, not just the instance — deprecating the tool, correcting the deletion-on-blank behaviour, and auditing the entire GCVE fleet — concluding 'It is not a systemic issue.'
Disclosed gaps (not fabricated)
The sources accessible this session do not provide exact deletion timestamps, the precise UniSuper member count or funds-under-management, byte-level recovered volumes, or per-member downtime. Figures such as '~600,000 members' and 'AU$135 billion,' which the proposed draft attributed to The Register, are NOT present in that article or in iTnews and could not be independently verified here; they are therefore omitted rather than asserted. Timing is scoped to 'early May 2024' and provisioning to '~2023' because no source pins exact dates.
Technical deep-dive
This is a rare, cleanly documented example of a control-plane deletion defeating a data-plane redundancy design, and of out-of-band backups succeeding exactly where geographic duplication failed.\n\nThe failure lived in Google's provisioning control plane, not in any running workload. During initial provisioning (~2023), a Google operator using a now-deprecated internal capacity-management tool left one required input parameter blank. Rather than rejecting the operation, the system substituted 'a then unknown default fixed 1 year term value' (Google RCA). That term was not inert metadata — it was bound to a lifecycle action. When it matured one year later (early May 2024), an automated lifecycle process interpreted expiry as an instruction to delete the GCVE Private Cloud. Because deletion operated at the subscription/Private Cloud level, it took out the whole logical construct, and the redundant copy in the second geography went with it: the deletion 'caused deletion across both of these geographies' (The Register); the deployment ran 'across two zones' (iTnews).\n\nCentral architectural lesson: geographic redundancy inside a single logical subscription protects against zone/region physical failure, but NOT against a control-plane action that names the subscription itself. A delete propagated to both zones precisely because both were dependents of the same deleted object. Redundancy and backup are not substitutes — redundancy shares fate with its parent; backup does not.\n\nRecovery succeeded because UniSuper held data outside that shared fate. Google confirms: 'Data backups that were stored in Google Cloud Storage in the same region were not impacted by the deletion, and, along with third party backup software, were instrumental in aiding the rapid restoration.' The Register adds a further out-of-band layer: 'UniSuper had backups at another cloud provider,' which prevented complete data loss. Note the subtlety — the GCS backups survived even though co-located in the same region, because they were a different service object, not a dependent of the deleted Private Cloud. That is the definition of out-of-band: independent blast radius, not merely independent geography.\n\nDetection was the weakest link. There was no automated alarm, no anomaly guard, and no human confirmation on a destructive lifecycle action. The first signal was the outage itself in early May 2024 — detection at the point of total loss. The ~one-year gap between misconfiguration and detonation meant the defect sat latent past any provisioning-review window, with no telemetry surfacing an impending destructive expiry. UniSuper's online services were then inaccessible for 'nine days' (iTnews).\n\nBlast radius was tightly contained to a single tenant. Per the RCA the event did not impact any other Google Cloud customer, service, or UniSuper's other GCVE Private Clouds/Account/Orgs/Folders/Projects; UniSuper ran 'more than one private cloud' and only one was destroyed (iTnews). Google's remediation attacked the mechanism, not just the instance: it deprecated the internal tool ('This aspect is now fully automated and controlled by customers via the user interface'), 'corrected the system behavior that sets GCVE Private Clouds for deletion,' and ran a fleet-wide audit — 'We scrubbed the system database and manually reviewed all GCVE Private Clouds to ensure that no other GCVE deployments are at risk' — concluding 'It is not a systemic issue.' Uncertainty disclosed: the public record accessible this session does not give exact deletion timestamps, the precise UniSuper member count or funds-under-management, byte-level recovered volumes, or per-member downtime, so those are described qualitatively or omitted rather than fabricated.
References & provenance
- official-postmortem Details of the Google Cloud GCVE incident (technical blog)“One input parameter was left blank when using an internal tool to provision the customer's Private Cloud. As a result of the blank parameter, the system assigned a then unknown default fixed 1 year term value for this parameter.”https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident
- official-postmortem Google Cloud blog — recovery and backups (official postmortem)“Data backups that were stored in Google Cloud Storage in the same region were not impacted by the deletion, and, along with third party backup software, were instrumental in aiding the rapid restoration.”https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident
- press iTnews — Google Cloud accidentally deletes UniSuper's online account“After the end of the system-assigned one year period, the customer's GCVE private cloud was deleted. ... UniSuper's online services were inaccessible for nine days.”https://www.itnews.com.au/news/google-cloud-accidentally-deletes-unisupers-online-account-608286
- press UniSuper's Google Cloud outage caused by 'unprecedented' misconfiguration (The Register)“When the deletion of UniSuper's Private Cloud subscription occurred, it caused deletion across both of these geographies.”https://www.theregister.com/2024/05/09/unisuper_google_cloud_outage_caused/
- news UniSuper's Google Cloud deletion traced to blank parameter in setup“UniSuper's online services being inaccessible for nine days and having to be rebuilt from backups.”https://www.itnews.com.au/news/unisupers-google-cloud-deletion-traced-to-blank-parameter-in-setup-608286
- official-statement A joint statement from UniSuper and Google Cloud (8 May 2024)“UniSuper currently invests over $135 billion on behalf of more than 647,000 members ... When the deletion of UniSuper's Private Cloud subscription occurred, it caused deletion across both of these geographies.”https://www.unisuper.com.au/about-us/media-centre/2024/a-joint-statement-from-unisuper-and-google-cloud
- news Google Cloud accidentally deleted UniSuper's Private Cloud Subscription“The deletion removed UniSuper's Private Cloud across both of its geographic locations.”https://www.datacenterdynamics.com/en/news/google-cloud-accidentally-deleted-unisupers-private-cloud-subscription/
- analysis Google Cloud & UniSuper outage timeline (community analysis)“From May 2nd to 13th, UniSuper experienced a major outage.”https://danielcompton.net/google-cloud-unisuper
Sourced from public post-incident reports. Quotes are short attributed excerpts for provenance only; the analysis above is original and substantially shorter than its sources. Last verified 2026-08-01.