Incident dossier · Rank #17

Google Global Authentication Outage — User ID Service Quota Collapse (Dec 14, 2020)

Google Cloud 2020-12-14 47m core impact Software

On December 14, 2020, an automated quota-management system wrongly cut capacity on the account database behind Google's central User ID Service to near-zero. An October change had registered that service with a new quota system while leaving parts of the old one in place; the residual system reported the service's usage as 0, which triggered the automated reduction. With the quota starved, the Paxos leader could no longer write, so credential issuance and account-metadata lookups failed for every Google account. Any request needing OAuth-based authentication returned errors, taking down Gmail, YouTube, Drive, Docs, Calendar, Play, Google Cloud, Workspace and even Google Home speakers globally. Engineers identified the root cause about 22 minutes after onset and mitigated by disabling quota enforcement datacenter by datacenter; the authentication core recovered in 47 minutes (03:46–04:33 US/Pacific).

Failure cascade

Trigger → primary fault → downstream blast radius, derived from the sourced root cause and affected-services record.

Facility & location

Operator: Google Cloud
Data center: Google global identity (User ID Service)
Location: USA, global
Date: 2020-12-14

Impact & scale

Users affected: Billions of Google account holders worldwide (no absolute count disclosed in any reachable source)
Financial: Not disclosed — no dollar/financial-impact figure exists in any reachable source; only qualitative SLA-credit exposure noted by community
Scope: SEV-1 / global (single-point-of-failure in shared identity layer)

Services / systems down

Gmail
YouTube
Google Drive
Google Docs
Google Calendar
Google Play
Google Meet
Google Cloud Console
Cloud Storage
BigQuery
Google Kubernetes Engine
Google Home smart speakers
all Google-account-authenticated services

Impact data & metrics

Total customer-facing outage duration	47 minutes
Detection lead over user impact	~3 min (alerts 03:43 vs errors 03:46)
Detection-to-root-cause time	~25 min (03:43 → 04:08)
Root-cause-to-first-mitigation time	~14 min (04:08 → 04:22)
Global mitigation-to-recovery time	~6 min (04:27 → 04:33)
Latent dormancy of misconfiguration	~6-10 weeks (October change → 14 Dec impact)
Reported User ID Service usage (false signal)	0
BigQuery streaming error rate	~75% (jobs ~10%)
Google Cloud Storage requests impacted	~15% (OAuth/HMAC/email auth)
GKE control plane API failures	~4% of requests
On-call paging time from detection	~5 min (03:43 → 03:48)

Magnitude profile

Magnitude 8.0 = blast 10×0.35 + users 10×0.25 + financial 6×0.20 + duration 4×0.20 (sub-scores 0–10 · weighted composite)

Blast radius maximal: a shared identity single-point-of-failure took out every Google-account-authenticated surface worldwide, including consumer IoT hardware. Users score maximal (whole global Google account base). Duration modest (47-min auth core; ~50–57 min user-perceived) thanks to fast root-cause identification. Financial score is a mid estimate reflecting massive reach but NO disclosed dollar figure in any reachable source.

Sequence of events (SOE)

October 2020TRIGGER During the migration of the User ID Service to a new quota system, a change registers the service with the new system but leaves parts of the previous quota system in place, causing usage to be 'incorrectly reported... as 0.' No fire/thermal event — this is the software 'ignition' (root-cause trigger).
October–December 2020 (~6-10 weeks)CASCADE The zero-usage misconfiguration sits dormant and undetected; monitoring at the latent stage does not flag it, so the defect persists until the quota grace period expires.
2020-12-14, pre-03:43 US/PacificTRIGGER The grace period on enforcing quota restrictions expires, 'triggering automated quota systems to decrease the quota allowed for the User ID service'; 'the quota for the account database was reduced, which prevented the Paxos leader from writing.' This is the propagation vector — the 'suppression' safeguard becoming the fault path.
03:43 US/PacificDETECTION Automated alerts for capacity detect the issue at 2020-12-14 03:43 — roughly 3 minutes BEFORE user-facing errors begin. This is the 'fire-detection' analogue at the acute stage.
03:46IMPACT User ID Service errors begin; global authentication lookups (OAuth token/cookie validation) start failing.
03:46CASCADE With the Paxos leader unable to write, 'the majority of read operations became outdated,' and stale reads on the account database propagate authentication-lookup errors across all regions.
~03:46 onwardIMPACT BigQuery affected: 'streaming requests returned ~75% errors, while BigQuery jobs returned ~10% errors on average globally.'
~03:46 onwardIMPACT Google Cloud Storage affected: 'approximately 15% of requests... were impacted... specifically those using OAuth, HMAC or email authentication.'
~03:46 onwardIMPACT GKE affected: '~4% of requests to the GKE control plane API failed.' Additionally 'all authenticated Google Workspace apps were down for the duration of the incident' (Gmail, Calendar, Docs, Drive, Meet).
03:48DETECTION On-call engineers are paged (incident-response 'emergency dispatch' analogue — there is no physical emergency-services response).
04:08DETECTION 'The root cause and a potential fix were identified' (~25 minutes after first automated alert; ~20 minutes after paging).
04:22MITIGATION Engineers begin 'disabling the quota enforcement in one datacenter' as the first mitigation — the effective 'manual suppression' after automated safeguards failed.
04:27MITIGATION 'The same mitigation was applied to all datacenters' — quota enforcement disabled globally.
04:33RECOVERY Error rates return to normal levels as the Paxos leader resumes writing and reads become current.
04:33 (end of window)RESTORED Customer-facing services fully restored; total user-facing outage duration 47 minutes ('for a duration of 47 minutes, customer-facing Google services that required Google OAuth access were unavailable').
Post-incidentRECOVERY Five remediation items published: review quota automation to prevent fast global changes; improve monitoring for incorrect configurations; improve external-comms tooling; add write-failure resilience to the User ID database; and more strictly limit data-plane impact during User ID Service failures.

Root cause

There was no physical ignition source, no equipment, and no thermal event — this was a global software authentication outage of Google's User ID Service, which "maintains a unique identifier for every account and handles authentication credentials for OAuth tokens and cookies" (Incident #20013). The forensic categories below are therefore mapped to their software analogues; the physical fire-forensics fields (make/model/age/chemistry of ignition, suppression agent, evacuation, emergency-services response, electrical de-energisation) have NO analogue and are non-applicable by nature of the fault, not merely undocumented. The technical "ignition" (root-cause trigger) was a quota-mis-accounting misconfiguration introduced by an incomplete migration. "As part of an ongoing migration of the User ID Service to a new quota system... a change was made in October to register the User ID Service with the new quota system, but parts of the previous quota system were left in place which incorrectly reported the usage for the User ID Service as 0" (Incident #20013). This is the latent, dormant defect: residual old-system state that produced a false zero-usage signal. It sat latent from the October change until 14 December impact (roughly six to ten weeks) with no inspection or monitoring regime that surfaced the zero-usage report in the interval. The failure mechanism proceeded from that dormant condition. The only buffer between the bad change and the outage was a grace period: "An existing grace period on enforcing quota restrictions delayed the impact, which eventually expired, triggering automated quota systems to decrease the quota allowed for the User ID service" — this masked rather than prevented the fault. When the grace period expired, "the quota for the account database was reduced, which prevented the Paxos leader from writing." With writes blocked to the Paxos-replicated account database, replicated reads went stale: "the majority of read operations became outdated which resulted in errors on authentication lookups." Because the User ID Service is a single global authentication dependency, OAuth token and cookie validation failed across all regions simultaneously. The latent root is three-layered: (1) a change-management/migration defect that left residual old-quota-system state and produced a false zero-usage signal; (2) a quota-enforcement automation designed to act globally on that signal, with only a grace period as a safeguard (and that safeguard merely delayed, never prevented); and (3) a resilience-design gap in which a write failure in the identity database was allowed to degrade read correctness and cascade into the data plane of dependent GCP services. Google's own remediation set names all three: "Review our quota management automation to prevent fast implementation of global changes," "Improve monitoring and alerting to catch incorrect configurations sooner," and "Evaluate and implement improved write failure resilience into our User ID service database."

Contributing factors

Incomplete migration / change-management defect: the October registration of the User ID Service with the new quota system left 'parts of the previous quota system... in place which incorrectly reported the usage for the User ID Service as 0' — residual state that was never detected or cleaned up (Incident #20013).
Monitoring/inspection gap at the latent stage: automated alerting did not flag the incorrect zero-usage configuration for the roughly 6-10 weeks it sat dormant; Google's own remediation is to 'Improve monitoring and alerting to catch incorrect configurations sooner' (Incident #20013).
Quota-enforcement automation permitted a fast, global change with only a grace period as a buffer: 'An existing grace period on enforcing quota restrictions delayed the impact, which eventually expired' — it masked rather than prevented, and remediation calls to 'Review our quota management automation to prevent fast implementation of global changes' (Incident #20013).
Single-point global dependency with no regional isolation: because the User ID Service is a single global authentication service, once 'the quota for the account database was reduced, which prevented the Paxos leader from writing' the fault propagated worldwide with no containment (Incident #20013).
Write-failure fragility in the identity database design: a blocked Paxos leader caused 'the majority of read operations [to become] outdated which resulted in errors on authentication lookups' — a write failure degraded read correctness, prompting remediation to 'implement improved write failure resilience into our User ID service database' (Incident #20013).
Weak data-plane isolation: dependent GCP services (BigQuery, GCS, GKE) had their data planes impacted by an identity-plane failure, prompting remediation to 'Improve resilience of GCP Services to more strictly limit the impact to the data plane' (Incident #20013).

Correction of errors (COE)

Review quota management automation to prevent fast implementation of global changes Google Cloud · Committed (post-incident remediation, published in Incident #20013)
Improve monitoring and alerting to catch incorrect configurations sooner Google Cloud · Committed (post-incident remediation, published in Incident #20013)
Evaluate and implement improved write failure resilience into the User ID service database Google Cloud · Committed (post-incident remediation, published in Incident #20013)
Improve resilience of GCP Services to more strictly limit the impact to the data plane during User ID Service failures Google Cloud · Committed (post-incident remediation, published in Incident #20013)
Improve reliability of tools and procedures for posting external communications during outages Google Cloud · Committed (post-incident remediation, published in Incident #20013)

Lessons learnt

Migrations must leave no residual dual-registration: an incomplete cutover left 'parts of the previous quota system... in place' that emitted a false usage=0, proving that partial migration state is itself a latent hazard requiring explicit teardown verification (Incident #20013).
Automation that acts globally needs an implausibility fail-safe, not just a delay: the grace period only postponed the fault; a live global identity service reporting 0 usage should have been rejected as implausible before any quota reduction (Incident #20013).
A write failure in a consensus-replicated identity store must not degrade read correctness: blocking the Paxos leader made 'the majority of read operations... outdated,' so write-path failures need graceful read fallbacks to avoid authentication-wide impact (Incident #20013).
Single global dependencies concentrate blast radius: routing every account's OAuth/cookie validation through one User ID Service meant one quota misconfiguration took down BigQuery, GCS, GKE and all authenticated Workspace apps at once — argues for regional isolation and blast-radius limits (Incident #20013).
Detection at the acute stage is not detection at the latent stage: alerts fired ~3 minutes before user impact, yet the same monitoring never caught the misconfiguration for weeks — latent-config validation is a distinct capability from runtime anomaly alerting (Incident #20013).
Outage communications tooling must survive the outage: Google committed to improving external-comms tools/procedures, since an identity-plane failure can also cripple the very consoles used to post status (Incident #20013).

Improvements & remediation

Design'Evaluate and implement improved write failure resilience into our User ID service database' so a blocked Paxos leader cannot cascade into stale reads and global authentication-lookup failures (Incident #20013).
Design'Improve resilience of GCP Services to more strictly limit the impact to the data plane,' isolating dependent services (BigQuery, GCS, GKE) from an identity-plane fault (Incident #20013).
Process'Review our quota management automation to prevent fast implementation of global changes,' so a single false usage signal cannot trigger an immediate global quota reduction (Incident #20013).
Maintenance'Improve monitoring and alerting to catch incorrect configurations sooner,' closing the ~6-10-week latent-detection gap in which the zero-usage misconfiguration sat unflagged (Incident #20013).
Safetyharden the quota-enforcement grace period into an active fail-safe that blocks or alarms on implausible signals (e.g. usage=0 for a live global service) rather than merely delaying enforcement, since the existing grace period only masked the fault (Incident #20013).
Process'Improve reliability of tools and procedures for posting external communications during outages,' so customer-facing status can be maintained when the identity plane itself is failing (Incident #20013).

Comprehensive analysis

Incident classification and forensic-analogue mapping

This is a global software authentication outage, not a physical data-center incident. There was no ignition source, equipment failure, thermal event, suppression agent, evacuation, or emergency-services response — those fire-forensics fields have no analogue and are non-applicable by the nature of the fault. Per Incident #20013, the failure was in Google's User ID Service, which 'maintains a unique identifier for every account and handles authentication credentials for OAuth tokens and cookies.' The 'ignition' maps to the root-cause configuration trigger; 'detection' to automated capacity alerting; 'suppression' to automated quota-enforcement safeguards; and 'manual suppression' to engineers disabling quota enforcement.

Root cause: latent migration defect and false zero-usage signal

An in-progress migration to a new quota system introduced the latent defect. In October a change 'register[ed] the User ID Service with the new quota system, but parts of the previous quota system were left in place which incorrectly reported the usage for the User ID Service as 0.' This residual state sat dormant for roughly 6-10 weeks with no monitoring that surfaced the implausible zero-usage reading — a maintenance/inspection gap Google explicitly committed to close ('Improve monitoring and alerting to catch incorrect configurations sooner').

Cascade mechanism: grace period, Paxos write block, stale reads

The only buffer was a grace period, which 'delayed the impact' but 'eventually expired, triggering automated quota systems to decrease the quota allowed for the User ID service.' Then 'the quota for the account database was reduced, which prevented the Paxos leader from writing. Shortly after, the majority of read operations became outdated which resulted in errors on authentication lookups.' The safeguard was the propagation vector: the grace period masked the fault, and enforcement then executed the harmful global quota reduction. A write-path failure was allowed to degrade read correctness across the consensus-replicated store.

Blast radius and single-dependency concentration

Because every account's OAuth/cookie validation flows through one global service, impact was worldwide and simultaneous: BigQuery 'streaming requests returned ~75% errors' (jobs ~10%), '~15% of requests to Google Cloud Storage' failed (OAuth/HMAC/email auth), '~4% of requests to the GKE control plane API failed,' and 'all authenticated Google Workspace apps were down for the duration of the incident' (Gmail, Calendar, Docs, Drive, Meet). The absence of regional isolation is why a single quota misconfiguration became a global identity outage.

Response, mitigation, and total duration

Automated alerts fired at 03:43 (~3 min before user errors at 03:46); engineers were paged at 03:48; 'the root cause and a potential fix were identified' at 04:08. Mitigation was staged: 'disabling the quota enforcement in one datacenter' at 04:22, then 'the same mitigation was applied to all datacenters' at 04:27, with error rates returning to normal at 04:33. Total customer-facing duration: 'for a duration of 47 minutes, customer-facing Google services that required Google OAuth access were unavailable.'

Remediation, lessons, and disclosed gaps

Google published five remediation items spanning design (write-failure resilience, data-plane isolation), process (quota automation guardrails, external-comms tooling), and maintenance (config monitoring). Genuinely unknown items remain disclosed: the exact October date, change-ID and author; why residual old-quota parts were left in place; the precise grace-period length; the exact reduced quota value and thresholds; and internal Paxos replica counts and staleness bounds. Incident #20013 is the sole primary source; press coverage additionally reported YouTube impact, but that is not attributed to #20013 and was not independently re-verified here.

Technical deep-dive

Chain of causation, per Incident #20013 (the sole primary source): An October 2020 configuration change registered the User ID Service with a new quota system as part of an in-progress migration, but "parts of the previous quota system were left in place," and this residual dual-registration "incorrectly reported the usage for the User ID Service as 0." That false signal was inert only because "an existing grace period on enforcing quota restrictions delayed the impact." When the grace period expired on 14 December, it triggered "automated quota systems to decrease the quota allowed for the User ID service"; at the reduced quota "the quota for the account database was reduced, which prevented the Paxos leader from writing." Paxos is a distributed-consensus protocol: a leader that cannot commit writes leaves replicas unable to advance, so "the majority of read operations became outdated," and stale reads on the account database "resulted in errors on authentication lookups." Because every Google account authentication flows through this one global service, OAuth token/cookie validation failed worldwide. Blast radius (all sourced to #20013): on Google Cloud Platform, BigQuery "streaming requests returned ~75% errors, while BigQuery jobs returned ~10% errors on average globally"; "approximately 15% of requests to Google Cloud Storage (GCS) were impacted... specifically those using OAuth, HMAC or email authentication"; and "~4% of requests to the GKE control plane API failed." Additionally "all authenticated Google Workspace apps were down for the duration of the incident" (Gmail, Calendar, Docs, Drive, Meet named in the report). Total customer-facing duration was 47 minutes: "for a duration of 47 minutes, customer-facing Google services that required Google OAuth access were unavailable." Forensic-category mapping to software analogues: DETECTION (fire-detection analogue) = Google's automated capacity/quota monitoring, which fired "at 2020-12-14 03:43," roughly 3 minutes BEFORE user-facing errors began at 03:46 — good acute detection, but detection failed at the latent stage: monitoring did NOT catch the incorrect configuration when introduced in October, hence remediation "Improve monitoring and alerting to catch incorrect configurations sooner." SUPPRESSION (fire-suppression analogue) = the automated quota-enforcement safeguards plus the grace period; these not only failed to suppress the fault, they were the propagation vector — the grace period masked the zero-usage misconfiguration, then enforcement itself reduced the quota and blocked the Paxos leader. Effective "manual suppression" was human intervention: engineers began "disabling the quota enforcement in one datacenter" at 04:22, "the same mitigation was applied to all datacenters" at 04:27, and error rates normalised at 04:33. CONTAINMENT/SPREAD = none; the fault spread globally because the identity service is a single global dependency with no regional isolation. EVACUATION / EMERGENCY-SERVICES = no analogue beyond the paging chain (auto-alert 03:43 -> engineers paged 03:48 -> root cause and potential fix identified 04:08 -> mitigation 04:22-04:33). Data-plane isolation is addressed by remediation "Improve resilience of GCP Services to more strictly limit the impact to the data plane." Genuinely unknown / silent in the record (disclosed): the exact October date and change-ID and who made the change; why the residual old-quota-system parts were left in place (tooling gap vs oversight vs rollback bug); the precise grace-period length and its rationale; the exact quota value the account database was reduced TO and the thresholds that triggered automated reduction; internal Paxos replica counts and staleness bounds; and whether any data was lost or only reads went stale. Google #20013 is the only primary source consulted; secondary press coverage (which additionally reported YouTube impact) restates the event but was not independently re-verified within this session and is not attributed to #20013.

References & provenance

official-postmortem Google Cloud Status — Incident #20013 (official postmortem)“A change was made in October to register the User ID Service with the new quota system, but parts of the previous quota system were left in place which incorrectly reported the usage for the User ID Service as 0.”https://status.cloud.google.com/incident/zall/20013
encyclopedia 2020 Google services outages“An accidental reduction of capacity on their central user ID management system, causing requests that required OAuth-based authentication to fail.”https://en.wikipedia.org/wiki/2020_Google_services_outages
news Google Issues Post Mortem on Gmail, YouTube Outage (SecurityWeek)“Google published a post mortem attributing the December 14 outage to its central identity system running out of storage quota.”https://www.securityweek.com/google-issues-post-mortem-gmail-youtube-outage/
forum Post Mortem of Google Outage on 14 December 2020 (Hacker News)“Community discussion observed the disruption lasting on the order of 57 minutes and debated how quota errors mistranslated into client-facing 5xx responses.”https://news.ycombinator.com/item?id=25472455

Sourced from public post-incident reports. Quotes are short attributed excerpts for provenance only; the analysis above is original and substantially shorter than its sources. Last verified 2026-07-31.