Incident dossier · Rank #4

Meta Global BGP + DNS Outage — Backbone Self-Disconnection Takes Facebook, Instagram & WhatsApp Offline Worldwide

Meta (Facebook) 2021-10-04 7h 11m core impact NetworkHuman error

On 4 October 2021 a single command issued during routine backbone maintenance — intended only to assess global backbone capacity — unintentionally took down every connection in Facebook's backbone, completely severing all of its data centers from the internet. With the backbone gone, Facebook's DNS-serving locations followed their designed safety behavior and declared themselves unhealthy, withdrawing the BGP route advertisements for the authoritative nameservers. That withdrawal pulled Facebook's DNS off the global internet: recursive resolvers everywhere lost any path to the nameservers, so Facebook, Messenger, Instagram, WhatsApp, Mapillary and Oculus all failed simultaneously rather than degrading gracefully. Recovery was slow because primary and out-of-band network access were also down, forcing engineers to travel on-site; the intentionally hardened physical and system security of the facilities made restarting systems difficult, with recovery advancing after a team reached and reset servers at the Santa Clara, California data center. The outage ran roughly six to seven hours (15:39 UTC start to general restoration around 22:50 UTC).

Failure cascade

Trigger → primary fault → downstream blast radius, derived from the sourced root cause and affected-services record.

Facility & location

Operator: Meta (Facebook)
Data center: Facebook Global Backbone / Authoritative DNS (Santa Clara data center recovery site)
Location: Menlo Park, USA, California
Date: 2021-10-04

Impact & scale

Users affected: All users of Facebook, Messenger, Instagram, WhatsApp, Mapillary and Oculus globally (family of apps served the same withdrawn authoritative DNS); precise user count not verified in primary sources
Financial: Not verified in primary Meta or Cloudflare sources; widely-cited revenue/market-cap figures remain unsourced and are excluded here
Scope: SEV-1 / Global total outage (characterized by CNBC as Facebook's worst outage since 2008)

Services / systems down

Facebook
Messenger
Instagram
WhatsApp
Mapillary
Oculus
Internal engineering tooling
Physical badge / building access

Impact data & metrics

Outage start	15:39 UTC, 4 Oct 2021
BGP routing restored	~21:50 UTC
DNS services resumed	22:05 UTC
General platform restoration	~22:50 UTC
Total duration	~6-7 hours (431 min, 15:39->22:50 UTC)
External DNS-availability window (Cloudflare vantage)	~5.5 hours (~15:50 to ~21:20 UTC)
Meta properties globally unavailable	6 (Facebook, Messenger, Instagram, WhatsApp, Mapillary, Oculus)
Backbone connectivity lost	100% — complete data-center-to-internet disconnection
Historical scale	Worst Facebook outage since 2008 (per CNBC, via Wikipedia)
Published numbered COE action items	0 (no enumerated corrective list published)

Magnitude profile

Magnitude 9.0 = blast 10×0.35 + users 10×0.25 + financial 8×0.20 + duration 7×0.20 (sub-scores 0–10 · weighted composite)

Blast radius and user impact maxed: every Meta property globally offline at once plus loss of internal tooling and badge access. Duration ~6-7h is severe but bounded. Financial score is a qualitative estimate — CNBC called it the worst outage since 2008 — because specific dollar figures are not verified in the primary sources.

Sequence of events (SOE)

2021-10-04 ~15:40 UTCTRIGGER During a routine backbone maintenance job a capacity-assessment command is issued that unintentionally withdraws all backbone connections between Meta data centers; Cloudflare simultaneously sees a peak of BGP routing changes leaving Facebook's network — 'That's when the trouble began.'
2021-10-04 ~15:40 UTCCASCADE The change-audit tool that should have blocked the command fails due to a bug: 'a bug in that audit tool prevented it from properly stopping the command.'
2021-10-04 ~15:40 UTCCASCADE The entire backbone is removed from operation, effectively disconnecting Meta's data centers globally.
2021-10-04 ~15:40 UTCCASCADE By health-check design, DNS servers cannot reach the data centers, declare themselves unhealthy, and withdraw their BGP advertisements — a self-protective mechanism firing on a global false-positive.
2021-10-04 ~15:40 UTCDETECTION External detection is near-immediate: Cloudflare observes the peak of routing changes as backbone connections begin dropping.
2021-10-04 ~15:50 UTCIMPACT facebook.com ceases to resolve on public resolvers as authoritative nameserver prefixes become unreachable — Cloudflare's 1.1.1.1 records it 'stopped being available at around 15:50 UTC.'
2021-10-04 15:58 UTCCASCADE Facebook stops announcing routes to its DNS prefixes via BGP — 'Facebook had stopped announcing the routes to their DNS prefixes,' effectively disconnecting itself from the Internet.
2021-10-04 ~15:58 UTCCASCADE Public resolvers 1.1.1.1, 8.8.8.8 and other major resolvers begin 'issuing (and caching) SERVFAIL responses,' propagating the failure across the DNS ecosystem.
2021-10-04 (during outage)IMPACT A global retry storm forms as clients worldwide re-query relentlessly: 'DNS resolvers worldwide handling 30x more queries than usual' — collateral load on third-party infrastructure.
2021-10-04 (during outage)DETECTION Internal detection is crippled: 'The total loss of DNS broke many of the internal tools we'd normally use to investigate and resolve outages like this.'
2021-10-04 (during outage)MITIGATION Remote data-center access is impossible because the networks are down; secure physical-access protocols must be activated — 'it took extra time to activate the secure access protocols needed to get people onsite.'
2021-10-04 (during outage)MITIGATION Engineers are physically dispatched into hardened data centers to debug and restart the systems, slowed by hardware 'designed to be difficult to modify even when you have physical access to them.'
2021-10-04 ~21:00 UTCRECOVERY Cloudflare sees renewed BGP activity from Facebook's network (peaking 21:17 UTC) as backbone connectivity is restored on-site.
2021-10-04 ~21:00 UTCRECOVERY Power-up is staged cautiously because data centers had power dips 'in the range of tens of megawatts' and a sudden reversal 'could put everything from electrical systems to caches at risk.'
2021-10-04 ~21:20 UTCRESTORED Facebook DNS returns to availability on Cloudflare's 1.1.1.1 ('returned at 21:20 UTC'); services progressively recover as backbone connectivity is restored, with full reconnection noted by 21:28 UTC.
2021-10-05RESTORED Meta (Santosh Janardhan) publishes the detailed post-incident writeup: capacity-assessment command + audit-tool bug + DNS self-withdrawal + tooling/physical-access difficulties, following the initial 10-04 statement.

Root cause

There was no ignition source, combustion, or equipment fire — this was a pure software-and-procedure failure, so the conventional root-cause chain (make/model/age/chemistry of a failed component) is inapplicable by the nature of the incident, and the record is explicit that router/DNS-server hardware makes, models and firmware were never causal. Meta states the event was "caused not by malicious activity, but an error of our own making" (engineering.fb.com/2021/10/05). Proximate trigger — a single erroneous maintenance command. During a routine backbone maintenance job, "a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Meta's data centers globally" (engineering.fb.com/2021/10/05). Latent root — a defective safety guardrail. The change-validation tooling that exists specifically to catch this class of error was itself broken: "Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool prevented it from properly stopping the command" (engineering.fb.com/2021/10/05). The failure is twofold: an erroneous command reached production, and the automated guardrail meant to stop it had failed. Design flaw that converted a backbone fault into total global disconnection — an automated self-protective mechanism firing on a correlated, global false-positive. "Our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection. In the recent outage the entire backbone was removed from operation, making these locations declare themselves unhealthy and withdraw those BGP advertisements" (engineering.fb.com/2021/10/05). The health-check worked exactly as designed, but because the backbone loss made every DNS location simultaneously "unhealthy," authoritative DNS withdrew itself from the Internet worldwide — the safety mechanism amplified rather than contained the failure. Recovery-compounding latent conditions — the loss of DNS also disabled Meta's own diagnostic tooling ("The total loss of DNS broke many of the internal tools we'd normally use to investigate and resolve outages like this"), and physical/system security hardening slowed on-site manual recovery ("once you're inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them"; "it took extra time to activate the secure access protocols needed to get people onsite") (engineering.fb.com/2021/10/05).

Contributing factors

Maintenance/change-control lapse: the erroneous capacity-assessment command was permitted to run against the live global backbone during a routine maintenance job — Meta: 'a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network' (engineering.fb.com/2021/10/05).
Defective safety guardrail: the automated audit tool meant to catch exactly this class of command was itself buggy and did not stop it — Meta: 'a bug in that audit tool prevented it from properly stopping the command' (engineering.fb.com/2021/10/05).
Fragile fail-safe design coupling backbone health to DNS reachability: DNS servers auto-withdraw BGP advertisements when they cannot reach data centers, so a whole-backbone loss made every location declare itself unhealthy simultaneously — 'making these locations declare themselves unhealthy and withdraw those BGP advertisements' (engineering.fb.com/2021/10/05).
Single-point dependency of internal tooling on the same DNS that failed: 'The total loss of DNS broke many of the internal tools we'd normally use to investigate and resolve outages like this' (engineering.fb.com/2021/10/05), crippling diagnosis and slowing response.
Physical/system security hardening impeded emergency recovery: remote access was dead and 'the hardware and routers are designed to be difficult to modify even when you have physical access to them... it took extra time to activate the secure access protocols needed to get people onsite' (engineering.fb.com/2021/10/05).
Recovery risk from power swings: individual data centers were reporting 'dips in power usage in the range of tens of megawatts,' and 'suddenly reversing such a dip in power consumption could put everything from electrical systems to caches at risk' (engineering.fb.com/2021/10/05), forcing a slow staged restart.

Correction of errors (COE)

Fix the change-audit tool bug and add regression tests that block whole-backbone / global-scope withdrawal commands before any maintenance job runs. Meta Network Engineering · Recommended (Meta committed generally to strengthen testing/drills; specific fix not publicly itemized)
Add a global-scope guard / minimum-anycast-floor to the DNS BGP health-check so authoritative DNS cannot fully self-withdraw during a correlated backbone loss. Meta DNS / Traffic Engineering · Recommended (derived from root cause; not publicly confirmed)
Establish an out-of-band control plane so internal diagnostic, access and comms tooling survive a total production-DNS loss. Meta Production Engineering · Recommended (derived from root cause; not publicly confirmed)
Pre-stage and rehearse break-glass physical-access procedures for hardened data centers to cut the secure-access activation delay from the recovery path. Meta Data Center Operations / Security · Recommended (derived from root cause; not publicly confirmed)
Codify and drill a staged power-up runbook to safely reverse tens-of-megawatts power dips during mass recovery. Meta Data Center Operations · Recommended (derived from root cause; not publicly confirmed)
Strengthen overall testing, drills and resilience for backbone maintenance. Meta Engineering · Committed by Meta (post-incident statement)

Lessons learnt

A safety mechanism (health-based BGP withdrawal) is only as safe as its assumptions: when the 'unhealthy' signal is the loss of the entire backbone, the designed self-protection amplifies a local fault into a global one.
Shared authoritative DNS and a shared backbone across a family of apps remove all fault isolation — a single control-plane event can take down every product at once.
Control-plane dependencies must not gate their own recovery: when the same backbone carries both production traffic and the operators' management/out-of-band access, an outage locks operators out of the fix.
Physical security hardening and emergency-recovery speed are in tension; recovery runbooks must plan for the case where the only path in is a manually secured, deliberately hard-to-enter facility.
Maintenance commands with facility-wide or backbone-wide reach need guardrails and pre-execution auditing proportional to their blast radius, not to their intended (narrow) purpose.
External vantage points (e.g. Cloudflare's BGP/DNS observations) independently corroborated the failure chain, showing the value of third-party visibility when internal telemetry is itself offline.

Improvements & remediation

Process (change control): Harden change control so no single command can act on the entire global backbone at once — require staged, blast-radius-limited execution with a mandatory dry-run for capacity-assessment commands; Meta committed to strengthen its testing, drills and overall resilience after the event (engineering.fb.com/2021/10/05).
MaintenanceFix and independently test the change-audit tool whose bug 'prevented it from properly stopping the command' (engineering.fb.com/2021/10/05), and add regression tests that assert the guardrail blocks whole-backbone-withdrawal commands before any maintenance job can run.
DesignRedesign the DNS-to-backbone health-check coupling so a total backbone loss cannot make every DNS location simultaneously declare itself unhealthy and withdraw all BGP advertisements globally (engineering.fb.com/2021/10/05) — add a global-scope guard / minimum-anycast-floor so authoritative DNS never fully self-disconnects.
Design (out-of-band control): Decouple internal diagnostic, access and communication tooling from the production DNS it depends on, because 'the total loss of DNS broke many of the internal tools we'd normally use to investigate and resolve outages' (engineering.fb.com/2021/10/05).
Safety (break-glass access): Pre-stage and rehearse emergency physical-access procedures so engineers can reach hardened data centers quickly when remote access is dead — removing the 'extra time to activate the secure access protocols' from the critical recovery path (engineering.fb.com/2021/10/05).
Process (staged power-up): Codify a cautious staged power-up runbook to manage tens-of-megawatts power swings that 'could put everything from electrical systems to caches at risk' (engineering.fb.com/2021/10/05).

Comprehensive analysis

Incident Overview

On 4 October 2021, Facebook, Instagram, WhatsApp, Messenger and Oculus went offline worldwide for roughly six hours. The cause was not a fire, hardware failure, or attack but a procedural error compounded by a broken guardrail. During routine backbone maintenance, a command intended only to assess global backbone capacity instead withdrew every backbone connection between Meta's data centers, and the audit tool designed to reject such a command had a bug that let it through. Meta characterizes the event as 'caused not by malicious activity, but an error of our own making' (engineering.fb.com/2021/10/05).

The Three-Layer Cascade: Backbone to BGP to DNS

The blast radius grew because three layers were tightly coupled. Losing the backbone (layer 1) severed every DNS point-of-presence from the data centers. Meta's DNS servers are designed to withdraw their BGP advertisements (layer 2) when they cannot reach the data centers, on the assumption that unreachability means a locally unhealthy site. Because the whole backbone vanished at once, every DNS location judged itself unhealthy simultaneously and withdrew in unison — removing facebook.com's authoritative nameservers (layer 3) from the global routing table. Cloudflare externally clocked the sequence: routing-change peak ~15:40 UTC, DNS unavailable ~15:50 UTC, DNS-prefix withdrawal 15:58 UTC (blog.cloudflare.com/october-2021-facebook-outage).

Why the Safety Mechanisms Amplified the Failure

Two protective systems turned a recoverable fault into a total, self-inflicted disconnection. First, the change-audit tool — a safety interlock meant to stop dangerous commands — was itself defective, so the erroneous command executed unchecked. Second, the DNS health-check acted precisely as designed, but its scope was unbounded: a mechanism built to route around one bad site instead removed authoritative DNS from the entire Internet when the failure was global. When DNS fell, public resolvers (1.1.1.1, 8.8.8.8 and others) began issuing and caching SERVFAIL, and worldwide retries drove resolver load to about 30x normal — spreading collateral impact onto infrastructure Meta does not own.

Recovery Constraints and the Physical-Access Bottleneck

Recovery was slow for reasons that were themselves latent design conditions. The remote management path relied on the same network that was down, and the DNS loss disabled many of Meta's internal investigation tools. Engineers therefore had to be physically dispatched into data centers hardened against intruders, where activating secure access protocols took extra time and where hardware is deliberately 'difficult to modify even when you have physical access.' Restoration was then intentionally staged, because data centers showed power dips of tens of megawatts and a sudden reversal 'could put everything from electrical systems to caches at risk.' Cloudflare observed renewed BGP from ~21:00 UTC, DNS back by ~21:20 UTC, and full reconnection by 21:28 UTC.

Systemic Lessons for Correlated-Failure Design

The incident is a canonical study in correlated failure and self-amplifying safety systems. Guardrails must be tested as rigorously as the systems they protect; health-driven automation must be bounded so a global fault cannot trigger universal self-withdrawal; diagnostic and recovery tooling must not depend on the very service that can fail; and physical security must be balanced against a rehearsed break-glass recovery path. Above all, resilience planning must account for blast radius beyond one's own estate — here a single command rippled out to third-party resolvers across the entire Internet.

Technical deep-dive

The failure propagated across three coupled layers: backbone (physical/IP connectivity between data centers), BGP (Internet route announcements), and DNS (authoritative name resolution). During a routine backbone maintenance job, a capacity-assessment command withdrew all backbone connections at once (engineering.fb.com/2021/10/05: "unintentionally took down all the connections in our backbone network"). The audit tool that should have blocked the command failed due to a bug ("a bug in that audit tool prevented it from properly stopping the command"). With the backbone gone, every Meta DNS point of presence lost its path to Meta's data centers. By design, a Meta DNS server withdraws its BGP route advertisements when it cannot reach the data centers, treating unreachability as a signal of an unhealthy connection. Normally this reroutes traffic around one bad location; here every location was simultaneously cut off, so all of them withdrew their BGP advertisements at once. Cloudflare observed this externally: at ~15:40 UTC "a peak of routing changes from Facebook. That's when the trouble began," and at 15:58 UTC "Facebook had stopped announcing the routes to their DNS prefixes." With those withdrawals, Facebook had effectively disconnected itself from the Internet. Because facebook.com's authoritative nameservers were now unreachable at the routing layer, no resolver on Earth could complete a lookup; Cloudflare's 1.1.1.1 recorded facebook.com resolution as unavailable "at around 15:50 UTC." Public recursive resolvers began issuing and caching failures: "1.1.1.1, 8.8.8.8, and other major public DNS resolvers started issuing (and caching) SERVFAIL responses." Every app, browser and retrying client worldwide then hammered resolvers on a loop, producing a retry storm — Cloudflare reported "DNS resolvers worldwide handling 30x more queries than usual," a collateral load spike on infrastructure Meta does not own. Recovery could not use the normal remote path because the network carrying it was down, and DNS loss had also disabled Meta's internal diagnostic and access tooling. Engineers were physically dispatched into hardened data centers, where "it took extra time to activate the secure access protocols needed to get people onsite and able to work on the servers," further slowed by hardware "designed to be difficult to modify even when you have physical access." Restoration was deliberately staged: data centers were reporting "dips in power usage in the range of tens of megawatts," and "suddenly reversing such a dip in power consumption could put everything from electrical systems to caches at risk," so the backbone was brought back in controlled steps. Cloudflare saw renewed BGP activity from ~21:00 UTC (peaking 21:17 UTC), DNS availability return by ~21:20 UTC, and full reconnection by 21:28 UTC.

References & provenance

official-postmortem Update about the October 4th outage — Engineering at Meta“Our services are now back online and we're actively working to fully return them to regular operations”https://engineering.fb.com/2021/10/04/networking-traffic/outage/
official-postmortem More details about the October 4 outage — Engineering at Meta“a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network”https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/
external-analysis Understanding How Facebook Disappeared from the Internet — The Cloudflare Blog“Due to Facebook stopping announcing their DNS prefix routes through BGP, our and everyone else's DNS resolvers had no way to connect to their nameservers”https://blog.cloudflare.com/october-2021-facebook-outage/
secondary 2021 Facebook outage — Wikipedia“Border Gateway Protocol (BGP) routing was restored for the affected prefixes at about 21:50”https://en.wikipedia.org/wiki/2021_Facebook_outage

Sourced from public post-incident reports. Quotes are short attributed excerpts for provenance only; the analysis above is original and substantially shorter than its sources. Last verified 2026-07-31.