Incident dossier · Rank #9

AWS S3 US-EAST-1 Service Disruption

Amazon Web Services 2017-02-28 4h 35m core impact Human errorSoftware

An authorized S3 engineer running an established capacity-removal playbook mistyped one input; the command removed far more servers than intended, taking down the S3 index and placement subsystems in US-EAST-1. Because a very large number of services (and AWS's own status dashboard) depend on S3, the ~4.5-hour disruption cascaded across a large fraction of the internet.

Failure cascade

Trigger → primary fault → downstream blast radius, derived from the sourced root cause and affected-services record.

Facility & location

Operator: Amazon Web Services
Data center: US-EAST-1 region (Northern Virginia)
Location: Ashburn (approx.), United States, US-EAST-1
Date: 2017-02-28

Impact & scale

Users affected: Thousands of websites and services worldwide dependent on S3 US-EAST-1
Financial: ≈ US$150M+ estimated loss to S&P-500 companies (Cyence, via press)
Scope: Hyperscale public-cloud region

Services / systems down

Amazon S3 (GET/LIST/PUT/DELETE)
EC2 instance launches
EBS volumes
AWS Lambda
AWS Service Health Dashboard (couldn't update — hosted on S3)

Impact data & metrics

Trigger to first partial recovery (index serving GET/LIST/DELETE)	~2 h 49 min (09:37 to 12:26 PST)
Trigger to index subsystem full recovery	~3 h 41 min (09:37 to 13:18 PST)
Trigger to full S3 restoration (placement/PUT restored)	~4 h 17 min (09:37 to 13:54 PST)
Duration status could not be updated on the Service Health Dashboard	~2 h (09:37 event start to 11:37 PST); icons 'stuck on green'
S3 subsystems requiring a full restart	2 (index + placement)
Time since the index/placement subsystems were last fully restarted in larger regions	'many years' (exact interval not disclosed)
Named major external sites/services reported down	11+ (Docker Registry Hub, Trello, Travis CI, GitHub, GitLab, Quora, Medium, Signal, Slack, Imgur, Twitch.tv)
AWS regions affected	1 (US-EAST-1, Northern Virginia)
Servers intended vs. actually removed / capacity thresholds	Not published — AWS states only that 'a larger set of servers was removed than intended'
Authoritative dollar-loss figure	Not disclosed in the primary source; only third-party estimates exist (uncertain)

Magnitude profile

Magnitude 8.6 = blast 10×0.35 + users 9×0.25 + financial 8×0.20 + duration 6×0.20 (sub-scores 0–10 · weighted composite)

Blast radius extreme: S3 is a foundational dependency for a large share of the public internet; cascade reached EC2/EBS/Lambda and even AWS's own status page.

Sequence of events (SOE)

2017-02-28 09:37 PSTTRIGGER An authorized S3 team member, using an established playbook, executes a command intended to remove a small number of servers from the S3 subsystem used by the S3 billing process.
2017-02-28 09:37 PSTTRIGGER One input to the command is entered incorrectly; a far larger set of servers is removed than intended.
2017-02-28 ~09:37 PSTDETECTION The inadvertently removed servers are found to have supported two other S3 subsystems — the index subsystem (metadata/location for all objects) and the placement subsystem (new-storage allocation, dependent on index). The fault is internal and effectively immediate; no external alarm.
2017-02-28 ~09:38 PSTIMPACT Index and placement capacity drop below operational minimum; each subsystem now requires a full restart. There was no guardrail to stop removal below the minimum required level.
2017-02-28 ~09:38 PSTIMPACT With the subsystems restarting, S3 in US-EAST-1 is unable to service GET, LIST, PUT and DELETE requests.
2017-02-28 morning PSTCASCADE S3-dependent AWS services degrade: the S3 console, new EC2 instance launches, EBS operations relying on S3 snapshots, and AWS Lambda are impaired.
2017-02-28 morning PSTCASCADE External sites go down, including Docker Registry Hub, Trello, Travis CI, GitHub, GitLab, Quora, Medium, Signal, Slack, Imgur and Twitch.tv.
2017-02-28 morning PSTCASCADE The Service Health Dashboard admin console, itself dependent on S3, cannot update per-service status; status icons are 'stuck on green lights' because the red warning icons were hosted in the downed systems.
2017-02-28 ~09:40 PSTMITIGATION Recovery begins with a full restart of the index subsystem first, because the placement subsystem depends on it functioning.
2017-02-28 ongoing PSTMITIGATION Metadata-integrity safety checks are run during restart and deliberately not shortcut; at S3's grown scale the restart-plus-validation 'took longer than expected.'
2017-02-28 until 11:37 PSTCASCADE AWS remains unable to update individual service status on the SHD; it communicates via the @AWSCloud Twitter feed and SHD banner text until the dashboard can be updated (regained around noon PST).
2017-02-28 12:26 PSTRECOVERY Index subsystem has activated enough capacity to begin servicing S3 GET, LIST and DELETE requests.
2017-02-28 13:18 PSTRECOVERY Index subsystem fully recovered; GET, LIST and DELETE APIs functioning normally.
2017-02-28 13:54 PSTRECOVERY Placement subsystem recovery completes, restoring PUT and new-object storage.
2017-02-28 13:54 PSTRESTORED All S3 APIs operate normally in US-EAST-1; full service restored roughly 4 hours 17 minutes after the mistyped command.

Root cause

There was no physical ignition source in this incident — no equipment make, model, chemistry, or combustion, and therefore no detection panel, suppression system, evacuation, or emergency-services response. The "ignition" was a human operational error. At 9:37AM PST on 2017-02-28, an authorized S3 team member, following an established debugging playbook, executed a capacity-removal command intended to take a small number of servers out of the S3 subsystem used by the S3 billing process. Per AWS: "one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended." That single mistyped input is the proximate cause. The failure mechanism is a cross-subsystem blast radius the operator did not anticipate. The inadvertently removed servers also supported two other S3 subsystems — the index subsystem, which "manages the metadata and location information of all S3 objects in the region" and is necessary to serve GET, LIST, PUT and DELETE requests, and the placement subsystem, which "manages allocation of new storage and requires the index subsystem to be functioning properly to correctly operate." Dropping their capacity below the operational minimum required each subsystem to be fully restarted, and "while these subsystems were being restarted, S3 was unable to service requests." Recovery was ordered by dependency: index first, then placement. The latent root is procedural/tooling rather than physical, and three-fold. First, a GUARDRAIL gap: the capacity-removal tool accepted an over-broad input with no safeguard against taking any subsystem below its minimum required capacity — AWS's own remediation adding exactly that safeguard is the admission it did not previously exist. Second, an UNTESTED-AT-SCALE recovery procedure: AWS had "not completely restarted the index subsystem or the placement subsystem in our larger regions for many years," and after massive growth the restart plus metadata-integrity validation "took longer than expected" — a cold-start path effectively unrehearsed at current scale. Third, a DEPENDENCY lapse in the status-reporting plane: the Service Health Dashboard admin console itself depended on S3 in the same region, so operators could not update per-service status during the very event it was meant to report.

Contributing factors

Tooling guardrail absence (Design): the capacity-removal tool accepted an over-broad input and removed capacity too fast with no safeguard against dropping a subsystem below its minimum required capacity — AWS's remediation adding exactly that safeguard confirms it did not previously exist (aws.amazon.com/message/41926/).
Recovery procedure untested at current scale (Maintenance/inspection analog): AWS states it had 'not completely restarted the index subsystem or the placement subsystem in our larger regions for many years'; after massive growth the restart and metadata validation 'took longer than expected,' indicating the recovery path was not rehearsed at production scale (aws.amazon.com/message/41926/).
Tight, undocumented cross-subsystem coupling: the removed billing-process servers also supported two other S3 subsystems (index and placement), so a command scoped to billing silently starved the request-serving and storage-allocation paths (aws.amazon.com/message/41926/).
Status-reporting plane depended on the failed service: the Service Health Dashboard admin console has a dependency on Amazon S3, so per-service status could not be updated until 11:37AM PST and icons stayed green, delaying accurate customer communication (aws.amazon.com/message/41926/ ; theregister.com/2017/03/01/aws_s3_outage/).
Blast-radius concentration in a single subsystem: the index subsystem was one monolithic unit per region, so its restart stalled all GET/LIST/PUT/DELETE at once; AWS committed to partitioning it into cells specifically to bound this (aws.amazon.com/message/41926/).

Correction of errors (COE)

Modify the capacity-removal tool to remove capacity more slowly and add safeguards preventing removal below any subsystem's minimum required capacity level AWS S3 team · Stated done in the post-mortem (aws.amazon.com/message/41926/)
Partition the index subsystem into smaller cells to reduce blast radius and speed recovery AWS S3 team · Committed / reprioritized to begin immediately (aws.amazon.com/message/41926/)
Re-architect the Service Health Dashboard administration console to run across multiple regions so status can be updated during a single-region S3 impairment AWS · Committed (aws.amazon.com/message/41926/)
Audit and rehearse full restart plus metadata-integrity validation for index and placement subsystems at current production scale AWS S3 team · Implied by the post-mortem; specific completion status not disclosed

Lessons learnt

Destructive operational tooling must enforce hard floors: a single mistyped input should never be able to take a subsystem below its minimum required capacity — guardrails, not operator care, are the control (aws.amazon.com/message/41926/).
Recovery procedures decay silently: a restart path unexercised 'for many years' becomes untrustworthy as scale grows, so cold-start and validation must be periodically rehearsed at production scale (aws.amazon.com/message/41926/).
Never let a monitoring/status plane depend on the system it monitors — the SHD's dependency on S3 blinded operators and customers during the exact window they needed visibility (aws.amazon.com/message/41926/ ; theregister.com/2017/03/01/aws_s3_outage/).
Hidden cross-subsystem coupling turns a scoped command into a systemic outage: servers labeled for 'billing' also underpinned index and placement, so scoping by intent is not scoping by effect (aws.amazon.com/message/41926/).
Blast-radius reduction via partitioning is a design discipline, not an afterthought — a monolithic per-region index subsystem meant one restart froze every S3 request type simultaneously (aws.amazon.com/message/41926/).

Improvements & remediation

DesignModify the capacity-removal tool to remove capacity more slowly and add safeguards that prevent capacity from being taken below any subsystem's minimum required level — AWS: 'We have modified this tool to remove capacity more slowly and added safeguards to prevent capacity from being removed when it will take any subsystem below its minimum required capacity level.' (aws.amazon.com/message/41926/)
DesignPartition the index subsystem into smaller cells to bound blast radius and speed recovery, so a single subsystem's restart no longer stalls all GET/LIST/PUT/DELETE at once; AWS reprioritized this partitioning work to begin immediately (aws.amazon.com/message/41926/).
ProcessRemove the status-reporting plane's dependency on the service it reports on — AWS committed to running Service Health Dashboard administration across multiple regions so status can be updated even when one region's S3 is impaired (aws.amazon.com/message/41926/).
MaintenanceRehearse and validate the full restart-and-metadata-validation path at current production scale, since the procedure had not been fully exercised in larger regions 'for many years' and ran slower than expected after massive growth (aws.amazon.com/message/41926/).
SafetyPreserve mandatory metadata-integrity safety checks during recovery rather than shortcutting them for speed — the incident affirmed that trading validation for a faster restart would risk metadata corruption (aws.amazon.com/message/41926/).

Comprehensive analysis

Nature of the incident: operational error, not a physical fire

This is a cloud-service disruption with no ignition, equipment, chemistry, or combustion — the standard fire-incident lenses (detection panel, suppression, evacuation, de-energisation, emergency-services response) do not apply and are honestly marked N/A. The proximate cause is a single mistyped input to an established capacity-removal playbook at 9:37AM PST on 2017-02-28. AWS states plainly: 'one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.' The fault was self-inflicted and detected internally and immediately, so there is no alarm latency to analyse — the analysis instead centres on why a scoped command produced a regional outage and why recovery took over four hours.

The blast radius: hidden cross-subsystem coupling

The command targeted a low-privilege pool serving the S3 billing process, but the removed servers also supported two request-critical subsystems. The index subsystem 'manages the metadata and location information of all S3 objects in the region' and is required for every GET, LIST, PUT and DELETE; the placement subsystem allocates new storage and 'requires the index subsystem to be functioning properly to correctly operate.' Over-removal pushed both below their operational minimum with no floor to stop it, forcing full restarts during which 'S3 was unable to service requests.' Scoping by operator intent ('billing') was not scoping by effect.

Why recovery was slow: an unrehearsed cold start at scale

AWS had 'not completely restarted the index subsystem or the placement subsystem in our larger regions for many years.' After massive growth, the cold start plus mandatory metadata-integrity validation 'took longer than expected.' AWS made the correct call to NOT shortcut the safety checks, trading recovery speed for metadata integrity. Recovery followed the dependency chain: index began serving GET/LIST/DELETE at 12:26PM, index fully recovered at 1:18PM, and placement restored PUT/new-object storage at 1:54PM — full restoration roughly 4h17m after the trigger.

The visibility failure: monitoring that depended on the monitored

The most instructive secondary failure was reflexive. The Service Health Dashboard admin console itself depended on S3 in the same region, so 'from the beginning of this event until 11:37AM PST, we were unable to update the individual services' status.' The Register adds that the icons 'were stuck on green lights because the red icons warning of failures were hosted in the downed systems.' AWS fell back to the @AWSCloud Twitter feed and dashboard banner text, and regained dashboard control around noon PST. A status plane must never share a failure domain with the service it reports on.

Systemic lessons and AWS's remediation

AWS committed to concrete, sourced fixes: the capacity-removal tool now removes capacity more slowly and enforces a minimum-capacity safeguard (the admission that no such guardrail existed); the monolithic index subsystem is being partitioned into cells to bound blast radius; and SHD administration was moved to run across multiple regions. The durable lessons generalise beyond AWS: destructive tooling needs hard floors rather than operator care, recovery paths decay silently and must be rehearsed at production scale, and blast-radius partitioning is a design discipline. External casualties (GitHub, Slack, Trello, Docker Registry Hub, Twitch and more) underscore how concentrated a dependency US-EAST-1 S3 had become.

Technical deep-dive

S3 in US-EAST-1 is decomposed into subsystems; two are load-bearing for this incident. The index subsystem is the metadata/location store — it "manages the metadata and location information of all S3 objects in the region" and is necessary to serve all GET, LIST, PUT and DELETE requests. The placement subsystem allocates storage for new objects and "requires the index subsystem to be functioning properly to correctly operate" — a hard directional dependency that dictated recovery order. A third, lower-privilege capacity pool served the S3 billing process; that pool was the intended target of the 9:37AM PST playbook command. The over-removal drove both index and placement below their operational minimum, and there was no floor to stop it. Removing a significant portion of the capacity caused each subsystem to require a full restart, and "while these subsystems were being restarted, S3 was unable to service requests." Because the two subsystems had not been completely restarted "in our larger regions for many years," and S3 had grown massively, the cold-start plus metadata-integrity validation ran far longer than a rehearsed procedure would predict. AWS deliberately did NOT shortcut the safety checks — a correct call trading recovery speed for metadata integrity — which is why the restart and validation "took longer than expected." The cascade radiated across everything treating S3 as a storage primitive: the S3 console, new EC2 instance launches, EBS operations relying on S3 snapshots, and AWS Lambda were impaired. Per The Register, external casualties included Docker Registry Hub, Trello, Travis CI, GitHub, GitLab, Quora, Medium, Signal, Slack, Imgur and Twitch.tv. The most instructive containment failure was reflexive: the AWS Service Health Dashboard admin console itself depended on S3, so "from the beginning of this event until 11:37AM PST, we were unable to update the individual services' status," and per The Register the icons "were stuck on green lights because the red icons warning of failures were hosted in the downed systems." AWS fell back to the @AWSCloud Twitter feed and SHD banner text. Recovery then proceeded along the dependency chain: index began servicing GET/LIST/DELETE at 12:26PM PST, index fully recovered at 1:18PM PST, and placement (restoring PUT/new-object storage) completed at 1:54PM PST, at which point S3 operated normally. The record is explicit that this was a self-inflicted fault detected internally and immediately — no alarm latency, no external responders.

References & provenance

official-postmortem Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region“an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers ... one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.”https://aws.amazon.com/message/41926/
press The Register — Amazon S3-izure blamed on innocent typo that snowballed horribly“the icons were stuck on green lights because the red icons warning of failures were hosted in the downed systems”https://www.theregister.com/2017/03/01/aws_s3_outage/
news Amazon's massive AWS outage was caused by a typo (The Verge)“an incorrectly typed command during a routine debugging exercise took down a chunk of S3 servers.”https://www.theverge.com/2017/2/28/14766086/amazon-s3-outage-cause-typo-internet-server
news Amazon cloud outage cost S&P 500 companies an estimated $150M (press, citing Cyence)“S&P 500 companies lost an estimated $150 million due to the outage, according to Cyence.”https://www.reuters.com/article/amazon-com-outage-idUSL2N1GG0D5

Sourced from public post-incident reports. Quotes are short attributed excerpts for provenance only; the analysis above is original and substantially shorter than its sources. Last verified 2026-07-31.