← All incidents
Incident dossier · Rank #14
AWS Kinesis OS Thread-Limit Cascade — US-EAST-1 (Nov 25, 2020)
Amazon Web Services
2020-11-25
17h 8m core impact
Software
A small addition of capacity to the Kinesis Data Streams front-end fleet in US-EAST-1 pushed every server past an operating-system maximum thread-count limit. Because each front-end server creates OS threads for every other server in the fleet, the added hosts made the per-server thread count breach the OS ceiling; cache construction failed and servers were left with useless shard-maps, unable to route requests. Kinesis errored and slowed, and because many AWS services depend on Kinesis, CloudWatch, Cognito, Lambda, EventBridge and customer workloads cascaded down. Recovery was throttled by slow membership propagation and restart contention, so full Kinesis restoration took roughly 17 hours.
Failure cascade
Trigger → primary fault → downstream blast radius, derived from the sourced root cause and affected-services record.
Facility & location
- Operator
- Amazon Web Services
- Data center
- AWS US-EAST-1 (Kinesis Data Streams front-end fleet)
- Location
- Ashburn, USA, us-east-1
- Date
- 2020-11-25
Impact & scale
- Users affected
- Not publicly quantified by AWS; broad impact across many AWS customers in US-EAST-1 whose applications depend on Kinesis and dependent services (CloudWatch, Cognito, Lambda, EventBridge/CloudWatch Events, and downstream ECS/EKS provisioning and AWS Auto Scaling).
- Financial
- Not published by AWS for this event. (The ~$34M Parametrix estimate belongs to the separate Dec 7, 2021 US-EAST-1 outage and does not apply here.)
- Scope
- Major regional multi-service outage
Services / systems down
- Amazon Kinesis Data Streams
- Amazon CloudWatch (metrics and alarms)
- Amazon Cognito (User Pools)
- AWS Lambda (invocation error rates)
- Amazon EventBridge / CloudWatch Events
- Amazon ECS / EKS (provisioning and scaling delays, via EventBridge)
- AWS Auto Scaling (metric-dependent policy delays)
Impact data & metrics
| Capacity-addition (trigger) window | 63 min (02:44–03:47 AM PST) |
| Detection lag (breach to first alarm) | ~1h28m (03:47 to 05:15 AM PST) |
| Time to confirmed root cause | ~5h52m after breach (03:47 to 09:39 AM PST) |
| Constrained restart rate | a few hundred servers per hour |
| Recovery span (restart start to full recovery) | 12h16m (10:07 AM to 10:23 PM PST) |
| Total Kinesis outage duration | ~18h36m from breach (03:47 AM to 10:23 PM PST) |
| Thread scaling relationship | O(N): threads per server proportional to fleet size |
| Downstream services impacted | 6+ (Cognito, CloudWatch, Lambda, EventBridge, ECS, EKS) |
| Cognito recovery time | 02:18 PM PST |
| CloudWatch full recovery | 10:31 PM PST |
Magnitude profile
Magnitude 8.2 = blast 9×0.35 + users 8×0.25 + financial 7×0.20 + duration 8×0.20 (sub-scores 0–10 · weighted composite)
High blast radius: a single control-plane data service (Kinesis) took down at least five dependent AWS services plus customer workloads in the busiest AWS region. Duration ~17h to full Kinesis recovery. User and financial magnitudes are inferred as large but are NOT quantified by AWS for this event; scores reflect scope of dependent services, not a published cost/user figure.
Sequence of events (SOE)
- TRIGGER A 'relatively small addition of capacity' begins to be added to the Kinesis front-end fleet — the trigger, explicitly not the root cause.
- TRIGGER Capacity addition finishes; the larger fleet pushes every server past the OS max-thread limit — 'the new capacity had caused all of the servers in the fleet to exceed the maximum number of threads allowed by an operating system configuration.'
- CASCADE Failure mechanism engages fleet-wide: 'cache construction was failing to complete and front-end servers were ending up with useless shard-maps,' disabling authentication, throttling and request-routing.
- DETECTION Automated monitoring raises the first alarms for elevated Kinesis error rates — the sensor detects the symptom, not the underlying thread-limit breach, roughly 90 minutes after the breach. CloudWatch Events/EventBridge errors also begin at 5:15 AM.
- MITIGATION Engineers narrow the root cause to candidate failures that would require a full front-end fleet restart.
- DETECTION Root cause confirmed: the new capacity caused all servers to exceed the OS-configured maximum thread count — the 'confirmation of the fire's origin.'
- RECOVERY Controlled, throttled restart of the front-end fleet begins; first servers take traffic again. Restart deliberately slow because too-fast recovery would overwhelm shard-map/metadata retrieval and re-trigger failure.
- MITIGATION Cognito remediation deployment begins to reduce reliance on the backlogged Kinesis Data Stream buffers that were blocking Cognito webservers.
- MITIGATION Engineers act to mitigate Lambda memory contention — metric buffering that could not drain to Kinesis had 'caused memory contention on the underlying service hosts used for Lambda function invocations' — resolving the increased error rates.
- IMPACT CloudWatch shows 'increased error rates and latencies for the PutMetricData and PutLogEvents APIs'; ECS and EKS, which use EventBridge for internal workflows, see delays provisioning new clusters and scaling existing ones.
- RECOVERY 'By 2:18 PM PST Cognito was operating normally' after authentication failures clear.
- RECOVERY Majority of EventBridge, ECS and EKS event-processing and cluster-provisioning issues resolved.
- MITIGATION De-energisation/isolation analogue: the added capacity is removed and the fleet restarted under the existing OS limit; AWS deliberately defers raising the limit 'without further testing,' adding only 'a few hundred per hour.'
- RESTORED Amazon Kinesis fully recovered — end of the ~18.5-hour 'burn' from the 3:47 AM breach.
- RESTORED CloudWatch metrics and alarms (PutMetricData / PutLogEvents) fully recovered.
- RECOVERY Structural remediation committed: greatly accelerate cellularization of the front-end fleet 'to match what we've done with the back-end,' move to 'larger CPU and memory servers, reducing the total number of servers,' add thread-consumption alarming, radically improve cold-start time, and raise OS thread limits after testing.
Root cause
The specific "ignition source" was not a physical component but an operating-system configuration limit: the maximum number of threads permitted per process on every server in the Amazon Kinesis front-end fleet in US-EAST-1. AWS states plainly that "the new capacity had caused all of the servers in the fleet to exceed the maximum number of threads allowed by an operating system configuration" (aws.amazon.com/message/11201). The precise failure mechanism is architectural: each front-end server maintains a cache called a "shard-map" ("a cache of information, including membership details and shard ownership for the back-end clusters"), and it populates that cache by opening OS threads to its peers — "each front-end server creates operating system threads for each of the other servers in the front-end fleet." Because "the total threads each server must maintain is directly proportional to the number of servers in the fleet," the design is O(N) thread-per-peer. When capacity was added, N grew, and every server's required thread count crossed the OS ceiling. Cache construction could no longer complete, so "cache construction was failing to complete and front-end servers were ending up with useless shard-maps." A front-end server with a useless shard-map cannot perform its core job — it "handles authentication, throttling, and request-routing to the correct stream-shards on the back-end clusters" — so the entire front end lost the ability to route records.
The trigger and the root cause are explicitly separated by AWS: the event was triggered by "a relatively small addition of capacity that began to be added to the service at 2:44 AM PST, finishing at 3:47 AM PST." The capacity add was routine and small; it merely exposed the latent defect. The latent root cause is the design decision to scale coordination threads linearly with fleet size, combined with an OS thread limit that had never been dimensioned against the fleet's growth trajectory, and — critically — the absence of any instrumentation that would have caught the fleet approaching that ceiling. AWS concedes this maintenance/observability gap in its remediation, committing to add "fine-grained alarming for thread consumption": such alarming did not exist beforehand, so operators had no early-warning telemetry on the single resource that failed. There was also no failure-isolation barrier — the front-end fleet was not cellularized, so a limit breach hit "all of the servers in the fleet" at once rather than being contained to a cell.
In physical-forensic terms mapped to this software incident: there was no combustion, no maintenance-worn part, and no aged equipment with a make/model/chemistry. The equivalent "latent defect left uninspected" is the un-alarmed, un-load-tested proximity of thread consumption to the OS limit as the fleet scaled — a capacity/observability lapse rather than a mechanical wear lapse.
Contributing factors
- Architectural design fault: coordination threads scale O(N) with fleet size — AWS confirms 'each front-end server creates operating system threads for each of the other servers in the front-end fleet' and 'the total threads each server must maintain is directly proportional to the number of servers in the fleet' — so routine horizontal scaling raised every server's thread count toward the OS ceiling simultaneously (aws.amazon.com/message/11201).
- Observability/maintenance gap: no fine-grained alarming existed for thread consumption; AWS commits to add 'fine-grained alarming for thread consumption' only as a post-incident action, meaning operators had no early warning as the fleet approached the OS thread limit (aws.amazon.com/message/11201).
- No failure-isolation barrier: the front-end fleet was not cellularized, so the limit breach hit 'all of the servers in the fleet' at once instead of being confined to a cell — accelerating cellularization is named as the medium-term fix, confirming it was absent (aws.amazon.com/message/11201).
- Detection sensed the symptom, not the cause: alarms fired at 5:15 AM for elevated Kinesis error rates — roughly 90 minutes after the 3:47 AM breach — and the thread-limit root cause was not confirmed until 9:39 AM, a slow path from alarm to diagnosis (aws.amazon.com/message/11201).
- Fragile, expensive recovery path: because shard-map construction is metadata-heavy at fleet scale, servers could only be restored 'at the rate of a few hundred per hour,' and the OS limit could not be safely raised mid-incident 'without further testing,' extending the outage (aws.amazon.com/message/11201).
- Unbounded downstream buffers: dependent services (Cognito, Lambda, CloudWatch) buffered data toward Kinesis with no back-pressure ceiling, so Cognito webservers 'begin to block on the backlogged Kinesis Data Stream buffers' and Lambda metric buffering 'caused memory contention on the underlying service hosts used for Lambda function invocations' — turning one service's outage into a multi-service cascade (aws.amazon.com/message/11201).
Correction of errors (COE)
- Greatly accelerate cellularization of the front-end fleet 'to match what we've done with the back-end' so a limit breach is contained to a single cell rather than the whole fleet.
- Move the front-end fleet to 'larger CPU and memory servers, reducing the total number of servers,' which cuts fleet size N and the per-server thread count directly.
- Add 'fine-grained alarming for thread consumption' so operators get early warning before the OS thread ceiling is approached.
- Raise the operating-system thread-count limit after further load testing, removing the specific ceiling that failed once validated.
- Radically improve the cold-start time for the front-end fleet so recovery is not constrained to 'a few hundred per hour.'
- Reduce Cognito, CloudWatch, and Lambda coupling to Kinesis by adding back-pressure/graceful degradation so a Kinesis backlog no longer blocks webservers or exhausts host memory.
Lessons learnt
- Resource limits that scale with fleet size are latent scaling traps: a thread-per-peer model makes per-host thread usage grow with the whole fleet, so an ordinary, small capacity add can silently push every host over an OS ceiling at once.
- Alarm on the failing resource, not just its symptoms: with no thread-consumption alarming, the first signal was downstream put/get errors ~1.5 hours after the breach — precious lead time lost to indirect detection.
- Blast radius follows dependencies: a single data-plane service (Kinesis) coupled to CloudWatch, Cognito, Lambda and EventBridge turned one fleet's failure into a multi-service regional outage.
- Design dependents to degrade, not die: latent buffering assumptions (Cognito exhausting buffers on upstream errors; CloudWatch metric buffers growing into memory contention that reached Lambda) converted a Kinesis fault into independent secondary failures.
- Recovery must be planned as carefully as failure: slow membership propagation (up to 1h) and restart contention capped restart at a few hundred servers/hour, so the fix itself took ~12 hours — cold-start speed is a first-class reliability property.
- Status-communication tooling must not share the failure domain it reports on: the primary dashboard depended on the impaired Cognito, so operators needed a rehearsed, independent backup channel.
Improvements & remediation
- Bound the fault domain via cellularization of the front-end fleet so thread growth (and any similar resource pressure) is contained within a cell rather than the entire fleet.
- Reduce per-host peer-thread pressure by consolidating onto larger CPU/memory servers and raising the tested OS thread-count limit, giving headroom against fleet-size-driven thread growth.
- Add fine-grained thread-consumption alarming to detect the actual limiting resource before it manifests as downstream request errors.
- Isolate high-fan-in tenants by moving large services like CloudWatch to a separate, partitioned front-end fleet and moving the front-end cache to a dedicated fleet.
- Harden dependents against upstream failure: Cognito webservers now tolerate Kinesis errors without exhausting buffers, and CloudWatch persists 3 hours of metrics locally so alarms survive a Kinesis outage.
- Make recovery faster and safer by radically improving cold-start time so a throttled, contention-limited restart no longer dominates the outage duration.
- Decouple operational status communications from the services being reported on, and keep support engineers regularly trained on the independent backup tool.
Comprehensive analysis
Trigger vs. Root Cause
AWS is unusually explicit in separating the two. The trigger was 'a relatively small addition of capacity' (02:44–03:47 AM PST) — routine, small, and by itself harmless. The root cause was a latent design defect: each front-end server 'creates operating system threads for each of the other servers in the front-end fleet,' so thread count scales O(N) with fleet size and had crept toward a fixed OS maximum. The capacity add simply pushed every server across that ceiling at once. Treating the capacity change as 'the cause' would miss the real hazard, which had been building silently as the fleet grew.
Failure Mechanism: The Shard-Map Cache
The front-end tier 'handles authentication, throttling, and request-routing to the correct stream-shards on the back-end clusters,' and depends on a local 'shard-map' cache of 'membership details and shard ownership.' Building that cache requires a thread per peer. Once the OS thread limit was exceeded, 'cache construction was failing to complete and front-end servers were ending up with useless shard-maps' — a server that cannot build its shard-map cannot route, so the failure was fleet-wide and synchronized rather than a gradual degradation.
Cascade Through Dependent Services
The blast radius followed data dependencies on Kinesis. Cognito webservers began to 'block on the backlogged Kinesis Data Stream buffers,' failing authentication until 2:18 PM. Lambda metric buffering that could not drain 'caused memory contention on the underlying service hosts used for Lambda function invocations.' CloudWatch degraded on PutMetricData/PutLogEvents; EventBridge errors from 5:15 AM propagated to ECS and EKS, which 'make use of EventBridge to drive internal workflows,' delaying cluster provisioning and scaling. Each was the same anti-pattern: an unbounded buffer backing up behind a stalled dependency with no back-pressure.
Recovery Constraints and Remediation
Recovery was slow by necessity: shard-map rebuild is metadata-heavy, so servers could be added only 'a few hundred per hour,' and AWS declined to raise the OS limit mid-incident 'without further testing,' instead removing the added capacity so thread count fell below the limit. Full Kinesis recovery came at 10:23 PM — about 18.5 hours after the breach. Structural fixes named by AWS attack every layer: cellularization for isolation, larger/fewer servers to cut N, thread-consumption alarming for early warning, faster cold-start for quicker recovery, and a tested higher thread limit.
Technical deep-dive
Kinesis's front-end fleet is a stateless-request tier fronting stateful back-end shard clusters. Its role is small but important: it "handles authentication, throttling, and request-routing to the correct stream-shards on the back-end clusters." To route correctly, each front-end server must know which back-end cluster owns which shard, held locally in a "shard-map" cache built from "membership details and shard ownership for the back-end clusters." The cache is populated via a coordination mechanism in which "each front-end server creates operating system threads for each of the other servers in the front-end fleet," and "it takes up to an hour for any existing front-end fleet member to learn of new participants." AWS confirms "the total threads each server must maintain is directly proportional to the number of servers in the fleet." This linear coupling is the structural fault: horizontal scaling, normally a mitigation, here increases per-server load, so the fleet becomes less stable as it grows.
The failure was therefore synchronized rather than progressive. A "relatively small addition of capacity" (02:44–03:47 PST) raised N enough that every server's thread requirement crossed the fixed OS limit at essentially the same moment. Past the limit, thread creation for cache construction fails, and "cache construction was failing to complete and front-end servers were ending up with useless shard-maps." With no valid shard-map, routing collapses fleet-wide. Recovery could not be a simple rollback because the shard-map build is itself expensive and metadata-intensive at fleet scale: bringing many servers back at once would hammer the metadata/shard-map retrieval path and re-trigger failure. Hence the throttled restart — "we could only add servers at the rate of a few hundred per hour" — and the deliberate choice not to raise the OS limit mid-incident: "We didn't want to increase the operating system limit without further testing," after which, having removed the added capacity, AWS "determined that the thread count would no longer exceed the operating system limit and proceeded with the restart." That conservatism traded recovery speed for safety, producing the multi-hour tail from first restart (10:07 AM) to full recovery (10:23 PM).
The blast radius was defined by which services buffer data through Kinesis. Amazon Cognito used Kinesis Data Streams for analytics, and the backlog caused "Cognito webservers to begin to block on the backlogged Kinesis Data Stream buffers," producing authentication failures until 2:18 PM. CloudWatch "experienced increased error rates and latencies for the PutMetricData and PutLogEvents APIs" (recovered 10:31 PM). Lambda's own metric buffering, unable to drain to Kinesis, grew until "it caused memory contention on the underlying service hosts used for Lambda function invocations, resulting in increased error rates," mitigated at 10:36 AM. CloudWatch Events and EventBridge "experienced increased API errors and delays in event processing starting at 5:15 AM PST"; because "ECS and EKS both make use of EventBridge to drive internal workflows used to manage customer clusters and tasks," they saw delays provisioning new clusters, scaling existing ones, and deprovisioning tasks, mostly resolved by 4:15 PM. Every downstream failure was the same pattern — an unbounded buffer backing up behind a stalled Kinesis — which is why the structural fix AWS names is cellularization "to isolate the effects of failure within a service," plus moving to "larger CPU and memory servers, reducing the total number of servers" to shrink N and thus the thread count directly.
References & provenance
- official-postmortem Summary of the Amazon Kinesis Event in the Northern Virginia (US-EAST-1) Region“the new capacity had caused all of the servers in the fleet to exceed the maximum number of threads allowed by an operating system configuration”https://aws.amazon.com/message/11201
- news AWS explains what went wrong at US-East-1“Reporting on AWS's own account of the November 25, 2020 Kinesis event in US-East-1.”https://www.datacenterdynamics.com/en/news/aws-explains-what-went-wrong-us-east-1/
Sourced from public post-incident reports. Quotes are short attributed excerpts for provenance only; the analysis above is original and substantially shorter than its sources. Last verified 2026-07-31.