Incident dossier · Rank #16

Cloudflare Global Outage: Oversized Bot Management Feature File (November 18, 2025)

Cloudflare 2025-11-18 5h 46m core impact Software

On 18 November 2025 an 11:05 UTC ClickHouse database permissions change exposed r0-database table metadata alongside the default database. A Bot Management feature-file generator ran a metadata query over system.columns with no database filter, which then returned duplicated column rows and more than doubled the feature file's size past a hardcoded 200-feature preallocation limit (normal usage ~60 features). When the oversized file propagated network-wide, Cloudflare's FL2 proxy module panicked on a Rust unwrap of an Err value, causing core traffic to return 5xx errors. Customer impact began at 11:28 UTC; responders first suspected a DDoS/Workers KV problem before identifying the config file, deployed a correct file globally at 14:30 UTC, and fully restored all operations by 17:06 UTC.

Failure cascade

Trigger → primary fault → downstream blast radius, derived from the sourced root cause and affected-services record.

Facility & location

Operator: Cloudflare
Data center: Cloudflare global edge network (FL2 core proxy)
Location: Global
Date: 2025-11-18

Impact & scale

Users affected: Global; a large share of internet users behind Cloudflare. Cloudflare published a 5xx-volume graph but no numeric affected-user or affected-account count. Secondary aggregators (not Cloudflare-confirmed) reported 2.1M+ Downdetector issue reports (435,000+ from the US) and estimated ~20% of webpages and ~1/3 of the top-10,000 sites affected at peak.
Financial: Not disclosed by Cloudflare (no revenue-loss or SLA-credit figure published). Third-party revenue-loss estimates ($180-360M) remain unverified secondary screening figures.
Scope: Tier 1 - global, multi-service critical outage

Services / systems down

Cloudflare core CDN/proxy HTTP traffic (5xx errors)
Workers KV
Cloudflare Access
Cloudflare Dashboard / control-plane login (via Turnstile dependency)
Downstream customer platforms: X, ChatGPT, Spotify, Canva, League of Legends (seed); secondary coverage adds Zoom, Coinbase, retailers, Downdetector, Claude AI

Impact data & metrics

Normal feature count in Bot Management file	~60 features
Hardcoded preallocation limit exceeded	200 features
File size change from unfiltered query	More than doubled (duplicate column rows)
Triggering permissions change deployed	11:05 UTC
Customer impact start	11:28 UTC (chart onset ~11:20)
Main impact resolved (correct config deployed)	14:30 UTC
All operations fully restored	17:06 UTC
End-to-end duration	~5h46m (11:20-17:06 UTC)
Main customer-impact window	~3h (11:28-14:30 UTC)
Time from impact to detection	~3 min (11:28 -> 11:31)
Downdetector issue reports (SECONDARY, unconfirmed)	2.1M+ total; 435,000+ from US
Estimated web scope at peak (SECONDARY, unconfirmed)	~20% of webpages; ~1/3 of top-10,000 sites

Magnitude profile

Magnitude 8.0 = blast 9×0.35 + users 9×0.25 + financial 7×0.20 + duration 6×0.20 (sub-scores 0–10 · weighted composite)

Blast radius and user impact scored high: a global edge-network failure took down core HTTP traffic plus Workers KV, Access and the dashboard, cascading into many top-tier consumer platforms. Duration moderate (~3h main impact, ~5h46m end-to-end). Financial score is inferential - Cloudflare disclosed no dollar figure; secondary estimates only. Scores reflect Cloudflare-confirmed scope where available and are dampened where only secondary aggregator data exists.

Sequence of events (SOE)

2025-11-18 11:05 UTCTRIGGER 'Ignition': a ClickHouse database access-control/permissions change is deployed (making implicit table access explicit), causing the Bot Management metadata query to also read the r0 database and return duplicate columns.
2025-11-18 11:05 UTCCASCADE The unfiltered query 'SELECT name, type FROM system.columns WHERE table = http_requests_features' now returns duplicate r0 columns, 'more than doubling the rows in the response'; the generated Bot Management feature file 'in turn, doubled in size.'
2025-11-18 11:20 UTCIMPACT Outage begins: Cloudflare's network starts experiencing significant failures delivering core network traffic.
2025-11-18 11:28 UTCCASCADE The oversized (>200-feature) file reaches customer-serving environments; the FL2 Rust proxy hits an unhandled unwrap() and panics ('thread fl2_worker_thread panicked'), killing worker threads and returning HTTP 5xx; legacy FL proxy instead emits a bot score of zero for all traffic.
2025-11-18 11:31 UTCDETECTION First automated test detects the issue — roughly 11 minutes after impact began.
2025-11-18 11:32 UTCDETECTION Manual investigation started by on-call engineers.
2025-11-18 11:35 UTCMITIGATION Internal incident call created (emergency-response analog) — about 15 minutes after impact.
2025-11-18 ~11:35 UTCMITIGATION Misdiagnosis: team 'initially wrongly suspected the symptoms we were seeing were caused by a hyper-scale DDoS attack'; a coincidental failure of the off-infrastructure status page reinforces the false attacker hypothesis, delaying correct diagnosis.
2025-11-18 (ongoing)CASCADE System flaps: as the node-by-node permissions rollout produces mixed good/bad files, 'every five minutes there was a chance of either a good or a bad set of configuration files being generated' and the entire system would recover and then fail again.
2025-11-18 ~11:30 UTCIMPACT Collateral blast radius: Turnstile fails to load (blocking dashboard logins), Workers KV returns significantly elevated 5xx, Access sees widespread authentication failures, and Email Security temporarily loses an IP reputation source.
2025-11-18 13:04-13:05 UTCMITIGATION Selective 'evacuation'/suppression: internal bypasses for Workers KV and Cloudflare Access route them to a prior version of the core proxy, reducing impact.
2025-11-18 13:10 UTCRECOVERY Workers KV and Access largely restored via the bypass; Turnstile recovers in this window before a second dip.
2025-11-18 13:37 UTCMITIGATION Work focuses on rollback of the Bot Management configuration file to a last-known-good version.
2025-11-18 ~14:40-15:30 UTCIMPACT Turnstile experiences a second outage window (~14:40 to ~15:30) before final stabilisation.
2025-11-18 14:24 UTCMITIGATION Decisive suppression/isolation: engineers stop the creation and propagation of the bad feature file and insert a known-good file into the feature file distribution queue, neutralising the poisoned config source.
2025-11-18 ~14:30 UTCRECOVERY After a successful recovery test and global deployment of the correct config, core traffic is largely flowing as normal.
2025-11-18 17:06 UTCRESTORED All systems at Cloudflare functioning as normal; full cleanup complete. CEO Matthew Prince publishes the public post-mortem the same day. No data lost; not a breach.

Root cause

The proximate "ignition source" was a routine access-control change deployed to Cloudflare's ClickHouse database cluster at 11:05 UTC on 18 November 2025. The change made previously implicit table permissions explicit, granting the querying user visibility into the underlying "r0" database in addition to the "default" database it had always seen. There was no cyber attack: per CEO Matthew Prince, "The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind." Instead it was triggered by a change to a database system's permissions. There is no physical equipment, make, model, or chemistry here; the implicated components are the ClickHouse database, the Bot Management feature-file generator, and the FL2 (Rust/Oxy) core proxy module. The exact failure mechanism was an unfiltered metadata query in the Bot Management feature-file generation pipeline: "SELECT name, type FROM system.columns WHERE table = 'http_requests_features' order by name;". The query lacked a database-name predicate. Once the 11:05 permissions change exposed the r0 database, the same query returned duplicate columns for the underlying tables stored in r0, "more than doubling the rows in the response." The generated Bot Management "feature file, in turn, doubled in size." The latent (design) root cause was a hardcoded runtime cap combined with a preallocated-memory design and an unhandled error path. "The Bot Management system has a limit on the number of machine learning features that can be used at runtime. Currently that limit is set to 200, well above our current use of ~60 features," and "for performance reasons we preallocate memory for the features." When the bad file with more than 200 features propagated to servers, the limit was hit. In the newer FL2 proxy, an unguarded Rust unwrap() on an error value panicked outright — "thread fl2_worker_thread panicked: called Result::unwrap() on an Err value" — killing worker threads and returning HTTP 5xx network-wide. On the legacy FL proxy the same file did not crash but silently corrupted output: "bot scores were not generated correctly, resulting in all traffic receiving a bot score of zero." The "maintenance / procedure" root was a change-validation and input-hardening gap. A Cloudflare-internally-generated configuration artifact was trusted and consumed without a size or row sanity check, and the permissions change was deployed without change-impact analysis of its downstream effect on the metadata query. Cloudflare's own remediation concedes this: "Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input." The gradual, node-by-node rollout of the permissions change created a mixed-fleet state that turned a hard failure into an intermittent one, masking the cause and prolonging the outage.

Contributing factors

Change-impact-analysis gap: the 11:05 UTC ClickHouse permissions change (making implicit table access explicit) was deployed without analysing that it would expose the r0 database to the Bot Management metadata query, doubling the returned rows. Source: official post-mortem, https://blog.cloudflare.com/18-november-2025-outage/
Missing input validation on internally-generated config: the feature file was consumed without a size/row sanity check. Cloudflare's remediation admits it must begin 'Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input.'
Unhandled error path in the FL2 core proxy: an unguarded Result::unwrap() panicked instead of degrading gracefully when the 200-feature cap was exceeded ('thread fl2_worker_thread panicked: called Result::unwrap() on an Err value'), turning a bad config into network-wide 5xx.
Gradual mixed-fleet rollout: the permissions change propagated node-by-node, so 'every five minutes there was a chance of either a good or a bad set of configuration files being generated,' producing recover-then-fail flapping that masked root cause and prolonged the incident.
Insufficient feature-level kill switches: there was no effective way to instantly disable Bot Management scoring network-wide; a listed remediation is 'Enabling more global kill switches for features.'
Brittle capacity assumption: a hardcoded 200-feature limit with preallocated memory ('for performance reasons we preallocate memory for the features') left no headroom and converted a doubled feature count directly into a fatal condition.
Error-reporting resource exhaustion: core dumps / error reports from the panicking process consumed system resources, compounding impact — remediation: 'Eliminating the ability for core dumps or other error reports to overwhelm system resources.'
Diagnostic misdirection: a coincidental status-page failure plus flapping symptoms led the team to a wrong hyper-scale-DDoS hypothesis, delaying correct diagnosis.

Correction of errors (COE)

Harden ingestion of Cloudflare-generated configuration files in the same way user-generated input is validated (explicit size/row bounds checks before load). Cloudflare Engineering · Committed (post-mortem remediation)
Enable more global, feature-level kill switches so features such as Bot Management scoring can be disabled instantly network-wide. Cloudflare Engineering · Committed (post-mortem remediation)
Eliminate the ability for core dumps or other error reports to overwhelm system resources during a panic storm. Cloudflare Engineering · Committed (post-mortem remediation)
Review failure modes for error conditions across all core proxy modules so oversized/malformed inputs degrade gracefully instead of panicking. Cloudflare Engineering · Committed (post-mortem remediation)
Add change-impact analysis and staged validation for ClickHouse permission changes so downstream metadata-query effects are caught pre-deployment. Cloudflare Database/Platform · Implied corrective action

Lessons learnt

Internally-generated configuration is an untrusted input: a Cloudflare-authored feature file, refreshed every few minutes and pushed globally, propagated a fatal fault as fast as any external attack could -- validation must apply to first-party artifacts too.
A benign, well-intentioned permissions change can have catastrophic emergent effects downstream; the blast radius of a database access-control change was not bounded by change review.
Intermittency is worse than a clean failure for diagnosis: node-by-node rollout produced 5-minute good/bad flapping that masked root cause and drove a wrong hyper-scale-DDoS hypothesis.
Unhandled error paths turn recoverable conditions into outages: an unguarded Rust unwrap() in FL2 converted 'file too big' into network-wide 5xx, while legacy FL's silent zero-score failure shows the opposite (dangerous) extreme of failing without erroring.
Missing kill switches cost hours: without an effective feature-level global disable, suppression required manually stopping the generation pipeline and injecting a known-good file -- the only durable stop was neutralising the source.
Correlated failures mislead responders: a status page that coincidentally went down at the same time pushed the team toward an attacker theory, showing how unrelated symptoms can anchor an incorrect diagnosis.

Improvements & remediation

SafetyEnable global, feature-level kill switches so Bot Management scoring can be disabled instantly network-wide -- Cloudflare remediation: 'Enabling more global kill switches for features.'
DesignTreat internally-generated config as untrusted -- 'Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input,' with explicit size/row bounds checks before load.
DesignReplace fatal error paths in core proxy modules with graceful degradation -- 'Reviewing failure modes for error conditions across all core proxy modules' so an oversized file cannot panic FL2 via an unguarded unwrap().
MaintenanceAdd change-impact analysis and staged validation for database permission changes so downstream query effects (e.g., r0 exposure doubling metadata rows) are caught before fleet-wide deployment.
ProcessContain error-reporting resource exhaustion -- 'Eliminating the ability for core dumps or other error reports to overwhelm system resources' during a panic storm.
ProcessHarden the external status page against correlated failure and improve diagnosis playbooks so flapping symptoms are not reflexively attributed to a DDoS attack.
DesignQuery the metadata table with an explicit database-name filter (and de-duplicate columns) so feature-file generation is deterministic regardless of the querying user's granted database visibility.

Comprehensive analysis

Incident Overview

On 18 November 2025 Cloudflare suffered its worst outage since 2019. Beginning at 11:20 UTC, the network returned widespread HTTP 5xx errors for core traffic and did not fully recover until 17:06 UTC. The trigger was not an attack but a routine ClickHouse database permissions change at 11:05 UTC. CEO Matthew Prince published a same-day public post-mortem stating 'The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind.' No data was lost and there was no breach.

Root Cause and Failure Mechanism

The 11:05 permissions change made implicit table access explicit, exposing the underlying r0 database to an unfiltered feature-generation query ('SELECT name, type FROM system.columns WHERE table = http_requests_features'). The query returned duplicate columns, 'more than doubling the rows,' and the Bot Management feature file 'in turn, doubled in size.' It exceeded a hardcoded 200-feature runtime cap (against ~60 in use) backed by preallocated memory. FL2 (Rust) then panicked on an unhandled unwrap() -- 'thread fl2_worker_thread panicked: called Result::unwrap() on an Err value' -- returning 5xx, while legacy FL silently scored all traffic as bot-zero.

Why It Was Hard to Diagnose

The permissions change rolled out node-by-node, and the feature file republishes every few minutes, so 'every five minutes there was a chance of either a good or a bad set of configuration files being generated.' The system recovered then failed repeatedly. This flapping, compounded by a coincidental failure of the off-infrastructure status page, led the team to 'wrongly suspect the symptoms we were seeing were caused by a hyper-scale DDoS attack,' delaying correct diagnosis and prolonging the outage.

Blast Radius Across Services

Beyond core CDN/proxy 5xx, dependent products failed: Turnstile could not load (blocking dashboard logins, with a second dip ~14:40-15:30), Workers KV returned significantly elevated 5xx, Access saw widespread authentication failures (existing sessions unaffected), and Email Security temporarily lost an IP reputation source and some Auto Move actions. Bypasses for Workers KV and Access at 13:04-13:05 UTC routed them to a prior core-proxy version, restoring them around 13:10.

Remediation and Systemic Lessons

The decisive fix at 14:24 UTC was to stop generation/propagation of the bad file and inject a known-good file. Cloudflare's committed remediations treat the deeper systemic gaps: hardening ingestion of internal config as if untrusted, enabling more global kill switches, eliminating core-dump resource exhaustion, and reviewing failure modes across all core proxy modules. The core lesson: first-party configuration is an untrusted input, and unhandled error paths convert bad data into total outages.

Technical deep-dive

The failure chain: a ClickHouse permissions change (11:05 UTC) exposed the r0 database to an unfiltered "SELECT name, type FROM system.columns WHERE table = 'http_requests_features'" query, which then returned duplicate column rows, "more than doubling the rows in the response." The Bot Management "feature file, in turn, doubled in size" and exceeded the hardcoded 200-feature runtime cap (against ~60 features actually in use). In FL2, the preallocated-memory path hit an unhandled Result::unwrap() and panicked ("thread fl2_worker_thread panicked"), returning HTTP 5xx across the network. The most operationally damaging property was intermittency. The permissions change propagated node-by-node across the ClickHouse cluster, and the feature file is refreshed every few minutes and published network-wide. Therefore "every five minutes there was a chance of either a good or a bad set of configuration files being generated," so the entire system would recover and then fail again. This flapping drove an initial misdiagnosis: "we initially wrongly suspected the symptoms we were seeing were caused by a hyper-scale DDoS attack." A coincidental failure of Cloudflare's external (off-infrastructure) status page reinforced the false hypothesis, leading some of the team to believe an attacker might be targeting both the systems and the status page. Two proxy generations failed differently — a critical detail. FL2 (Rust/Oxy) crashed loudly with 5xx; legacy FL degraded silently, emitting a uniform bot score of zero, which would have quietly mis-scored customer traffic rather than erroring. Suppression was manual and staged rather than a single feature-level kill switch. The decisive fix at 14:24 UTC was to stop the creation and propagation of the bad feature file and insert a known-good file into the feature file distribution queue — isolating the poisoned config source rather than the ClickHouse trigger — after which, following a successful test, core traffic was largely flowing as normal by ~14:30 and all systems were restored by 17:06. No data was lost and it was not a breach.

References & provenance

official-postmortem Cloudflare blog: Cloudflare outage on November 18, 2025“a permissions update made table metadata in the r0 database visible alongside default database tables ... This query lacked a database filter, so it now returned duplicate column entries from both databases, more than doubling the rows in the response.”https://blog.cloudflare.com/18-november-2025-outage/
official-postmortem Cloudflare postmortem - FL2 proxy panic detail“thread fl2_worker_thread panicked: called Result::unwrap() on an Err value”https://blog.cloudflare.com/18-november-2025-outage/
news ThousandEyes: Cloudflare Outage Analysis November 18, 2025“(Paraphrase, not a verbatim quotation) ThousandEyes' independent network telemetry attributes the global surge in HTTP 5xx responses to Cloudflare's edge during the outage window.”https://www.thousandeyes.com/blog/cloudflare-outage-analysis-november-18-2025
secondary-aggregator ControlD: The Biggest Cloudflare Outages (secondary/screening)“Downdetector recorded more than 2.1 million issue reports during the outage. More than 435,000 reports were from the United States, while the United Kingdom, Japan, and Germany were also among the most heavily affected regions.”https://controld.com/blog/biggest-cloudflare-outages/

Sourced from public post-incident reports. Quotes are short attributed excerpts for provenance only; the analysis above is original and substantially shorter than its sources. Last verified 2026-07-31.