← All incidents
Incident dossier · Rank #1

CrowdStrike Falcon Channel File 291: The Global Windows BSOD Outage of July 2024

CrowdStrike (impacting Microsoft Windows / Azure hosts) 2024-07-19 1h 18m core impact SoftwareHuman error

On 19 July 2024 CrowdStrike pushed a Rapid Response Content update to Channel File 291 that its own sensor could not safely parse. The IPC Template Type defined 21 input parameter fields, but the integration code supplied only 20 values; a new non-wildcard matching criterion introduced on the 21st field drove the Falcon Content Interpreter to read past the end of the input array. That out-of-bounds read crashed the Windows kernel, sending an estimated 8.5 million hosts into an immediate BSOD boot-loop. Because the content shipped globally with no staged rollout and slipped past a buggy Content Validator, a single defective file grounded airlines, disrupted 911 lines, forced hospitals to cancel surgeries, and froze banks within minutes. The faulty content was live for roughly 78 minutes, but because boot-looping machines could not pull the reverted file, recovery required a per-machine manual fix and stretched across days.

Failure cascade

Failure cascade: trigger → fault → downstream impactTriggerPrimary faultDownstream impactTrigger — Software (2024-07-19)Trigger · Software2024-07-192024-07-19Primary fault at CrowdStrike (impacting Microsoft Windows / Azure hosts) — Global Windows fleet (all cloud and on-prem regions)CrowdStrikeGlobal Windows fleet (all cloud and on-prem regions)Global Windows fleetDownstream service degraded by the fault: Airline check-in/dispatch and airport operationsAirline check-in/dispatch andDownstream service degraded by the fault: 911 emergency call systems (Alaska, Arizona, New Hampshire, Ohio)911 emergency call systemsDownstream service degraded by the fault: Hospital scheduling and elective surgery (Mass General Brigham, Mount Sinai, UVA Health, Harris Health, University of Miami Health)Hospital scheduling and electiveDownstream service degraded by the fault: Retail and consumer banking (Chase, Charles Schwab, TD Bank, Bank of America, Bradesco)Retail and consumer banking+2 more downstream services

Trigger → primary fault → downstream blast radius, derived from the sourced root cause and affected-services record.

Facility & location

Operator
CrowdStrike (impacting Microsoft Windows / Azure hosts)
Data center
Global Windows fleet (all cloud and on-prem regions)
Location
Global
Date
2024-07-19

Impact & scale

Users affected
~8.5 million Windows devices worldwide (<1% of all Windows machines), disproportionately critical enterprise, aviation, healthcare, finance and emergency-services hosts
Financial
~$25B CrowdStrike market-value loss; Delta alone claimed ~$500M in damages; broad third-party economic disruption across sectors
Scope
Sev-1 / Global critical outage — described as one of the largest IT outages in history
Services / systems down
  • Airline check-in/dispatch and airport operations
  • 911 emergency call systems (Alaska, Arizona, New Hampshire, Ohio)
  • Hospital scheduling and elective surgery (Mass General Brigham, Mount Sinai, UVA Health, Harris Health, University of Miami Health)
  • Retail and consumer banking (Chase, Charles Schwab, TD Bank, Bank of America, Bradesco)
  • London Stock Exchange Group data/RNS news service
  • Broadcast, retail POS and government services globally

Impact data & metrics

Windows devices affected8.5 million (< 1% of all Windows machines)
Source-side exposure window (bad file live)~78 minutes — 0409 UTC (bad) to 0527 UTC (reverted), 2024-07-19
Parameter-count mismatch21 input fields defined vs 20 values supplied
Out-of-bounds access indexindex 0x14 (21st element) into a 20-pointer array
Windows bugcheck codePAGE_FAULT_IN_NONPAGED_AREA (0x50), IMAGE_NAME csagent.sys
Automated test cases in the suite12 static cases, all wildcarding the 21st field
Fleet recovery~99% of Windows sensors online by 2024-07-29 17:00 PT (vs ~1% normal weekly variance)
Affected sensor scopeWindows sensor v7.11 and above, online in-window; Mac and Linux hosts not impacted
Faulting read addressffffd6030000006a (canonical kernel pointer, not-present/unmapped PTE)

Magnitude profile

Magnitude sub-scores (0–10)Magnitude sub-scores (0–10)Users 10Users affected (0–10) — breadth of the user/customer population impacted. — scored 10/10.Financial 10Financial impact (0–10) — direct + consequential cost. — scored 10/10.Duration 8Outage duration (0–10) — how long service was degraded/down. — scored 8/10.Blast 10Blast radius (0–10) — how wide the fault propagated across systems/regions. — scored 10/10.
Magnitude 9.6 = blast 10×0.35 + users 10×0.25 + financial 10×0.20 + duration 8×0.20 (sub-scores 0–10 · weighted composite)

Blast radius and user impact are maxed: a global, cross-sector outage hitting aviation, emergency services, healthcare and finance simultaneously, widely called one of the largest IT outages in history. Financial score reflects ~$25B in CrowdStrike market value erased plus large downstream losses (Delta alone ~$500M). Duration is scored 8 rather than 10 because the causative content window was only ~78 minutes, but per-machine manual remediation extended real-world disruption over several days (19-22 July and beyond).

Sequence of events (SOE)

Phased sequence of events2024-02-28 · TRIGGER — Sensor 7.11 released (GA per Preliminary PIR) introducing a new IPC Template Type 'to enable visibility into and detection of novel attack techniques that abuse named pipes'; it defines 21 input parameter fields while the integration code supplies only 20 to the Content Interpreter — a latent mismatch masked because all instances wildcard the 21st field.TRIGGER2024-02-282024-07-19 04:09 UTC · TRIGGER — Two additional IPC Template Instances are deployed as a new version of Channel File 291 (C-00000291*.sys, 0409 UTC timestamp); one 'introduced a non-wildcard matching criterion for the 21st input parameter,' now requiring the sensor to inspect a 21st value.TRIGGER2024-07-19 04:09 U2024-07-19 ~04:09 UTC · DETECTION — The Content Validator evaluates the new instances but 'based its assessment on the expectation that the IPC Template Type would be provided with 21 inputs'; the problematic instance passes and is sent to the Content Interpreter.DETECTION2024-07-19 ~04:09 U2024-07-19 (first named-pipe event post-delivery) · CASCADE — At 'the next IPC notification from the operating system,' the Content Interpreter evaluates the new instances, attempts to access the 21st value against a 20-input array, and performs 'an out-of-bounds memory read beyond the end of the input data array.'CASCADE2024-07-19 (firs2024-07-19 · IMPACT — Kernel page fault in csagent.sys: bugcheck 'PAGE_FAULT_IN_NONPAGED_AREA (50)', faulting read at index 0x14 (21st element) of a 20-pointer array via 'mov r9d,dword ptr [r8]' against invalid, not-present address ffffd6030000006a.IMPACT2024-07-192024-07-19 · IMPACT — Because csagent.sys is a boot-start kernel driver 'loaded from an early phase of system boot,' the bugcheck presents as a Windows BSOD and re-triggers on reboot, producing a boot loop on affected hosts.IMPACT2024-07-192024-07-19 within minutes · CASCADE — The bad file propagates to online 7.11+ Windows hosts worldwide; Microsoft later estimates the update 'affected 8.5 million Windows devices, or less than one percent of all Windows machines,' cascading into airlines, banking, healthcare and emergency services.CASCADE2024-07-19 withi2024-07-19 05:27 UTC · MITIGATION — CrowdStrike publishes the reverted (good) Channel File 291: a 'C-00000291*.sys with timestamp of 2024-07-19 0527 UTC or later is the reverted (good) version,' closing the exposure window at source (~78 minutes after 0409 UTC).MITIGATION2024-07-19 05:27 U2024-07-19 post-05:27 UTC · CASCADE — Already-crashed hosts stuck in a boot loop cannot connect to receive the reverted file, so the cloud fix does not self-heal bugchecked machines; recovery must be performed locally on each host.CASCADE2024-07-19 post-05:27 U2024-07-19 · MITIGATION — CrowdStrike issues manual remediation: boot Windows into Safe Mode / recovery, locate the files matching C-00000291*.sys and delete them, then reboot — de-energising the fault on each host.MITIGATION2024-07-192024-07-19 · IMPACT — Recovery is slowed on hardened fleets because BitLocker-encrypted hosts may require a recovery key before Safe Mode file deletion can proceed, adding per-machine manual key retrieval.IMPACT2024-07-192024-07-19 · RECOVERY — A Sensor Content Compiler patch that 'validates the number of inputs provided by a Template Type' is developed the same day (compile-time guard, later promoted to production).RECOVERY2024-07-192024-07-25 · RECOVERY — 'Bounds checking was added to the Content Interpreter function that retrieves input strings,' plus a check that input-array size matches the expected input count — the runtime guard that directly prevents the out-of-bounds read.RECOVERY2024-07-252024-07-27 · RECOVERY — The Sensor Content Compiler input-count patch 'went into production' and 'verified that no other Template Types were providing an incorrect number of inputs, on any platform.'RECOVERY2024-07-272024-07-29 17:00 PT · RECOVERY — 'As of July 29 at 5 p.m. PT, using a week-over-week comparison, ~99% of Windows sensors are online compared to before the content update' (against a typical ~1% week-over-week variance) — near-full fleet restoration.RECOVERY2024-07-29 17:00 P2024-08-06 · RESTORED — CrowdStrike publishes the External Technical Root Cause Analysis for Channel File 291, confirming the 21-vs-20 mismatch, the out-of-bounds read, the validator/test gaps, and engagement of 'two independent third-party software security vendors.'RESTORED2024-08-062024-08-09 · RESTORED — Runtime bounds-check and input-count fixes are 'backported to all Windows sensor versions 7.11 and above through a sensor software hotfix release,' generally available by this date.RESTORED2024-08-092024-08-19 · RESTORED — The Content Validator is modified to reject Template Instances that 'match over more fields than are being provided as input,' and 'to only allow wildcard matching criteria in the 21st field' — the content-layer defense released to production, closing the last gap.RESTORED2024-08-19

Root cause

The July 19, 2024 global Windows BSOD outage originated from a latent parameter-count mismatch inside CrowdStrike's Falcon sensor that lay dormant for roughly five months before a routine content update triggered it. With the release of sensor version 7.11 in February 2024, CrowdStrike introduced a new "IPC Template Type" to detect abuse of Windows named-pipe interprocess communication. Per CrowdStrike's External Technical Root Cause Analysis (2024-08-06), the Template Type Definitions file declared this detection type would receive 21 input parameter fields, but the sensor's integration code that actually invokes the Content Interpreter supplied only 20: "The new IPC Template Type defined 21 input parameter fields, but the integration code that invoked the Content Interpreter with Channel File 291's Template Instances supplied only 20 input values to match against." This mismatch "was not detected during development of the IPC Template Type." The delivery vehicle was Rapid Response Content. Falcon separates Sensor Content (compiled into the signed sensor) from Rapid Response Content, delivered as numbered Channel Files and interpreted at runtime by a regular-expression-based Content Interpreter; IPC Template Instances are delivered via Channel File 291. From the first IPC Template Instance (released to production March 5, 2024) through the first several successful field deployments, every instance used a regex wildcard matching criterion in the 21st field, so the interpreter never dereferenced a real 21st value and the defect stayed hidden — CrowdStrike attributes the evasion "to the use of wildcard matching criteria for the 21st input during testing and in the initial IPC Template Instances." On July 19, 2024, two additional IPC Template Instances were deployed; one "introduced a non-wildcard matching criterion for the 21st input parameter," forcing the interpreter to read an element that did not exist. The failure mechanism was a kernel out-of-bounds memory read. At the next IPC (named-pipe) notification from the OS, the interpreter evaluated the new instances and attempted to compare against the 21st input value; because it had been provided only 20, "the attempt to access the 21st value produced an out-of-bounds memory read beyond the end of the input data array and resulted in a system crash." Because this code path runs inside csagent.sys — CrowdStrike's kernel-mode file system filter driver, loaded early at boot — the fault was unrecoverable, producing a PAGE_FAULT_IN_NONPAGED_AREA (bugcheck 0x50) and a Windows BSOD. The crash dump identifies csagent.sys as the faulting module, READ_ADDRESS ffffd603`0000006a, with register rax holding the base of a 20-input pointer array and r11 indexing 0x14 (the 21st element). Multiple safeguards failed in sequence. There was no sensor-compile-time validation of the input-field count, so the 21-vs-20 discrepancy was never caught in the build. The Content Validator contained a logic error: it "based its assessment on the expectation that the IPC Template Type would be provided with 21 inputs," so it approved content the runtime interpreter could not safely consume — CrowdStrike states (PIR / Remediation Hub): "Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data." Testing did not expose the defect because the 12 automated test cases and the stress-test Template Instance all "included a regex wildcard matching criterion in the 21st field for all Template Instances," so no test ever exercised a non-wildcard 21st field. The Content Interpreter also had no runtime array-bounds check to guard the access, and Template Instances were not deployed through a staged/canary rollout, so the July 19 content reached the online Windows fleet essentially at once. Escalation was therefore immediate and simultaneous. Windows hosts running sensor 7.11+ that were online between 04:09 and 05:27 UTC received the update and crashed on their next named-pipe IPC event; CrowdStrike reverted the defective update at 05:27 UTC. Mac and Linux hosts were not impacted. Microsoft publicly estimated the update affected 8.5 million Windows devices (less than one percent of all Windows machines). Recovery required per-host intervention because affected machines re-loaded the same channel file on reboot and re-crashed (boot loop); CrowdStrike reported that as of July 29 at 5 p.m. PT approximately 99% of Windows sensors were back online. CrowdStrike summarizes the cause as "the confluence of these issues": the 21-vs-20 mismatch between what the Content Validator validated and what the interpreter received, the latent out-of-bounds read in the Content Interpreter, and the lack of a specific test for a non-wildcard 21st field. An independent third-party review confirmed the flaw is a read-only fault, "not exploitable by a threat actor."

Contributing factors

Correction of errors (COE)

Lessons learnt

Improvements & remediation

Comprehensive analysis

What actually broke, in one line

A boot-start Windows kernel driver (csagent.sys) read one element past the end of a 20-pointer array because a content file (Channel File 291) told it to inspect a 21st input that the sensor code never provided. The out-of-bounds read hit an unmapped kernel page (ffffd6030000006a, not-present PTE), triggering a PAGE_FAULT_IN_NONPAGED_AREA (0x50) bugcheck. Because the driver loads early in boot, the BSOD recurred on every restart, producing boot loops on ~8.5 million Windows devices.

Why the safety nets all failed together

This was not one bug but a confluence. The 21-vs-20 mismatch shipped in Feb 2024 and hid for months behind wildcard matching on the 21st field. The Content Validator — the last gate — validated against the wrong contract (it assumed 21 inputs), so it green-lit the fatal instance. The test suite (12 static cases) never exercised a non-wildcard 21st field, so nothing forced the dormant read into the open. The Content Interpreter had no runtime bounds check to fail safe. Remove any one of these and there is no BSOD; the RCA itself calls it 'the confluence of these issues.'

Why recovery was slow despite a fast fix

CrowdStrike reverted the channel file within roughly 78 minutes (0409 UTC bad, 0527 UTC good). But the fix was useless to already-crashed hosts: a boot-looping machine cannot reach the network to pull the corrected file, and BitLocker-encrypted endpoints needed recovery keys before Safe-Mode file deletion. Recovery therefore became a manual, per-host, hands-on-keyboard effort — the reason full-fleet restoration to ~99% took until July 29 rather than minutes. The lesson: the reach of recovery, not the speed of the code fix, sets the outage length for kernel-early-boot faults.

The structural lesson for content-driven security agents

CrowdStrike's own framing — 'Rapid Response Content is configuration data; it is not code or a kernel driver' — is precisely the trap. When configuration data is interpreted in kernel context by an early-boot driver, it has the blast radius of code and must inherit code-grade controls: strict producer/consumer contracts, runtime bounds/fail-safe on untrusted content, negative-path and fuzz testing, and staged canary delivery instead of instantaneous global push. Every remediation CrowdStrike shipped (compile-time count check, runtime bounds check, validator hardening, canary rings, customer-controlled delivery, third-party review) is an instance of that single principle.

Scope, accuracy, and disclosed limits

Impact was confined to online Windows 7.11+ sensors that received the bad file in-window; Mac and Linux were unaffected because the vulnerable IPC path and channel file were Windows-specific. The crash address ffffd6030000006a is a canonical kernel-space address whose PTE is 'not valid' — i.e., unmapped/not-present, not 'non-canonical' or a wild pointer (a corrected characterization). Exact-minute timestamps (0409/0527 UTC) and the 8.5M figure originate from the Remediation Hub and Microsoft's blog respectively, not the RCA; the Feb 28 GA date is from the Preliminary PIR (the RCA says only 'February 2024'). All core mechanism, register, bugcheck, test-case, and remediation-date claims are verbatim-verified against the RCA PDF re-fetched this session.

Technical deep-dive

The crash is fully attributable from the kernel dump reproduced in the External RCA. The bugcheck is "PAGE_FAULT_IN_NONPAGED_AREA (50) Invalid system memory was referenced," with "IMAGE_NAME: csagent.sys / MODULE_NAME: csagent." Arg1 (memory referenced) is ffffd6030000006a, and the PTE examination (!pte ffffd603`0000006a) shows the page is "not valid" — the address is a canonical kernel-space pointer that is simply not mapped (not-present PTE), not a wild user pointer. The faulting instruction is a read through r8: "csagent+0xe14ed: mov r9d,dword ptr [r8] ds:ffffd603`0000006a=????????" Tracing register state, "the address of the 20-input pointer array is held in register rax, and register r11 indicates that the input to be retrieved is at index 0x14, i.e., the 21st element. Examining the input array, we indeed find an array of 20 pointers to input string structures, followed by a 21st value which does not point to valid memory." Index 0x14 = decimal 20 = the 21st element in a zero-based array of 20, so the read walks exactly one slot past the end. Control flow: the IPC Template Type is compiled sensor content, not executable content; Channel File 291 carries Template Instances whose matching criteria the Content Interpreter (a regex-based C++ engine) evaluates "at the next IPC notification from the operating system" — a named-pipe creation event. The Interpreter was built to supply 20 input values, but the 2024-07-19 instance demanded a comparison against a 21st. Because the array holds only 20 valid string-structure pointers, the retrieve-input-string routine (annotated TemplateGetString in the RCA disassembly) loads the invalid 21st pointer into r8, passes a NULL check, then dereferences it — producing the page fault in kernel context and an immediate bugcheck. Since csagent.sys is a boot-start driver, the machine BSODs and re-enters the same code path on restart — a self-perpetuating boot loop that also blocks the host from pulling the already-reverted good channel file, which is why remediation had to be manual and per-host. Mac and Linux hosts were unaffected: the vulnerable IPC Template Type / Content Interpreter path and this channel file applied to the Windows sensor. Fixes trace directly to code. Runtime guard: "Bounds checking was added to the Content Interpreter function that retrieves input strings on July 25, 2024. An additional check that the size of the input array matches the number of inputs expected by the Rapid Response Content was added at the same time," backported to all 7.11+ Windows sensors (GA by August 9, 2024). Compile-time guard: a Sensor Content Compiler patch "that validates the number of inputs provided by a Template Type was developed on July 19, 2024, and went into production on July 27, 2024," and "verified that no other Template Types were providing an incorrect number of inputs, on any platform." Content guard (defense-in-depth): the Content Validator "was modified to only allow wildcard matching criteria in the 21st field, which prevents the out-of-bounds access in the sensors that only provide 20 inputs," with additional over-match checks released to production by August 19, 2024. CrowdStrike also engaged "two independent third-party software security vendors."

References & provenance

Sourced from public post-incident reports. Quotes are short attributed excerpts for provenance only; the analysis above is original and substantially shorter than its sources. Last verified 2026-07-31.

Root access required

The DC Incidents dossier is a root-only module. Sign in with an authorized account to continue.

Back to Home