Lessons from the Global CrowdStrike Incident

On July 18, 2024, a troubling scene unfolded across numerous organizations worldwide as Windows PCs and servers were crippled by the Blue Screen of Death, attributed to a faulty update from the CrowdStrike IS application. This significant disruption affected critical sectors including corporations, banks, and airports, spotlighting the fragility of our interconnected IT infrastructure.

The Root of the Problem

The primary issue stemmed from a Rapid Response Content update for the Falcon sensor, published on July 19, 2024, at 04:09 UTC, to Windows hosts running sensor version 7.11 and above. This update was intended to gather telemetry on new threat techniques observed by CrowdStrike. However, it triggered crashes (BSOD) on systems online between 04:09 and 05:27 UTC. Notably, Mac and Linux hosts, as well as Windows hosts that were not online or did not connect during this period, were not impacted.

The crashes were due to a defect in the Rapid Response Content that went undetected during validation checks. When loaded by the Falcon sensor, it caused an out-of-bounds memory read, leading to the Windows crashes. This oversight not only caused severe disruptions but also exposed potential vulnerabilities in how critical updates, especially those involving kernel-level interactions, are deployed across diverse environments.

Immediate Workarounds and Long-term Solutions

In the face of this crisis, organizations had to act swiftly to mitigate damage. Immediate steps included:

• • Check and follow the most up-to-date instructions from CrowdStrike
  • Booting systems into Safe Mode or the Windows Recovery Environment to circumvent the faulty update.
  • For those dealing with encrypted drives, Action1 offers a helpful “Bitlocker Keys Report” which might save crucial hours during recovery.

Here’s also a potential fix that does not require the Bitlocker key, although it has not been widely tested:

1. Cycle Through BSODs: Continue through the BSODs until the recovery screen appears.
2. Accessing Advanced Options:
   • • Navigate to Troubleshoot > Advanced Options > Startup Settings.
     • Press “Restart.”
3. Bypassing Bitlocker:
   • • At the first Bitlocker recovery prompt, press “Esc.”
     • At the second prompt, choose “Skip This Drive.”
4. Command Prompt Adjustments:
   • • Navigate back to Troubleshoot > Advanced Options > Command Prompt.
     • Enter bcdedit /set {default} safeboot minimal and press enter.
5. Return to Normal Operation:
   • • Return to the WinRE main menu and select “Continue.” It may cycle 2-3 times.
     • If booted into safe mode, log in as usual.
     • Navigate to C:\Windows\System32\drivers\Crowdstrike and delete the file starting with C-00000291*.sys.
     • Open Command Prompt as administrator, type bcdedit /deletevalue {default} safeboot, then restart normally to confirm behavior.

Broader Implications for Industries

The transportation sector, which heavily relies on timely and continuous operations, was among the hardest hit. This incident underscores the need for robust disaster recovery plans and redundant systems to maintain critical functions even during IT outages.

Comparisons and Reflections

Ironically, the software designed to prevent incidents like the 2017 WannaCry ransomware attack became the source of a major disruption. This situation compels a reevaluation of our dependency on single vendors for critical security functions and the practices surrounding update testing and deployment.

Accountability and Moving Forward

CrowdStrike must address this mishap with transparent communications and remedial measures. At the same time, organizations should introspect on their internal processes to better manage and mitigate risks associated with critical updates.

Regulatory and Market Impact

Regulatory bodies might intervene to ensure stricter compliance with update testing protocols. This incident might also reshape strategies around cloud-based and global security solutions, potentially driving demand for more decentralized and resilient approaches.

Conclusion

The CrowdStrike incident is more than a temporary technical glitch; it is a critical wake-up call for enhancing cybersecurity frameworks. By adopting rigorous testing environments, phased deployments, and robust disaster recovery protocols, organizations can strengthen their defenses against such catastrophic failures, ensuring reliability and trust in the digital era.
At Action1, we embody this commitment by proactively maintaining high security and operational standards to preempt the risk of IT outages.

About Action1

Action1 reinvents patch management with an infinitely scalable, highly secure, cloud-native platform configurable in 5 minutes—and it just works, with no VPN needed. Featuring unified OS and third-party patching with peer-to-peer patch distribution and integrated real-time vulnerability assessment, it enables autonomous patch compliance that preempts ransomware and security risks, all while eliminating costly routine labor. Trusted by thousands of enterprises managing millions of endpoints globally, Action1 is certified for SOC 2 and ISO 27001.