CrowdStrike, a leading cybersecurity technology provider, offers security services for endpoints, cloud workloads, identity, and data. Trusted by over 298 of the Fortune 500, 43 U.S. states, 6 out of the top 10 healthcare providers, and 8 out of the top 10 financial services firms, CrowdStrike is a prominent player in the industry.
Its Falcon platform is a unified, cloud-delivered security solution designed to prevent all types of attacks, including malware and beyond. However, a recent update to the Falcon Sensor agent on Windows has triggered a critical issue: a Blue Screen of Death (BSOD) boot loop that renders affected systems unusable. This widespread problem has disrupted operations across various sectors, notably impacting airlines, banks, and healthcare providers.
CrowdStrike has acknowledged the issue and halted further deployment of the faulty update. An alert sent to users confirms that they are aware of crashes on Windows hosts related to the Falcon Sensor, specifically bugcheck/blue screen errors. Unfortunately, an official solution to recover Windows PCs caught in the BSOD boot loop remains elusive. There are several workarounds to fix the issue, read about them below.
Official Workaround for CrowdStrike BSOD issue on Windows PCs:
- Boot your Windows PC into Safe Mode or Windows Recovery Environment.
- Go to C:\Windows\System32\drivers\CrowdStrike
- Locate and delete file matching "C-00000291*.sys"
- Boot normally
Another way is to prevent CrowdStrike from starting using either of the following methods:
Method 1:
- Go into Command Prompt from Recovery options.
- Navigate to C:\Windows\System32\Drivers
- Rename CrowdStrike to Crowdstrike_Old
- Restart the PC.
Method 2:
- Boot your Windows PC into Safe Mode or Windows Recovery Environment.
- Go to Windows Registry
- Edit the following key to disable the csagent.sys from loading.
- HKLM:\SYSTEM\CurrentControlSet\Services\CSAgent\Start from a 1 to a 4
If you are running Windows on a AWS EC2 instance, you can try the following method:
-
Detach the EBS volume from the impacted EC2
-
Attach the EBS volume to a new EC2
-
Fix the CrowdStrike driver folder as per the workaround suggested by CrowdStrike
-
Detach the EBS volume from the new EC2 instance
-
Attach the EBS volume to the impacted EC2 instance
The above method can also be applied for Windows instances running on Google Cloud Platform.
Update 1:
CrowdStrike CEO George Kurtz tweeted the following in response to the outages caused by CrowdStrike.
CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed. We…
— George Kurtz (@George_Kurtz) July 19, 2024
Here's the official summary of the details published by CrowdStrike:
Summary
CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.
Details
Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted.
Windows hosts which are brought online after 0527 UTC will also not be impacted
Hosts running Windows 7/2008 R2 are not impacted
This issue is not impacting Mac- or Linux-based hosts
Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
Channel file "C-00000291*.sys" with timestamp of 0409 UTC is the problematic version.
Current Action:
CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:
Workaround Steps for individual hosts:
Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:
Note: Bitlocker-encrypted hosts may require a recovery key.
Boot Windows into Safe Mode or the Windows Recovery Environment
NOTE: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation.
Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Boot the host normally.
Workaround Steps for public cloud or similar environment including virtual:
Option 1:
Detach the operating system disk volume from the impacted virtual server
Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
Attach/mount the volume to to a new virtual server
Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Detach the volume from the new virtual server
Reattach the fixed volume to the impacted virtual server
Option 2:
Roll back to a snapshot before 0409 UTC.
AWS-specific documentation:
To attach an EBS volume to an instance
Detach an Amazon EBS volume from an instance
Azure environments:
Please see this Microsoft article
Bitlocker recovery-related KBs:
BitLocker recovery in Microsoft Azure
BitLocker recovery in Microsoft environments using SCCM
BitLocker recovery in Microsoft environments using Active Directory and GPOs
BitLocker recovery in Microsoft environments using Ivanti Endpoint Manager
Source: CrowdStrike
31 Comments - Add comment