CrowdStrike: Content Update Glitch Caused Worldwide IT Crash

CrowdStrike says a glitch in test software led to last week’s massive IT outage.

The cybersecurity company released a report Wednesday (July 24), five days after a crash affected 8.5 million Windows machines around the world, leading to chaos at banks, airports and hospitals.

“CrowdStrike delivers security content configuration updates to our sensors in two ways: Sensor Content that is shipped with our sensor directly, and Rapid Response Content that is designed to respond to the changing threat landscape at operational speed,” the report said. “The issue on Friday involved a Rapid Response Content update with an undetected error.”

The report also outlines what CrowdStrike aims to do to prevent the problem from recurring, such as implementing “a staggered deployment strategy for Rapid Response Content in which updates are gradually deployed to larger portions of the sensor base,” while also giving customers more control over the delivery of these updates, letting them choose when and where they are deployed.

As PYMNTS wrote last week, the incident has shone a spotlight on software updates. In an interview here Friday (July 19), CompoSecure/Arculus Chief Product and Innovation Officer Adam Lowe noted that when a software update fails, companies usually have contingency plans.

But issues with essential security software like CrowdStrike can quickly escalate, and disruptions to core functions, especially at the Windows startup level, can be difficult to correct.

“In such cases, it may necessitate a complete system reinstallation from a prior backup, akin to wiping a hard drive and starting anew,” that report said. “This process is both complex and time-consuming, especially for systems that are locked out at startup, leaving limited options for a swift resolution.”

Meanwhile, PYMNTS discussed the crash earlier this week with Finexio CEO Ernest Rolfson, who stressed the importance of quality control in software updates.

Rolfson said he was taken aback by the timing of a recent software update from a Finexio partner, rolled out in the morning and during the workweek.

“Typically, updates are scheduled after hours or on weekends to minimize disruptions, given the fact that best practices suggest staggering the release to avoid widespread issues if problems arise,” PYMNTS wrote.

Meanwhile, the fallout from the outage continues, with the Department of Transportation launching an investigation into Delta Air Lines’ handling of the incident, and two members of the House Homeland Security Committee calling on CrowdStrike’s CEO to testify about the crash.