Airline And Emergency Services Halted Worldwide Thanks to A Simple Update 

On Friday morning, Karen came to work for Delta Airlines at 4:30AM like she always did to help the early bird travelers check in and catch their flights.  When she booted up her computer, she saw something she had not seen in 20 years.  It was the “Blue Screen of Death.”   She asked a co-worker, and her computer was showing the same thing.   What was she going to do with all those travelers that can’t check in?  By 10:00AM EDT, Delta had cancelled more than 600 flights.    By Saturday, July 20th, over 4,000 flights would be cancelled throughout the airline industry globally leaving passengers stranded or dealing with hours of delay.   

What happened?  Shortly after midnight, CrowdStrike, a security software provider, pushed out a single content update to its 24,000 customers worldwide.  It was a small update designed to stop new attacks hackers have been using.   On installation, the configuration update triggered a logic error that resulted in the famous Blue Screen of Death.  CrowdStrike could not just back out the patch.  The customer computers were inoperable.  There is no automated way to back out the software.  It required a “Safe Mode” boot which requires someone to be physically next to the device and enter a set of keystrokes during boot.  Only then could the bogus file be removed allowing the computer to operate as normal.   

The impact of this mistake was felt worldwide.  Several states, including Arizona, experienced 911 service outages.   By 3:00AM, the Federal Aviation Administration announced that all Delta, United, Allegiant, and American flights were grounded.  Transportation services in the Northeast, including trains and buses were experiencing delays.  Global banks reported services disruptions, from Australia, South Africa, Israel, and New Zealand.  Hospitals in Germany and the UK were cancelling all non-urgent surgeries due to the event.   Even locally, Maricopa County reported that their Dominion voting machines were malfunctioning due to the automatic update.   

CrowdStrike is a leader in the cybersecurity space.   Their Falcon Sensor product is an endpoint detection response tool.  It goes onto each individual computer and searches and stops known malware from firing.  The company was founded in 2011.   Some may recall that CrowdStrike was called to investigate the alleged Democratic National Convention server hack in 2016.  Since then, the small company has enjoyed tremendous growth and success.  The company says its customers include 298 Fortune 500 companies, eight out of the top 10 financial services firms, seven out of the top 10 manufacturers, six of the top 10 healthcare providers and eight out of the top 10 food and beverage companies.  With this many big names, you can see why the impact of this failed Falcon Sensor update caused such a huge problem.  

It is appalling that any company, much less a global leader like this, would automatically push out software which they had not validated.      There have been rumblings on the internet that this could have been done on purpose for some nefarious reason, but I disagree.   CrowdStrike should have manually validate their software at the developer level and then again at an independent test and verification department level and then again at a pilot customer site before pushing anything out to the world.    

As for the customers caught up in this, we would not recommend immediate auto-updates for anything.   While working in the industry, we regularly waited a day to test the vendor updates and ran through a suite of tests before releasing it to our customers.  The fact that there was no control at the customer level made this event that much worse. 

This event shows us the need for every business to have disaster recovery and contingency plans. Whether it’s due to cyberattacks, technical issues, or natural disasters, having an effective plan is crucial for maintaining business continuity and minimizing downtime. 

In a world where we are increasingly dependent on computers for our businesses to function, be ready to run the old school way as a backup – just in case.    

The original article was published in the Sierra Vista Herald and can be found here.

The Cyber Guys: Critical Vulnerabilities in Voting Machines – Easy To Hack

J. Alex Halderman, a Computer Science professor at the University of Michigan, walks into a courtroom in Georgia. He borrowed a pen from the defense attorney and in under a minute he had broken into a Dominion voting machine where he could make the results anything that he wanted without a trace of his breach. 

Dr. Halderman was an expert witness that demonstrated just how vulnerable these voting machines are to tampering. He used a pen to hold down the power button on the voting machine. He waited 7 seconds until it came up in “safe” mode. From there he could open files and change the contents of files to include the results and audit files without a password.

Later Dr. Halderman showed how with just a $30 purchase on Amazon, he was able to create a technician card for the voting machines that gave him super user access. Once programmed, a hacker could make as many technician cards as needed and distribute across the voting area.

At this point you might be thinking, OK, but how many computer science professors are going to hack a voting machine? Well, it turns out in August of 2018 at a DEFCON hackathon conference, it took an 11-year-old boy 10 minutes to hack a simulated Florida state voting website and change the results of the election. There was not just one child, but 30 of the 50 children with age ranging from 8 to 16 were able to hack the simulated election website. 

Over the last 6 years there have been many lawsuits concerning the use of these machines all over the country. Not only in Georgia, but Pennsylvania, Michigan, Texas, Arizona, and more.

But it’s not just Dominion machines that have vulnerabilities. In the summer of 2020, students from the University of Pennsylvania conducted an audit of the ES&S voting system1. ES&S claims to be the world’s largest e-voting system vendor, supporting more than 67 million voter registrations with 97,000 touchscreen voting machines installed in 20 states, with optical ballot readers in 43 states. 

The team reported numerous critical vulnerabilities existed in nearly every component of the ES&S system. They identified serious and undetectable attacks that could be carried out by poll-workers and even individual voters. What makes matters worse is that these attacks are not limited to the local machines. There are several attacks that propagate like a virus to the backend systems on the network affecting all the results of a precinct or an entire county. According to their report, virtually every mechanism for assuring the integrity of precinct results and backend systems can be circumvented. With these machines, they found that almost every major component of ES&S can be altered or replaced by other components with which it communicates. In other words, there are many ways to get to the back end to modify the results. 

The calibration of the touchscreen affects how the voters’ input maps to different locations on the screen. If the calibration is incorrect, it could alter the voters’ choices. For example I vote for Alice for the school board on the touch screen, but the machine selected the opponent, Bob. This happened in Pennsylvania in the 2023 Superior Court election. When a voter would select ‘yes’ or ‘no’ on their ballot for one of the candidates, the vote was recorded on the paper ballot and the machine for the other candidate.

Some countries like Argentina and the Philippines have recently banned the use of the machines due to their vulnerabilities. There is talk in different states around the country about doing the same. What should we do to ensure that each voter’s choice counts?

The original article was published in the Sierra Vista Herald here.