Oldsmar’s water supply attack is a warning, experts say. It could’ve been worse.

An attack on Oldsmar’s water supply last week highlighted cybersecurity experts’ longstanding concerns about the security of the country’s critical infrastructure.

Though a change the attackers made to the city’s water treatment process was quickly reversed, experts said it got farther than any other hacking attempt to physically impact critical infrastructure in the U.S.

And yet Oldsmar’s residents got lucky, experts said. Instead of causing widespread injury or loss of life, the attack could stand as a huge red flag marking quietly pervasive flaws in national security.

“This could’ve been so much worse,” said David Kennedy, a former hacker for the National Security Agency. “And it wasn’t. But this is a wakeup call.”

The attackers briefly multiplied the amount of sodium hydroxide, or lye, used in the city’s water treatment by a factor of more than 100. Though the attack was quickly recognized, that action alone could make it the most successful cyberattack on critical infrastructure in the U.S. to date.

More sophisticated assailants “could’ve easily pulled this off and then some,” added Kennedy, who’s now chief executive of cybersecurity firm TrustedSec, which tests vulnerabilities in a variety of infrastructure including water treatment plants.

But Friday’s attack, which is being investigated by the Pinellas County Sheriff’s Office and federal agencies, didn’t showcase a high level of hacking skill, experts said, and it’s unlikely that the culprit was another country or a state-backed group. Instead, the hacker got into the plant’s computer system through software that allows supervisors to access the system remotely. It was the equivalent of walking through an unlocked front door.

‘It could be a nightmare’

Cyberattacks have become increasingly familiar to the general public in recent years as more aspects of life moved online. But critical infrastructure attacks present some of the most devastating potential for digital weapons.

The earliest known instance was discovered in 2010 when malware caused an Iranian nuclear facility to malfunction, destroying key parts of itself. Six years later in Ukraine, attackers targeted the electric grid, cutting power for 230,000 residents. And the same year, Iranian hackers were suspected of breaking into the control systems for a dam in New York, which did not result in any damage.

A 2019 report from the American Water Works Association cited an assertion by multiple federal agencies that cyberattacks were the biggest threat to America’s critical infrastructure. It also noted that there have been high-profile attacks on water providers in recent years, including one on Atlanta utilities that left employees unable to turn on their work computers for a week after the attack.

Spend your days with Hayes

Subscribe to our free Stephinitely newsletter

Columnist Stephanie Hayes will share thoughts, feelings and funny business with you every Monday.

Loading...

You’re all signed up!

Want more of our free, weekly newsletters in your inbox? Let’s get started.

Explore all your options

While officials didn’t say exactly how the attacker breached the Oldsmar system, they did say that it happened through a program called TeamViewer. The software is used for remote access, said Gus Serino, principal industrial control systems analyst for Dragos Inc., but isn’t designed to be used in critical infrastructure.

“TeamViewer is aware of media reports regarding an unauthorized remote access to the Oldsmar water treatment facility and we are monitoring the situation very closely,” the company said in a statement. “We don’t have any indication that our software or platform has been compromised.”

Friday’s attack became public knowledge — and national news — on Monday. White House Press Secretary Jen Psaki addressed it during a Tuesday news conference, saying that the Biden administration is “focused on elevating cybersecurity as a threat that has only increased over the past several years.” The incident also became a hot topic among the national cybersecurity community.

Oldsmar’s breach, experts say, is a symptom of a larger security issue within the critical infrastructure sector. While other utilities might not have the same issue as Oldsmar, security on a whole is something many water system managers are playing catch-up on.

Oldsmar Mayor Eric Seidel said the investigating agencies had asked him to withhold further comment on the attack. But other Tampa Bay-area water providers said they don’t have the same vulnerabilities that opened the door for the Oldsmar attack.

Chuck Carden, interim general manager of Tampa Bay Water, which supplies Pinellas, Hillsborough and Pasco counties, said the same breach would not be possible in Tampa Bay Water’s system because of strict safeguards.

The regional water system is on a private network, which makes it less susceptible to outside attacks. The system also does not use remote access, meaning employees have to be physically at the operations center to log on. Officials in Dunedin and St. Petersburg said the same.

St. Petersburg’s water utility and Tampa Bay Water completed a cybersecurity vulnerability assessment through the Department of Homeland Security within the past six months, and no vulnerabilities were found, Water Resources Director John Palenchar said. Dunedin is expected to complete a $32 million upgrade next month to its water treatment plant including new pumps, processes, software, “a total retrofit,” said Paul Stanek, Dunedin’s assistant public works and utilities director.

Pinellas County administrator Barry Burton also assured officials at the county commission meeting on Tuesday that “the situation that existed in Oldsmar doesn’t exist within our Pinellas County systems.”

“There were a number of specific situations that occurred there where we have safeguards,” Burton said.

Officials in Hillsborough County, New Port Richey and Port Richey said they use remote access systems they believe to be secure; Port Richey said it uses a proprietary system. A Tampa water official said he is confident that Tampa has the necessary firewall protection to prevent a similar attack.

Oldsmar was fortunate, experts said, that a plant operator was watching the computer screen as the hacker took over, clicked through software and changed the amount of lye from 100 parts per million to 11,100 parts per million. After the attack, which Sheriff Bob Gualtieri said lasted three to five minutes, the plant operator immediately reversed the change.

It was lucky in another way, too: Despite lye’s poisonous reputation — in addition to being used in small amounts in water treatment to control acidity, it’s commonly found in household drain cleaner — even if the increased amount had it made it to the water supply, it probably wouldn’t have been enough to kill those who drank it or cause them to fall seriously ill, said Katherine Alfredo, a University of South Florida assistant professor who specializes in water quality and treatment.

Still, she said, water with that much lye could irritate skin and cause rashes. That’s what happened in 2007 when a small city in Massachusetts accidentally used too much lye in water treatment. And, depending on other variables, it could raise the pH of the water enough to leach other harmful chemicals from pipes.

“I think it’s just the overall vulnerability of our infrastructure — I would say, honestly, infrastructure the American people don’t think too much of,” Alfredo said. “Water utilities function day in and day out, and people don’t think about it.”

Oldsmar city officials said Monday that other safeguards would have caught the contaminated water before it entered the water supply. Experts said that alarms and manual testing could have noticed the unbalanced pH, and that many treatment plants do have those processes in place. It’s unclear exactly what Oldsmar’s procedures are.

But that doesn’t change the fact that the plant’s security was so easily breached, experts said, or that it got closer to harming the public than any other apparently malicious attack on critical infrastructure they knew of.

It highlights the need for many of the nation’s water, electrical, nuclear and other facilities to be prepared for such an assault, even as the basic functions of those places become more and more intertwined with the internet, especially during the pandemic and accompanying work-from-home revolution, which has pushed more people toward potentially insecure technology.

“It could be a nightmare,” said Austin Berglas, a former FBI agent who specialized in cybercrime. “Every single new device that’s put on the network is an avenue for the bad guys to compromise.”

‘It’s going to happen in the future’

Much of the lack of preparation across the industry comes from two issues: budget constraints and antiquated systems.

While larger utilities and companies are well prepared to weather such attacks because of their resources, smaller municipalities and private firms hired by them may not have the same capabilities.

“They’re really working on thin margins. Their whole mission is to be as efficient as they can,” said Gus Serino, principal industrial control systems security analyst for Dragos Inc.

The technology, too, presents a problem. Many critical infrastructure systems, such as water treatment, were retrofitted with internet connectivity to help with information sharing and troubleshooting.

“When they were designed and deployed in the field, security (was) an afterthought,” said Guillermo Francia III, professor of electrical and computer engineering at the University of West Florida.

President Joe Biden, during his campaign, released an infrastructure plan that included promises to make clean water available to all, hold polluters accountable and invest in more efficient water technology. But it didn’t address cybersecurity issues.

“There’s not money out there for utilities to do this,” Alfredo said. “The production of safe water is going to trump everything. But by solely focusing on that, you’re leaving yourself vulnerable.”

Securing these systems, however, is not impossible. Serino said it’s important to educate the engineers and systems administrators that build and interact with the infrastructure to help get security at the forefront.

That means understanding the needs of the system, building in detection systems to catch threats early and having a plan to respond appropriately when an attack happens.

“If security isn’t integrated into that process,” he said, “it’s going to be hard to get ahead of this.”

But conversations about how to do that are still sorely lacking at local, state and federal levels, Kennedy said. Plugging the cybersecurity holes plaguing critical infrastructure will take a combination of federal guidance, design innovation and money. And things will only improve if people in power take Oldsmar as a warning.

“The general thought is, it hasn’t happened in the past, so it won’t happen today,” he said. “It’s going to happen in the future. How do we prevent it from being as bad as it can be?”

Times staff writers Tracey McManus, Barbara Behrendt, CT Bowen, Josh Solomon and Charlie Frago contributed to this report.