Critical Infrastructure Security

Hacker Breached Florida City's Water Treatment System

Officials Halted Dangerous Change in Level of Lye in System, Set by Remote Intruder
Hacker Breached Florida City's Water Treatment System
City Manager Al Braithwaite (left), Pinellas County Sheriff Bob Gualtieri (center) and Mayor Eric Seidel of Oldsmar, Florida, say a hacker infiltrated the city's water treatment network and dangerously increased the level of lye. (Photo: YouTube)

A hacker breached a Florida city's water treatment network on Friday, increasing the amount of lye that would have been added to the water to a dangerous level.

See Also: Resolving Security's Biggest Productivity Killer

But city officials in Oldsmar, Florida, say they were able to spot the intrusion and quickly reverse the setting before it took effect. Reuters reports that the intruder was able to access the water treatment network software after first gaining access to TeamViewer remote access and control software.

"Importantly, the public was never in danger," Pinellas County Sheriff Bob Gualtieri said during a Monday press conference. Oldsmar, Florida, which is about 17 miles northwest of Tampa, has a population of about 15,000.

In recent years, officials have focused increasing attention on the security of industrial control systems used to manage municipalities' electricity and water. Such systems often are connected to the internet and could pose vast public safety risks if infiltrated by hackers.

Questions will likely now be raised about how the city used and configured TeamViewer for remote access, including which access controls were in place. TeamViewer has long been an attractive target because it's designed to give administrators full, remote access to and control of systems. But any hacker able to access the software can also obtain the same level of remote control over systems.

TeamViewer spokesman Patrick Pickhan tells Information Security Media Group that his company has been tracking the hack attack report and "condemns any malicious behavior" involving its software.

"We don't have any indication that our software or platform has been compromised," Pickhan says. "TeamViewer stands ready to support relevant authorities in their investigation of the technical details, such as how the cybercriminals potentially obtained login credentials, which are set and encrypted solely on the device."

Tracing the Hacker

Did the hacker leave traces that digital forensic responders can track? Ideally, anyone who logged into the city's systems would have first been securely authenticated via multifactor authentication, says Gil Kirkpatrick, chief architect at the cybersecurity company Semperis.

"That process would identify the person remotely accessing the system and their identity would show up in the authentication and access logs," Kirkpatrick says. On the other hand, the system may only have been protected by a username and password, which attackers may have been able to brute-force crack. Or attackers, in theory, might have stolen a working username and password from a city employee, or even shared credentials. "If they had some sort of shared admin account that 'everyone' used, that's a cardinal sin from a security point of view."

TeamViewer strengthened its security controls five years ago following a wave of account takeovers involving cybercriminals repurposing login credentials stolen from other breaches. Features added to TeamViewer included alerts whenever someone logs in from an unknown device, as well as more vigilant monitoring of the location of login attempts (see: TeamViewer Bolsters Security After Account Takeovers).

Hacker Increased Lye Concentration

Gualtieri says the hack of Oldsmar's systems is under investigation, and the FBI and U.S. Secret Service have been notified. Officials don't know if the attacker was located inside or outside the U.S.

At around 8 a.m. on Friday, a plant operator noticed that someone was remotely accessing the computer system he was monitoring because the mouse pointer was moving, Gualtieri says. The system controls the chemicals that are added to the water to make it safe to drink.

"That remote access was brief, and the operator didn't think much of it because his supervisor and others will remotely access his computer screen to monitor the system at various times," Gualtieri says.

A satellite view of Oldsmar's water reclamation facility (Photo: Google Maps)

Around 1:30 p.m., the operator noticed someone again in the system. This time, the intruder opened up software that controls the water being treated in the plant. The hacker then increased the level of lye, aka sodium hydroxide, from 100 parts per million to 11,100 parts per million.

Lye is used to reduce the acidity of water to make it more alkaline. But too much sodium hydroxide can be deadly. For example, in higher concentrations, it's used in drain cleaner to dissolve organic matter.

"This is obviously a significant and potentially dangerous increase," Gualtieri says.

The hacker was in the system for roughly three to five minutes, Gualtieri says. Officials immediately took steps to restrict remote access.

Oldsmar Mayor Eric Seidel says there are controls in place that would have prevented the tainted water from leaving the plant. Notably, the change to the lye levels would not have affected the water for 24 to 36 hours, and the system has alarms that would have been triggered by the increased lye levels.

"Even if we hadn't noticed it right away, it would have alarmed to all of our people," says City Manager Al Braithwaite.

Braithwaite was asked if the city was prepared for such an event. "I think we anticipated that this day was coming," Braithwaite said. "We talk about it, we think about it, we study it."

Focus: Industrial Control System Security

U.S. officials have been particularly alert for foreign hacker interest in industrial control systems, which if inappropriately adjusted or crashed could result in serious, real-world disruption or damage.

In July 2020, the Cybersecurity and Infrastructure Security Agency, which falls under the Department of Homeland Security, warned that attackers are increasingly looking for vulnerabilities in internet-accessible operational technology. Russia, for example, has been known to gain access to systems, potentially prepositioning for disruption in the event of a bigger geopolitical event (see: 6 Takeaways: Russian Spies Accused of Destructive Hacking ).

CISA has a role to play in protecting water supplies, says Tom Kellermann, who's head of cybersecurity strategy for VMware and a member of the Cyber Investigations Advisory Board for the U.S. Secret Service.

"CISA must proactively conduct threat hunting across water facilities particularly as both the nation-state and domestic terrorist threat increases," he says.

Similar operational technology hacking incidents have happened before in the U.S. In 2016, the Department of Justice blamed Iranians for August and September 2013 breaches of the supervisory control and data acquisition system of the Bowman Dam in Rye, New York.


About the Author

Jeremy Kirk

Jeremy Kirk

Executive Editor, Security and Technology, ISMG

Kirk was executive editor for security and technology for Information Security Media Group. Reporting from Sydney, Australia, he created "The Ransomware Files" podcast, which tells the harrowing stories of IT pros who have fought back against ransomware.




Around the Network

Our website uses cookies. Cookies enable us to provide the best experience possible and help us understand how visitors use our website. By browsing careersinfosecurity.com, you agree to our use of cookies.