City Water Hacked? Don’t Call 911

911 went down yesterday for two hours statewide. The state was in a panic. I received many messages via IAR (I Am Responding app for first responders). There were local numbers that nobody knew. My wife would have become dispatch if it went on any longer. Everyone calls her when they hear sirens.

You would have thought that would have made national news. (Or at least a little more coverage than that little boy Dylan Mulvaney breaking a nail). I knew instantly when it happened. I monitored national and local news. Nothing. I couldn’t even find a Boston channel with ‘Breaking Coverage’.

Now the response is, “Nothing to see here, move along.” It’s not hacking. Okay, I don’t believe you. In fact, my instinct tells me that whatever the media says, it’s usually the opposite.

When I started my environmental controls company years ago, we set up communications for many towns in the water/wastewater industry. We now have well over a hundred cities and towns in the northeast. From (redacted) Maine to (redacted) New Jersey, as far west as (not redacted because I forgot who they were) Ohio. We had bid on the entire Island of Guam but lost the bid to a Japanese company. So much for buying American.

I program water plants and integrate all the buildings in that given community, usually pump stations and wells. My controllers that run everything are PLCs. (Programmable Logic Controllers.) They make all the pumps run, feed the chemicals, and call out alarms.

We are always preparing for hackers. There are many variations of city-wide locations that need to communicate with each other, but for the sake of argument, I will just use a well, tank, and treatment plant.

The plant will talk to the tank to get the level. If it’s too low it will tell the well it needs water, and off you go. For many years all the towns used radio. I would send or request data to the radio and it would take care of the rest. It is very slow but once you buy the frequency from the FCC it’s yours.

To hack this you would need to be physically in the community. Each radio is set up to only talk to its counterpart and no one else. I have never heard of this setup ever being hacked. Sure, you could listen in but, believe me, the data will be really boring.

What people are changing to now is cellular. It is extremely fast, however, there is a subscription fee, and you are relying on an outside company. Not to mention you are exposed to the entire world. A good configuration will only allow one IP (your address) to talk to another IP, and it needs to be very specific data.

I have had only one system that was hacked. They re-programmed the operator interface with a threatening message and gave instructions on a ransom payment. I told them to unplug the ethernet and run the station manually until I got there. They were in a complete panic, trying to get the funds together to pay the ransom. I laughed, gave them a metaphorical pat on the head, and just downloaded the programs again.

Here is where the idiot IT departments come into play. I have a method for setting up cellular radios. They insisted it be done their way. All they did was get a static IP address that was limited to the Verizon network. So, they limited hackers from six billion people to a few million. Wow, good move guys. I sent a CYA email and let it happen.

Arrogant IT people think they are smarter than everyone. They have no idea how a PLC operates. They think that because it’s on ethernet, it has a Microsoft operating system. So many times they would tell me they needed to do an antivirus scan or check for software updates. The foolish ignorance goes all the way to the Biden administration…

'Some of the fixes are straightforward,' McCabe said. 'Water providers, for example, shouldn't use default passwords. They need to develop a risk assessment plan that addresses cybersecurity and set up backup systems. The EPA says they will train water utilities that need help for free.'

A PLC does not use passwords for sharing data, and passwords are rarely set to access the program. The network itself has all the restrictions. All the messaging is limited between IP addresses. It cannot be hacked without very restricted access. Meaning you have to be physically plugged into the network. It needs to be behind this ‘walled garden,’ because once there you are considered safe. The computers exchange information quickly and have to make city-wide decisions almost instantaneously. There isn’t time to check credentials on every message.

The operator interface (SCADA) McCabe is referring to is a desktop computer with software to talk to my controllers. Yes, they need passwords like everyone else in the world. She has no clue of the real vulnerabilities. I highly doubt she is being sly with her press release. She sounds like a mind-numbing IT specialist, which I am sure her speech originated from.

It is a real risk. It’s a real risk because of arrogant people who don’t set up communications properly. I don’t blame the small town with one water guy that doesn’t have a clue. You can’t be an expert in everything. And the bureaucracy in charge will not follow a quarter of the information here to protect themselves.

My story could be far more interesting, but I didn’t even hint at any issue that is not common knowledge. You would think if it’s a big deal they would at least consult a company like mine along with my competitors.

I’m not sure of the hierarchy here. I answer to the DEP and the municipality I am working for. From my experience, the DCR owns the DEP. They own all the land and the water. Where does the EPA come in? I have never had any interaction with that agency.

Hey McFly, all you have to do is invite the top five integrators to a roundtable meeting on this. Us five do the controls for over 60% of New England. But, that would mean you actually have to do something about it instead of a CYA soundbite.

Everyone has the hardware to have a protected system. It is not a cost issue. It should never happen. When it does you can blame McCabe and the EPA for not telling them what they need. Also, maybe the single town operator for not being prepared for the unexpected. I’m not worried about my systems because it’s not going to happen unless you are actually in one of the buildings and plugged in.

Sleep well. You’re on your own.

I wrote this two months ago based on this Reuters article on Drudge. The 911 incident that wasn’t hacking reminded me to post it.