City Water Hacked? Don’t Call 911

 

911 went down yesterday for two hours statewide. The state was in a panic. I received many messages via IAR (I Am Responding app for first responders). There were local numbers that nobody knew. My wife would have become dispatch if it went on any longer. Everyone calls her when they hear sirens.

You would have thought that would have made national news. (Or at least a little more coverage than that little boy Dylan Mulvaney breaking a nail). I knew instantly when it happened. I monitored national and local news. Nothing. I couldn’t even find a Boston channel with ‘Breaking Coverage’.

Now the response is, “Nothing to see here, move along.” It’s not hacking. Okay, I don’t believe you. In fact, my instinct tells me that whatever the media says, it’s usually the opposite.

When I started my environmental controls company years ago, we set up communications for many towns in the water/wastewater industry.   We now have well over a hundred cities and towns in the northeast. From (redacted) Maine to (redacted) New Jersey, as far west as (not redacted because I forgot who they were) Ohio. We had bid on the entire Island of Guam but lost the bid to a Japanese company. So much for buying American.

I program water plants and integrate all the buildings in that given community, usually pump stations and wells. My controllers that run everything are PLCs. (Programmable Logic Controllers.) They make all the pumps run, feed the chemicals, and call out alarms.

We are always preparing for hackers. There are many variations of city-wide locations that need to communicate with each other, but for the sake of argument, I will just use a well, tank, and treatment plant.

The plant will talk to the tank to get the level. If it’s too low it will tell the well it needs water, and off you go. For many years all the towns used radio. I would send or request data to the radio and it would take care of the rest. It is very slow but once you buy the frequency from the FCC it’s yours.

To hack this you would need to be physically in the community. Each radio is set up to only talk to its counterpart and no one else. I have never heard of this setup ever being hacked. Sure, you could listen in but, believe me, the data will be really boring.

What people are changing to now is cellular. It is extremely fast, however, there is a subscription fee, and you are relying on an outside company. Not to mention you are exposed to the entire world. A good configuration will only allow one IP (your address) to talk to another IP, and it needs to be very specific data.

I have had only one system that was hacked. They re-programmed the operator interface with a threatening message and gave instructions on a ransom payment. I told them to unplug the ethernet and run the station manually until I got there. They were in a complete panic, trying to get the funds together to pay the ransom. I laughed, gave them a metaphorical pat on the head, and just downloaded the programs again.

Here is where the idiot IT departments come into play. I have a method for setting up cellular radios. They insisted it be done their way. All they did was get a static IP address that was limited to the Verizon network. So, they limited hackers from six billion people to a few million. Wow, good move guys. I sent a CYA email and let it happen.

Arrogant IT people think they are smarter than everyone. They have no idea how a PLC operates. They think that because it’s on ethernet, it has a Microsoft operating system. So many times they would tell me they needed to do an antivirus scan or check for software updates. The foolish ignorance goes all the way to the Biden administration…

 

‘Some of the fixes are straightforward,’ McCabe said. ‘Water providers, for example, shouldn’t use default passwords. They need to develop a risk assessment plan that addresses cybersecurity and set up backup systems. The EPA says they will train water utilities that need help for free.’

A PLC does not use passwords for sharing data, and passwords are rarely set to access the program. The network itself has all the restrictions. All the messaging is limited between IP addresses. It cannot be hacked without very restricted access. Meaning you have to be physically plugged into the network. It needs to be behind this ‘walled garden,’ because once there you are considered safe. The computers exchange information quickly and have to make city-wide decisions almost instantaneously. There isn’t time to check credentials on every message.

The operator interface (SCADA) McCabe is referring to is a desktop computer with software to talk to my controllers. Yes, they need passwords like everyone else in the world. She has no clue of the real vulnerabilities. I highly doubt she is being sly with her press release. She sounds like a mind-numbing IT specialist, which I am sure her speech originated from.

It is a real risk. It’s a real risk because of arrogant people who don’t set up communications properly. I don’t blame the small town with one water guy that doesn’t have a clue. You can’t be an expert in everything. And the bureaucracy in charge will not follow a quarter of the information here to protect themselves.

My story could be far more interesting, but I didn’t even hint at any issue that is not common knowledge. You would think if it’s a big deal they would at least consult a company like mine along with my competitors.

I’m not sure of the hierarchy here. I answer to the DEP and the municipality I am working for. From my experience, the DCR owns the DEP. They own all the land and the water. Where does the EPA come in? I have never had any interaction with that agency.

Hey McFly, all you have to do is invite the top five integrators to a roundtable meeting on this. Us five do the controls for over 60% of New England. But, that would mean you actually have to do something about it instead of a CYA soundbite.

Everyone has the hardware to have a protected system. It is not a cost issue. It should never happen. When it does you can blame McCabe and the EPA for not telling them what they need. Also, maybe the single town operator for not being prepared for the unexpected. I’m not worried about my systems because it’s not going to happen unless you are actually in one of the buildings and plugged in.

Sleep well. You’re on your own.

I wrote this two months ago based on this Reuters article on Drudge. The 911 incident that wasn’t hacking reminded me to post it.

Published in General
This post was promoted to the Main Feed at the recommendation of Ricochet members. Like this post? Want to comment? Join Ricochet’s community of conservatives and be part of the conversation. Join Ricochet for Free.

There are 57 comments.

Become a member to join the conversation. Or sign in if you're already a member.
  1. Jimmy Carter Member
    Jimmy Carter
    @JimmyCarter

    I’m in the car business.

    Search CDK Hack.

    It has just got hacked and brought over 15,000 dealerships to a halt.

    We’re talking customers info in car sales including: names, addresses, phone numbers, banking info on loans, insurance, driver’s license info… You name it.

    And there ain’t no estimate to restore it.

    • #1
  2. David Foster Member
    David Foster
    @DavidFoster

    Is there still an option to get a simple data line from the telco for purposes like communicating the water level to a controller?  Still potentially hackable, but requires not only being in the area but also achieving physical access.

    • #2
  3. Chowderhead Coolidge
    Chowderhead
    @Podunk

    David Foster (View Comment):

    Is there still an option to get a simple data line from the telco for purposes like communicating the water level to a controller? Still potentially hackable, but requires not only being in the area but also achieving physical access.

    Absolutely. It’s called a dedicated line. Whenever there’s a problem these tele techs have no idea that you actually have a twisted pair of copper between here and there. They still need to support it.

    It’s old tech that needs to go away.

    • #3
  4. Chowderhead Coolidge
    Chowderhead
    @Podunk

    Jimmy Carter (View Comment):

    I’m in the car business.

    Search CDK Hack.

    It has just got hacked and brought over 15,000 dealerships to a halt.

    We’re talking customers info in car sales including: names, addresses, phone numbers, banking info on loans, insurance, driver’s license info… You name it.

    And there ain’t no estimate to restore it.

    Let me guess. MSM made the story go away.

    • #4
  5. kedavis Coolidge
    kedavis
    @kedavis

    Chowderhead (View Comment):

    David Foster (View Comment):

    Is there still an option to get a simple data line from the telco for purposes like communicating the water level to a controller? Still potentially hackable, but requires not only being in the area but also achieving physical access.

    Absolutely. It’s called a dedicated line. Whenever there’s a problem these tele techs have no idea that you actually have a twisted pair of copper between here and there. They still need to support it.

    It’s old tech that needs to go away.

    If you mean regular copper-wire phone service needs to go away, I disagree vehemently.  Everything else requires power at the site, whereas POTS (Plain Old Telephone Service) is power supplied by the telco.  That means it works if you need to call 911 even with a power outage, etc.  As long as you’re not stupid and only have cordless handsets, etc.

    • #5
  6. Brickhouse Hank Contributor
    Brickhouse Hank
    @HankRhody

    Chowderhead:

    To hack this you would need to be physically in the community. Each radio is set up to only talk to its counterpart and no one else. I have never heard of this setup ever being hacked. Sure, you could listen in but believe me, the data will be really boring.

    Fear the man in the middle.

    But yeah, basically any hack that requires the hacker to leave his keyboard and touch grass is going to be much less of a security concern.

    • #6
  7. Chris B Member
    Chris B
    @ChrisB

    In defense of IT, a LOT of people installing PLCs haven’t the foggiest idea of Network Security, and SCADA can be a serious vulnerability in the network (Stuxnet being the most famous example).

    I can’t tell you how many times I’ve had HVAC techs, manufacturing equipment vendors, security door control and surveillance installers, and phone system vendors tell me I had to do insane things with the network because they didn’t know how to do network security properly. It’s always best to start from a position of respect, knowing the limitations of your own knowledge domain, and asking questions when something doesn’t make sense.

    It’s also important to know when the other guy is just plain wrong and stick to your guns. 

    • #7
  8. Clavius Thatcher
    Clavius
    @Clavius

    This coming November 24th will be the 10 anniversary of the attack by North Korea on Sony Pictures.  That pretty much wiped out the entire server, network, and application infrastructure.  We survived, but it was a slog.

    You can’t rely on “being on the inside” being ok if you are in any way connected to the outside.  Secure authentication is key or a nation-state level actor can get inside the perimeter and wreak havoc.  It needn’t be burdensome.  As you point out the connections are IP address limited.  But adding an initial handshake and a key exchange for each re-contact should not be a great burden. 

    The castle walls may be high and strong, but don’t you want a password inside them?

    • #8
  9. kedavis Coolidge
    kedavis
    @kedavis

    You know what was hard to hack?

    A PDP-8.

    They did a LOT of process control.

     

    • #9
  10. Brickhouse Hank Contributor
    Brickhouse Hank
    @HankRhody

    Clavius (View Comment):

    As you point out the connections are IP address limited.  But adding an initial handshake and a key exchange for each re-contact should not be a great burden. 

    The castle walls may be high and strong, but don’t you want a password inside them?

    It can be difficult for industrial machines. Let’s say you set up a pump for continuous monitoring from a central location. That pump’s going to have a couple different characteristics you’re going to track; pressure, flow, conductivity of the water (a quick and easy check for contamination), temperature, possibly more. Comparatively speaking the handshake and key exchange is going to transfer a great deal more information than the data you’re actually interested in. If you get an update every thirty seconds for every pump in the area it can add up to a strain on the network.

    Then again perhaps the extra copper in the network is necessary. But given that the entire IT world seems to be in “I don’t have to outrun the bear, I just have to outrun you” mode maybe not.

    • #10
  11. Clavius Thatcher
    Clavius
    @Clavius

    Brickhouse Hank (View Comment):

    Clavius (View Comment):

    As you point out the connections are IP address limited. But adding an initial handshake and a key exchange for each re-contact should not be a great burden.

    The castle walls may be high and strong, but don’t you want a password inside them?

    It can be difficult for industrial machines. Let’s say you set up a pump for continuous monitoring from a central location. That pump’s going to have a couple different characteristics you’re going to track; pressure, flow, conductivity of the water (a quick and easy check for contamination), temperature, possibly more. Comparatively speaking the handshake and key exchange is going to transfer a great deal more information than the data you’re actually interested in. If you get an update every thirty seconds for every pump in the area it can add up to a strain on the network.

    Then again perhaps the extra copper in the network is necessary. But given that the entire IT world seems to be in “I don’t have to outrun the bear, I just have to outrun you” mode maybe not.

    Yes, it struck me after writing this comment that there could be issues.  Authentication failure adds another failure mode, perhaps more likely than an attack.  And your point is relevant too.

    So, perhaps only require some sort of authentication or certificate transfer for action commands as opposed to data gathering?  Transmit all action commands in Klingon?  I don’t know.  

    • #11
  12. kedavis Coolidge
    kedavis
    @kedavis

    Clavius (View Comment):

    Brickhouse Hank (View Comment):

    Clavius (View Comment):

    As you point out the connections are IP address limited. But adding an initial handshake and a key exchange for each re-contact should not be a great burden.

    The castle walls may be high and strong, but don’t you want a password inside them?

    It can be difficult for industrial machines. Let’s say you set up a pump for continuous monitoring from a central location. That pump’s going to have a couple different characteristics you’re going to track; pressure, flow, conductivity of the water (a quick and easy check for contamination), temperature, possibly more. Comparatively speaking the handshake and key exchange is going to transfer a great deal more information than the data you’re actually interested in. If you get an update every thirty seconds for every pump in the area it can add up to a strain on the network.

    Then again perhaps the extra copper in the network is necessary. But given that the entire IT world seems to be in “I don’t have to outrun the bear, I just have to outrun you” mode maybe not.

    Yes, it struck me after writing this comment that there could be issues. Authentication failure adds another failure mode, perhaps more likely than an attack. And your point is relevant too.

    So, perhaps only require some sort of authentication or certificate transfer for action commands as opposed to data gathering? Transmit all action commands in Klingon? I don’t know.

    How about more on-site monitoring and control, which means little or no networking etc?

    • #12
  13. Chowderhead Coolidge
    Chowderhead
    @Podunk

    kedavis (View Comment):

    You know what was hard to hack?

    A PDP-8.

    They did a LOT of process control.

    What the heck is that? I have no idea what a PDP-8 is. Why does it only have 11 bits? Where are the other 5? That’s terrifying! Now if you had a PDP-11 I would understand. I worked on the computer belt (RT 128 in MA) at Prime Computer back in the day (actually both were on RT 495). It was right down the road from DEC. I dissected one and re-assembled it a dozen times for the engineers.

    Like a Prime machine, to get it to boot you needed to set the bits then the momentary switch to the right said ‘load accumulator’. Do a couple dozen words and it would load the bootstrap program off of a floppy disk. Good times. I almost forgot.

    • #13
  14. Chowderhead Coolidge
    Chowderhead
    @Podunk

    kedavis (View Comment):

    Chowderhead (View Comment):

    David Foster (View Comment):

    Is there still an option to get a simple data line from the telco for purposes like communicating the water level to a controller? Still potentially hackable, but requires not only being in the area but also achieving physical access.

    Absolutely. It’s called a dedicated line. Whenever there’s a problem these tele techs have no idea that you actually have a twisted pair of copper between here and there. They still need to support it.

    It’s old tech that needs to go away.

    If you mean regular copper-wire phone service needs to go away, I disagree vehemently. Everything else requires power at the site, whereas POTS (Plain Old Telephone Service) is power supplied by the telco. That means it works if you need to call 911 even with a power outage, etc. As long as you’re not stupid and only have cordless handsets, etc.

    It could stay, or go, I don’t really care. The dedicated line I’m talking about is a hardwired copper pair that never hits a switch or a relay. It’s one solid pair from start to finish. It is primarily used to send a tank level or a pump run command. Whenever the phone company works on a line they usually don’t know what it is. They disconnect it. The town calls me. I go there and tell them to call Verizon. Round and round it goes.

    • #14
  15. Chowderhead Coolidge
    Chowderhead
    @Podunk

    Clavius (View Comment):

    Brickhouse Hank (View Comment):

    Clavius (View Comment):

    As you point out the connections are IP address limited. But adding an initial handshake and a key exchange for each re-contact should not be a great burden.

    The castle walls may be high and strong, but don’t you want a password inside them?

    It can be difficult for industrial machines. Let’s say you set up a pump for continuous monitoring from a central location. That pump’s going to have a couple different characteristics you’re going to track; pressure, flow, conductivity of the water (a quick and easy check for contamination), temperature, possibly more. Comparatively speaking the handshake and key exchange is going to transfer a great deal more information than the data you’re actually interested in. If you get an update every thirty seconds for every pump in the area it can add up to a strain on the network.

    Then again perhaps the extra copper in the network is necessary. But given that the entire IT world seems to be in “I don’t have to outrun the bear, I just have to outrun you” mode maybe not.

    Yes, it struck me after writing this comment that there could be issues. Authentication failure adds another failure mode, perhaps more likely than an attack. And your point is relevant too.

    So, perhaps only require some sort of authentication or certificate transfer for action commands as opposed to data gathering? Transmit all action commands in Klingon? I don’t know.

    As an integrator it’s very difficult to do anything outside of the software’s capabilities. We basically read or write a data table to another PLC at a specific IP on a timetable I choose. I have towns that take five minutes for a round trip polling of all the stations. I do rely on IT to keep people off my network, but they need to stay out of my face. 

    This is how the Stuxnet virus infected the Siemens PLC’s. (Someone) got past the network firewall. 

    • #15
  16. Percival Thatcher
    Percival
    @Percival

    Chowderhead (View Comment):

    kedavis (View Comment):

    You know what was hard to hack?

    A PDP-8.

    They did a LOT of process control.

     

    What the heck is that?

    Dude, that is the heart and soul – the center of the universe. That is the battlefield upon which the cognoscenti regularly put the hackers* in their place. That’s the control panel. You can do it all from there! Does the code need a patch? Toggle it in!


    * In the original formulation, hacker was to programmer as duffer was to golfer.

    • #16
  17. Stad Coolidge
    Stad
    @Stad

    First on the list of Chinas’s plan for invading Taiwan – distract the US:

    1.  Knock out as much of the energy infrastructure as possible, via hacking of power sources and physical destruction of power substations.
    2.  Knock out 911 service in as many major cities as possible.
    3.  Create as much civil unrest as possible.  This includes murdering innocent blacks and quickly spreading lies on the Internet about white cops killing massive numbers of blacks.  Once the riots start, continue fanning the flames.
    4.  Shut down as many public services as possible via hacking or physical attacks – water and sewer.
    5.  Destroy as many food production facilities as possible.  Ditto with fuel storage and refineries.
    6.  Do all of this within as short a time frame as possible, starting at 8 PM Eastern and only finishing when agents inside the US are captured or killed, and hacking activities thwarted.
    7.  Last act before being shut out of the Internet?  Shut down the Internet.

    Next on the list?  Invade Taiwan . . .

    • #17
  18. kidCoder Member
    kidCoder
    @kidCoder

    I very much enjoyed https://berthub.eu/articles/posts/cyber-security-pre-war-reality-check/ and it gives a wonderful context to this post.

    The threats are not in our PLCs, but the people who run them, and the people who insist on ever-growing amounts of complexity to manage the networks.

    • #18
  19. kidCoder Member
    kidCoder
    @kidCoder

    Chowderhead (View Comment):

    kedavis (View Comment):

    You know what was hard to hack?

    A PDP-8.

    They did a LOT of process control.

    What the heck is that? I have no idea what a PDP-8 is. Why does it only have 11 bits? Where are the other 5? That’s terrifying! Now if you had a PDP-11 I would understand. I worked on the computer belt (RT 128 in MA) at Prime Computer back in the day (actually both were on RT 495). It was right down the road from DEC. I dissected one and re-assembled it a dozen times for the engineers.

    Like a Prime machine, to get it to boot you needed to set the bits then the momentary switch to the right said ‘load accumulator’. Do a couple dozen words and it would load the bootstrap program off of a floppy disk. Good times. I almost forgot.

    12 bits, 0 indexed. It was a 12 bit minicomputer, before the days when we standardized on powers of 2.

    • #19
  20. Front Seat Cat Member
    Front Seat Cat
    @FrontSeatCat

    Did this have anything to do with the 911 outage during the heat wave last few days up there? I saw it on the national news (unless you were referring to something you posted before)?  

    • #20
  21. Chowderhead Coolidge
    Chowderhead
    @Podunk

    Front Seat Cat (View Comment):

    Did this have anything to do with the 911 outage during the heat wave last few days up there? I saw it on the national news (unless you were referring to something you posted before)?

    That is the outage I was referring to. The claim now is it’s a firewall issue. That’s one bit IT oops if true. I hope nobody died because of it. I was specifically watching the news to see how they were going to cover it. I wasn’t looking to gain any information. 

    • #21
  22. Chowderhead Coolidge
    Chowderhead
    @Podunk

    kidCoder (View Comment):

    Chowderhead (View Comment):

    kedavis (View Comment):

    You know what was hard to hack?

    A PDP-8.

    They did a LOT of process control.

    What the heck is that? I have no idea what a PDP-8 is. Why does it only have 11 bits? Where are the other 5? That’s terrifying! Now if you had a PDP-11 I would understand. I worked on the computer belt (RT 128 in MA) at Prime Computer back in the day (actually both were on RT 495). It was right down the road from DEC. I dissected one and re-assembled it a dozen times for the engineers.

    Like a Prime machine, to get it to boot you needed to set the bits then the momentary switch to the right said ‘load accumulator’. Do a couple dozen words and it would load the bootstrap program off of a floppy disk. Good times. I almost forgot.

    12 bits, 0 indexed. It was a 12 bit minicomputer, before the days when we standardized on powers of 2.

    Pardon my faux pas. I forgot the leading zero. I was stunned by the short word. That explains why the cheaper PLC analogs only have a resolution of 4095 rather than the standard 32767(plus sign). It’s an ancient carryover. 

    • #22
  23. kedavis Coolidge
    kedavis
    @kedavis

    Chowderhead (View Comment):

    kedavis (View Comment):

    You know what was hard to hack?

    A PDP-8.

    They did a LOT of process control.

    What the heck is that? I have no idea what a PDP-8 is. Why does it only have 11 bits? Where are the other 5? That’s terrifying! Now if you had a PDP-11 I would understand. I worked on the computer belt (RT 128 in MA) at Prime Computer back in the day (actually both were on RT 495). It was right down the road from DEC. I dissected one and re-assembled it a dozen times for the engineers.

    I expect you know, PDP-8 (actually 12-bit) came before -11, starting in 1965.  12-bits among other things, allowed for storing 2 6-bit reduced-ASCII characters (uppercase only) in a single word.  And nobody should get all uppity or whatever about 16 bits, the PDP-1 in 1959 was 18-bit, and the PDP-5, predecessor to the -8, was also 12-bit in 1963.  UNIX began in 1969 on an 18-bit PDP-7.  Later versions were the PDP-9, and -15, a great machine.  The PDP-6 leading to the -10 were, of course 36-bit.  And nobody bitched that they should have been 16-bit.

     

    Like a Prime machine, to get it to boot you needed to set the bits then the momentary switch to the right said ‘load accumulator’. Do a couple dozen words and it would load the bootstrap program off of a floppy disk. Good times. I almost forgot.

    PDP-8 did that too, and many systems had a sticker with the loader instructions on the front panel (as I recall the number of instructions was rather small, like 12 or so); although later versions had a ROM-based boot loader available.  The PDP-12, on the other hand, could be booted from DECtape/LINCtape with a single instruction from the front panel.  The PDP-12 was gorgeous, I used to have several when I was working on starting a computer school.

     

    MINOLTA DIGITAL CAMERA

    • #23
  24. kedavis Coolidge
    kedavis
    @kedavis

    Chowderhead (View Comment):

    kedavis (View Comment):

    Chowderhead (View Comment):

    David Foster (View Comment):

    Is there still an option to get a simple data line from the telco for purposes like communicating the water level to a controller? Still potentially hackable, but requires not only being in the area but also achieving physical access.

    Absolutely. It’s called a dedicated line. Whenever there’s a problem these tele techs have no idea that you actually have a twisted pair of copper between here and there. They still need to support it.

    It’s old tech that needs to go away.

    If you mean regular copper-wire phone service needs to go away, I disagree vehemently. Everything else requires power at the site, whereas POTS (Plain Old Telephone Service) is power supplied by the telco. That means it works if you need to call 911 even with a power outage, etc. As long as you’re not stupid and only have cordless handsets, etc.

    It could stay, or go, I don’t really care. The dedicated line I’m talking about is a hardwired copper pair that never hits a switch or a relay. It’s one solid pair from start to finish. It is primarily used to send a tank level or a pump run command. Whenever the phone company works on a line they usually don’t know what it is. They disconnect it. The town calls me. I go there and tell them to call Verizon. Round and round it goes.

    A lot of the newer telco people have no idea what anything is other than fiber optic.  My first computer job 1979-81 included a leased line from Oregon to a branch office in Colorado.  Gandalf 9600 bps* long-haul modems were new. But we didn’t have those kinds of problems because the telco people in those days still knew what’s what.

    A regular phone service can be invaluable for families with children, especially.  Being able to pick up the phone even during a power outage and they know where you are for an ambulance or whatever, your phone doesn’t have to be charged and the cell towers operating, etc.

     

    * people who don’t know, might say “baud,” but you and I know – or at least, I know – that “baud” and bps are not the same.

    • #24
  25. kedavis Coolidge
    kedavis
    @kedavis

    And the PDP-12, behind the green panel on the left, included multiple Analog-To-Digital and Digital-To-Analog channels, plus 6 SPDT relay connections whose status was constantly displayed on the front panel.

    • #25
  26. Chowderhead Coolidge
    Chowderhead
    @Podunk

    kedavis (View Comment):

    And the PDP-12, behind the green panel on the left, included multiple Analog-To-Digital and Digital-To-Analog channels, plus 6 SPDT relay connections whose status was constantly displayed on the front panel.

    Do you heat your house with that thing? Will it still boot? Very cool. At least they were easy to fix. No surface mount chips in that. Before I transferred to CPU development at Prime I would have to fix the boards that got returned. I wouldn’t even use schematics. I put two test stands side-by-side with a good and bad card running. I would just see what the bad card wasn’t doing that the good card was. Most of the time it was a dead chip. Sometimes it was an internal layer. The boards were nine layers thick.

    ———–

    I still have a telephone modem that’s functioning. It’s my only access to a certain customer. I need to use MagicJack to fire it up or find someone with a landline. They are all VOIP now anyway.

     

    • #26
  27. kedavis Coolidge
    kedavis
    @kedavis

    Chowderhead (View Comment):

    kedavis (View Comment):

    And the PDP-12, behind the green panel on the left, included multiple Analog-To-Digital and Digital-To-Analog channels, plus 6 SPDT relay connections whose status was constantly displayed on the front panel.

    Do you heat your house with that thing? Will it still boot? Very cool. At least they were easy to fix. No surface mount chips in that. Before I transferred to CPU development at Prime I would have to fix the boards that got returned. I wouldn’t even use schematics. I put two test stands side-by-side with a good and bad card running. I would just see what the bad card wasn’t doing that the good card was. Most of the time it was a dead chip. Sometimes it was an internal layer. The boards were nine layers thick.

    That was back in the early to mid 80s, but in 1988 I had my first bout of then-undiagnosed ulcerative colitis, which returned in 1989 and became permanent.  I was unable to work, and lost everything I had in storage including the 12s, most of an original PDP-10 (KA model, everything but the front panel which someone had kiped to hang on a wall somewhere), an SMP KI pair from Intel in Hillsboro, a 2065 system from Columbia University (cost me a $1000 cashier’s check to the Navy Research Department which originally funded it), and lots more.

     

    ———–

    I still have a telephone modem that’s functioning. It’s my only access to a certain customer. I need to use MagicJack to fire it up or find someone with a landline. They are all VOIP now anyway.

     

    I have a bit of a “vintage modem collection” actually.  Including a Cray Systems one, somewhere.  And in the past I helped out a few people who were alarm system installers/techs, many of the alarm systems they dealt with remotely were designed to only work with actual Hayes modems that had certain manufactured response codes etc.  Fortunately I had a few real Hayes modems for them.

    • #27
  28. Chowderhead Coolidge
    Chowderhead
    @Podunk

    kedavis (View Comment):
    I was unable to work, and lost everything I had in storage including the 12s, most of an original PDP-10

    I’m very sorry to hear that. The pic looked new because it looks like an HP laser printer in the background. 

    Last year I upgraded a water plant. The old controller was a drum controller. For those that don’t know it’s a cylindrical gold plated tube that rotated slowly with a motor. Much like a player piano. There were a series of contacts running the length. You would put in pegs when you wanted the contact on. 

    I tried really hard to get my hands on it. I was going to get glass jeweled indicator lights and hang it on my office wall. I can’t even even find a pic of one on the internet. 

    • #28
  29. kedavis Coolidge
    kedavis
    @kedavis

    Chowderhead (View Comment):

    kedavis (View Comment):
    I was unable to work, and lost everything I had in storage including the 12s, most of an original PDP-10

    I’m very sorry to hear that. The pic looked new because it looks like an HP laser printer in the background.

    Last year I upgraded a water plant. The old controller was a drum controller. For those that don’t know it’s a cylindrical gold plated tube that rotated slowly with a motor. Much like a player piano. There were a series of contacts running the length. You would put in pegs when you wanted the contact on.

    I tried really hard to get my hands on it. I was going to get glass jeweled indicator lights and hang it on my office wall. I can’t even even find a pic of one on the internet.

    The photo is very nice, it’s from a computer history museum in Sweden, as I recall.  Not sure if the museum still exists.

    There was a site with more photos of the PDP-12 “guts” but I can’t find it now, maybe it no longer exists.  Fortunately I saved the images that I found there.  Note that in the last photo, the display is not a “CRT terminal,” it’s a 12″ OSCILLOSCOPE!  All of the text etc shown there, is drawn by software.

     

    • #29
  30. Al Sparks Coolidge
    Al Sparks
    @AlSparks

    Chowderhead: Arrogant IT people think they are smarter than everyone.

    Every field has people who are arrogant.  Not everyone in a field is.  You seem to be one of those who are (arrogant).

    I work for a power company in industrial IT (also referred to as OT).  My job includes managing microwave radio communications, and communications between various PLC’s in power plants.

    I will have more to say in a subsequent post, but I thought I’d start with the above quote.

    With all your trashing of IT folks in technical fields, it seems to me you’re projecting.  I see a lot of arrogance in your OP, and frankly, you seem to think you’re the one who is smarter than everyone.

    You’re not the first technician / engineer to express disdain for my field.  I’m used to it.  Perhaps you should take a look in the mirror.

    • #30
Become a member to join the conversation. Or sign in if you're already a member.