Post-Mortem on First US Grid Cyberattack: ‘Update Your Routers’

A post-mortem has been released confirming and explaining the first official cyberattack on US electrical grid infrastructure, according to EENews. Surprisingly, it didn’t involve a squirrel.

On March 5 this year, an electrical operator on the US Western Interconnection experienced a series of communications disruptions spanning the full day, causing brief communications dropouts with facilities in southern California, Utah, and Wyoming. EENews got on that story as soon as it happened and the writer was later interviewed for an NPR Weekend Edition feature. The individual disruptions were minor, purportedly lasting only about five minutes each and not affecting power delivery. But the frequency and pattern was suspicious and a report was filed for further investigation.

The culprit? A very ordinary firmware vulnerability specific to a router model deployed in the utility’s outer communications ring. The solution? A firmware update that hadn’t been identified and installed on those units. The intruder, possibly without even knowing who they were targeting, was invoking a basic Denial of Service (DoS)  attack. When hit, any router with the vulnerable firmware version would be forced to reboot. The router model isn’t identified, but these kinds of weaknesses are increasingly routine across multiple vendors and equipment types. If something must face the Internet and defend critical infrastructure assets, then someone needs to be monitoring vendor updates and various channel publications for that market to see what’s happening on the security side.

NERC (the North American Electric Reliability Council) recently completed their review and issued a Lessons Learned memo. It says “keep your Internet exposure as low as possible and be aware of security patches, stupid.” (I quote from memory.) Since it doesn’t seem to be available on the NERC website right now, you can also read the copy that EENews has cached.

Utility-Grade Comm Equipment

Typical utility-grade fiber, copper Ethernet, Time-Synchronization, and related communications
gear in a modular/blade style format, demonstrated at a Denver, Colorado expo (2018).

The electrical grid doesn’t respond well to sudden disturbances and has correspondingly been built and maintained primarily by plodding, change-resistant entities. Into this once-slow market a wave of new networking and remote management technologies have exploded in the past 10-15 years. These are pushed in part by renewable generation developers who need to centrally manage remote, distributed assets, and also like to move fast and break things. The change has brought many large improvements, but it has also brought new risks that need a new generation of persons with industrial communications expertise to deploy and manage them.

On one hand, Internet connectivity has enabled an unprecedented level of centralized, real-time control and data collection. In the best case, that means more efficient operation and reduced outages. It is increasingly rare to find an electrical power substation or switchyard in the US that doesn’t have at least a rack or two of communications support equipment, often configured for 19″ style mounting and living in bog-standard TrippLite rolling cages along with infrastructure-type network switches.

On the other hand, my view from the inside says that IT departments within the utility industry are subject to the same pressures that affect IT in any other business. If it isn’t a revenue generator, then it’s a cost center. Cost centers tend to fall under the earliest and highest degrees of scrutiny by distant bean counters.  Eventually, we can end up with some variation of Jurassic Park Syndrome.

Fortunately, this particular attack wasn’t serious and there was no impact on the infrastructure. But, it appears that happy outcome was partly because the attack wast unsophisticated and mainly targeted specific hardware rather than specific infrastructure. The real-world possibilities can go way beyond, and it remains to be seen whether the US electrical providers can continue to keep ahead of their black-hatted competition.

Aaron Vienot

Engineer by day, hobbyist by night, occasional contributor, and full-time wise guy.

avatar
3 Comment threads
2 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
4 Comment authors
WirkopsuedonymouschuckulaKrogoth Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
Wirko
Guest
Wirko

Oh, but gerbils. They have so many megawatts of TDP under control at the tips of their claws. Yet they have not caused a single power outage in many years, not since Tech-report exists. And they will not, at leasts while Tech Report lasts. What a noble and gentle species. I wish there were more gerbils.

psuedonymous
Guest
psuedonymous

At least it was a boring my-internet-is-down peripheral infrastructure attack, rather than an exiting my-generator-has-exploded Aurora-style attack.

chuckula
Guest
chuckula

I knew I shouldn’t have trusted GeekSquad to setup my multi-gigawatt nuclear reactor complex!

#WhatsChernobyl

Krogoth
Guest
Krogoth

RMBKs have been cancelled!

#3.4NotBadAndTerrible

chuckula
Guest
chuckula

AMD: This is the first time that we won’t claim Intel is melting down because of a product cancellation!

Pin It on Pinterest

Share This