On March 5 this year, an electrical operator on the US Western Interconnection experienced a series of communications disruptions spanning the full day, causing brief communications dropouts with facilities in southern California, Utah, and Wyoming. EENews got on that story as soon as it happened and the writer was later interviewed for an NPR Weekend Edition feature. The individual disruptions were minor, purportedly lasting only about five minutes each and not affecting power delivery. But the frequency and pattern was suspicious and a report was filed for further investigation.
The culprit? A very ordinary firmware vulnerability specific to a router model deployed in the utility’s outer communications ring. The solution? A firmware update that hadn’t been identified and installed on those units. The intruder, possibly without even knowing who they were targeting, was invoking a basic Denial of Service (DoS) attack. When hit, any router with the vulnerable firmware version would be forced to reboot. The router model isn’t identified, but these kinds of weaknesses are increasingly routine across multiple vendors and equipment types. If something must face the Internet and defend critical infrastructure assets, then someone needs to be monitoring vendor updates and various channel publications for that market to see what’s happening on the security side.
NERC (the North American Electric Reliability Council) recently completed their review and issued a Lessons Learned memo. It says “keep your Internet exposure as low as possible and be aware of security patches, stupid.” (I quote from memory.) Since it doesn’t seem to be available on the NERC website right now, you can also read the copy that EENews has cached.
Typical utility-grade fiber, copper Ethernet, Time-Synchronization, and related communications
gear in a modular/blade style format, demonstrated at a Denver, Colorado expo (2018).
The electrical grid doesn’t respond well to sudden disturbances and has correspondingly been built and maintained primarily by plodding, change-resistant entities. Into this once-slow market a wave of new networking and remote management technologies have exploded in the past 10-15 years. These are pushed in part by renewable generation developers who need to centrally manage remote, distributed assets, and also like to move fast and break things. The change has brought many large improvements, but it has also brought new risks that need a new generation of persons with industrial communications expertise to deploy and manage them.
On one hand, Internet connectivity has enabled an unprecedented level of centralized, real-time control and data collection. In the best case, that means more efficient operation and reduced outages. It is increasingly rare to find an electrical power substation or switchyard in the US that doesn’t have at least a rack or two of communications support equipment, often configured for 19″ style mounting and living in bog-standard TrippLite rolling cages along with infrastructure-type network switches.
On the other hand, my view from the inside says that IT departments within the utility industry are subject to the same pressures that affect IT in any other business. If it isn’t a revenue generator, then it’s a cost center. Cost centers tend to fall under the earliest and highest degrees of scrutiny by distant bean counters. Eventually, we can end up with some variation of Jurassic Park Syndrome.
Fortunately, this particular attack wasn’t serious and there was no impact on the infrastructure. But, it appears that happy outcome was partly because the attack wast unsophisticated and mainly targeted specific hardware rather than specific infrastructure. The real-world possibilities can go way beyond, and it remains to be seen whether the US electrical providers can continue to keep ahead of their black-hatted competition.