Crash Override Chronicles: Background
The most up to date version of this post can be found on my website at: https://sroberts.io/posts/2017-10-27-building-better-security-presentations/
There’s a lot to ICS networks, the systems they run, and the protocols that control them. This is the barest treatment, and I have more to do, but I wanted to share what I have learned so far. Also it is a lot (and I mean a lot even for me) of lists, links, references, and embedded resources like videos and PDFs. You’ve been warned.
When I started this series it was really all about the things I didn’t know (and mostly still don’t). I didn’t know about the adversary. I didn’t know about the campaign against Ukrainian power beyond reading the news and some content from FOR578. More than any of those I didn’t (and still don’t to any measurable depth) know about the environment where these attacks took place; the industrial control networks that manage power generation and transmission.
ICS Security alone is a huge world that someone could (and I would argue should) spend years of their lives mastering, and it’s still just a small subset of ICS as a whole. Beyond that the security concerns and needs of the various a parts of ICS are dramatically different (IE ICS Security for Power Grid operations seems vastly different than Manufacturing Systems, both of which are different from Water Systems). In industrial control systems, just like regular networks, the true competition between adversary and defender is who knows the environment better and can leverage that to their advantage.
Generally I write about two different things: Stuff I know well and want to share knowledge about or things I don’t know and want to share the process of learning. This post is certainly just the beginning of my dive into ICS but none the less I wanted to share the process.
Research
When reviewing the CRASHOVERRIDE report it touched on a pretty wide variety of things I didn’t understand and thus wanted to gather some context on. As a defender we’re limited by how well we understand our environment and so contextualizing these things is key. I’ve added quite a few things since my previous post, rearranged a number, and basically done whatever I can to build up my meager understanding of this complex world. I will include my sources as much as I can.
Not knowing where to start digging my first stage is collecting resources. I took two approaches. First of all I went to Rob Lee’s (CEO of Dragos, the company that put out the CRASHOVERRIDE report) site where he shared a collection of ICS & ICS CND resources:
This is a really wide ranging set of resources and while all are interesting and things I should look into eventually not all of them are 100% necessary to understanding this intrusion. So for now I’m focusing on general understanding of SCADA/ICS and power transmission networks. The chemical, factory control, IoT, and waterworks stuff will have to wait for another day. Here are the most relevant videos. You can watch them now, I’ll wait:
Along with the videos I also purchased Thomas Rid’s book Rise of the Machines:
Now Rob’s site provided some great resources, but a good analyst doesn’t rely on a single source. So I did what any good analyst does when they don’t know much about something: I Googled it and read Wikipedia. Now before you wonder I don’t consider Wikipedia a definitive source for (actually it’s really good at roller coasters), but it is a useful place to identify two things: grand themes and links. Specifically I used this to dig into the various protocols and basically looked for things that commonly came up and researched those on their own.
One of the biggest ones I kept seeing was IEC. The IEC, or International Electrotechnical Commission, describes themselves as:
(…) the world’s leading organization for the preparation and publication of International Standards for all electrical, electronic and related technologies. These are known collectively as “electrotechnology”.
IEC standardizes the protocols for electric systems including management (hence why I kept bumping into them). Here’s the key set I spent some time browsing:
While digging into IEC I also came across NEMA. Turns out just like the Imperial vs Metric system the US has to do everything a little bit different (seriously though, the Metric system just makes sense) and as a result the US has it’s own industry group that sets standards for electrical systems in the United States. Given CRASHOVERRIDE was used in Ukraine the IEC standards are the relevant ones, but I bookmarked NEMA for later.
Lastly I also spent a lot of time (most of my last two Saturdays) watching videos from Jim Pytel. Jim is an instructor at Columbia Gorge Community College and teaches Electro-Mechanical Technology. His YouTube channel BigBadTech covers a wide variety of ICS related topics, such as PLCs programming, pump & motor control, and many others.
What are Industrial Control Systems?
After learning about this for the last few weeks I can’t decide if this is a very simple question or a very complicated one. In the end I think the idea is simple, but the implementation is highly complex. An industrial control system is aptly named; A combination of systems to send commands and monitor information about a physical industrial process like generating power or running manufacturing machinery. This can be as simple as a series of simple electrical switches and lights and as complicated as our next topic: SCADA.
What is SCADA?
Standing for Supervisory Control And Data Acquisition SCADA systems are the combination of software, computer hardware, protocols, and electronic/mechanical hardware for manipulating real world actions. SCADA is typically used in industrial settings such as power generation & transmission, water works management, manufacturing, or any other large scale physical processes. SCADA has a standardized arrangement of systems and components, most of which we’ll discuss a bit later (RTUs, PLCs, HMIs, etc).
What is the significance of voltage transforming?
Different electrical actions need different voltages. In particular long distance transmission (moving electricity from the generation station to a distribution point) vs use (when I plug my Ikea lamp into the wall and get the light I’m sitting next to right now). In the US power is transmitted at a 110 kV standard, where a home outlet is only 120V (aka ~1/1000th the transmission voltage). I haven’t tested it, but I’m willing to bet 110,000V would trip a home circuit breaker and then some. 💥
What is an electric grid feeder?
This was another phrase that sounded somewhat straightforward, but I wanted to be sure. Based on this document from GE on Distribution Feeder Principles the feeder part of the system is the larger part of voltage transforming. The feeder is the “last mile” in shipping terms taking power from the larger distribution systems (substations) to individual customers (homes, offices, factories, etc).
Protocols:
While going through the report I noted quite a few referenced protocols, mostly in reference to the capabilities identified in CRASHOVERRIDE. Clearly something as complicated as grid operations is going to require a lot of structured communication protocols for a lot of different functions. For now I’m just focusing on the ones references in the report. I did reorder them a bit based on their hierarchy.
IEC 61850: At the top of the the hierarchy I inadvertently discovered is standard IEC 61850. Created by IEC Technical Committee 57 (which based on all the references to it must have been a very busy group) this standard defines a wide variety of electric substation automation. Now I don’t know much yet, but that seems like a pretty huge amount of area to cover and indeed a Googling looking around confirms that.
Want a security specific reference:
The standard is divided into multiple parts. Based on the other protocols referenced our focus will be on Part 5: Transmission Protocols.
IEC 61850–5-101: I’d taken note of IEC 101 & 104 but only after reading through information about 61850 was I able to properly contextualize these protocols. Part 5 protocol 101 defines the protocols for basic telecontrol of tasks. Makes sense malware that wanted to disrupt electricity transmission would know how speak that dialect.
IEC 61850–5–104: I was ready for 104 to be an entirely different animal, but after digging into the preview of the 104 spec I learned it’s not different at all really, at least in function. 104 is just about adapting the 101 protocols for another link level transport: TCP/IP.
Distributed Network Protocol (DNP) 3: This one I recorded even though it wasn’t implemented in the actual CRASHOVERRIDE malware samples Drgos or ESET studied. I thought it was interesting since it’s the North American standard and the modular nature of CRASHOVERRIDE means there could be a DNP3 protocol module (though there’s no evidence of that yet). DNP.org describes DNP3 as:
The development of DNP3 was a comprehensive effort to achieve open, standards-based Interoperability between substation computers, RTUs, IEDs (Intelligent Electronic Devices) and master stations (except inter-master station communications) for the electric utility industry.
Also interesting to me was the justification for the differences between the European standards and the North American standard:
DNP3 has been designed to be as close to compliant as possible to the standards as they existed at time of development with the addition of functionality not identified in Europe but needed for current and future North American applications (e.g. limited transport layer functions to support 2K block transfers for IEDs, RF and fiber support).
In short like the IEC standards DNP3 defines the protocols used by various components to send commands and return telemetry. Were I going to be working in a North American based facility I’d spend some more time with this.
OPC DA: Open Platform Communications Data Access — I’ll be honest, this is one I need to talk through with someone who understands it. Researching it is a bit all over the place; sometimes it sounds like a set of data formats, sometimes it sounds like a communication protocol. I’m not really sure, either way reading with context from the CRASHOVERRIDE report it seems like a data format that is being disrupted by overwriting a good value with an invalid one.
Devices:
RTUs: Remote Terminal Units — A RTU allows remote commands of complex systems, as well as feeding back telemetry via the control network (in fact sometimes they’re referred to as Remote Telemetry Units). These receive control signals to specify how to manipulate connected systems.
PLCs: Programmable Logic Controllers — PLCs were a tough one until I started watching some videos about programming them. Basically they’re small, advanced switches (yes, like a light switch) that accept & send multiple simple inputs and outputs. The key though is the boolean logic in between those inputs and outputs. These are visualized as Line (aka Ladder) diagrams:
For example in this case we see the first set of gates on the left (I 57, I31, R25, & R26). It’s not apparent at first, but you could also represent it in Python as:
The logic, even relatively simple logic, is powerful because it allows a control to be changed without being rewired, just reprogrammed. In systems where three or four different actions happen at once (to activate a safety mechanism when a machine starts for instance) there may be multiple motors started at once. Using a PLC allows the programmer to change the order and conditions where those motors start without having to physically rewire the control signals. It’s a bit more nuanced than an RTU, but really cool once I understood it.
RTUs and PLCs started to sound very similar to me, but clearly had different expectations/uses. I dug in and came across Supervisory Control And Data Acquisition Remote Telemetry Unit; Why You May Want More Than A Programmable Logic Controller. Basically PLCs are simpler (which makes sense) where RTUs have a much broader set of capabilities. Simplicity is good, but sometimes you need to do more complicated things, thus the need for RTUs.
HMIs: Human Machine Interfaces — All this stuff we’ve discussed so far is about how computers talk to machines. At some point though a human needs to interact with the system, whether it’s making small changes or reprogramming the entire system. HMIs are the systems people use to reach PLCs, RTUs, and other systems.
These range from touch screen systems on the floor of production facilities:
To room filling command centers:
As a network defender nothing surprised me more though than learning that most of the systems are running Windows based operating systems. I had some crazy idea that these were built on large massive Unix distributions or some kind of old school SolidWorks style real time operating system. I don’t know exactly why but that was my preconception. It changes my understanding the environment drastically in both a good way and a bad way.
On the positive side it means the ability to use commodity tools, even if they have to be customized or tweaked for the environment. On the negative side that means commodity threats are in scope and attackers can repurpose a lot of their tool chains (something we saw with BlackEnergy2). This is one of the places traditional CND intersects with ICS CND.
Intelligent Electronic Devices — This is another device type that I feel like is more nuanced than I’m 100% grasping. In many ways an Intelligence Electronic Device (I don’t use the acronym since it’s pretty confusing; see Improvised Explosive Device) seems a lot like an RTU. I might not be the only one feeling that way after reading this University of Kentucky PDF describing the differences. I need to spend more time on it, but for now I know an Intelligence Electronic Device is both a telemetry generating device and an advanced control device specifically suited for managing electric generation and transmission equipment.
ABB PCM600 — Finally a relatively easy one. ABB it turns out is a company that makes control systems. The PCM600 is a specific model they sell and is a management tool for a system of IED. No coincidence it happens to be compliant with IEC 61850 systems and standards.
IOAs — ❓No clue. Maybe Google was just screwing with me but I didn’t track this one down. I’m adding it to my list for when I get to talk to a real control engineer or future research.
Yes, I know for a post like this it seems odd but I think calling out gaps is as important as sharing answers. This whole post is about understanding my knowledge gaps.
Technology Families
These were pulled from the historic discussion of past ICS threats, specifically all theree of these families were targeted by BlackEnergy2.
Taking a look at their websites these are comprehensive SCADA systems. I will hold off on digging into those until I spend some time on BlackEnergy2.
Securing It
With all the focus on what’s what I haven’t even gotten to the security aspects yet. I’m planning on starting with the following paper, then moving on to the following book:
Paper
Maybe it’s just the fact that I grew up (in the professional sense) reading NIST series 800 papers, but this is the first place I went:
Book
I haven’t gotten to it yet, but this book is on my reading list.
Conclusion
I admit I was not planning to write another background post before getting to the Victim piece but I felt it was ultimately worthwhile. As I’ve called attention to previously the scary thing about ELECTRUM wasn’t their malware or APT style tradecraft, but their understanding of the ICS environments they were in and how to play within those rules. As a result defenders need as much understanding as possible.
To be honest this isn’t easy. While understanding Windows security can start with a Virtual Machine and a few open source tools there aren’t many easy to play with PLCs or RTUs (though that’s changing, thanks Kyle Maxwell), let alone power or water plants with full SCADA environments. These videos, books, and articles are just the start of a much longer learning journey. As an exercise in “learning out loud” this isn’t easy, but one thing I love about blogging is not just sharing what I know, but how it causes others to come share with me. That’s key for something like this where there’s so much to learn. Thanks for playing along.
Next (and yeah, this time 100% for real) up: the Victim.
