¶ Intro / Opening
[Music]
Chain of events, cause and effect. We analyse what went right, and what went wrong, as we discover that many outcomes can be predicted, planned for, and even prevented. I'm John Chidgey, and this is Causality. Causality is supported by you: our listeners. If you'd like to support the show you can do so in via Patreon or by becoming a premium subscriber for early release high quality ad-free episodes. Visit https://engineered.network /causality to learn how you can
help. Thank you. "Black Energy" On the 23rd
¶ Black Energy
of December 2015 in the late evening major sections of the Ukrainian power grid blacked out in quick succession. The blackouts impacted 225,000 residences over a huge physical area, shedding some 170MW of load from the electrical power grid. There were no storms, no equipment failures, and no human error. Well, no human error at that
moment in time anyway. Before you wonder why a power blackout is worthy of discussion, is that this particular power outage was the first, large-scale power disruption caused by a cyber intrusion. The incident has been referred to as "Black Energy" not to be confused with "Industroyer" which happened 1 year later. The name of the incident is linked to the name of the malware that the incident made well-known. Which interestingly, was actually the 3rd generation of the Black Energy malware
that was identified first in 2014. The truth is that the Black Energy Three malware wasn't the cause of the outages, but rather it was the key that initially let the attackers in. But we're getting ahead of ourselves. Ukraine has 24 regions with each region varying from 11 to 20 seven provinces in each, and covering these to various scales, there is a different power distribution company serving that region, called Obelnergos. The Black Energy incident was targeted
¶ The Incident
at multiple of those Obelnergos. At approximately 3:25pm Eastern European Time on Wednesday the 23rd of December, 2015 the call centers for reporting power grid issues were flooded with a barrage of telephone calls, simultaneously. These calls appeared to have all originated from Moscow, overloading the telephone system
completely. Five minutes later, in the Prykarpattya-oblenergo control center, the SCADA remote control system seemingly possessed, locked the operators out and began executing a manual isolation sequence. Thirty substations were systematically switched off one by one as massive segments of the power grid were sent into a blackout. Specifically x7 110 kV (or kiloVolt) and x23 35 kV substations were disconnected. They were distribution substations, not major transmission and not generation plants.
Approximately 1 minute later the same attack began on a second Oblenergo, and then a third at 4:00pm. Six Oblenergos were targeted in the attack though only three began to experience an operational impact. And they were Chernivtsi, Kyiv and the aforementioned Prykarpattya. Two of those three control rooms were also blacked out themselves despite having UPS's fitted with the third control room remaining powered through out the
incident. At 4:10pm engineers disabled a HMI administrator account in an attempt to stop the attack and regain control. However the hackers had installed an additional administrator account beforehand and hence this action had no impact. In the following minutes the Obelnergo engineers initiated a shutdown of the SCADA system and remote access VPN and were thus able to end the attack, however this occurred only in time to prevent the final blacking out of just one solitary sub-station
that remained. All of the others had already been blacked out. At 5:00pm the Prykarpattya-oblenergo published a note on their site stating the following: (this is a Russian translation to English) "To the attention of consumers of PJSC, due to malfunctioning of telemechanics, a considerable part of the region and Ivano-Frankivsk remained without power supply. Specialists are currently searching for the causes and finding out the extent of the accident. We will provide more
information as we receive it. Please refrain from making phone calls to the call center as the dispatchers do not have any information about the timing of the accident until the cause of the accident is known. Thank you for understanding." (end quote) At 5:30pm an additional note was also added. Each Oblenergo attempted remotely re-energizing their substations to no avail, and then systematically dispatched field operators to each substation one by one.
Upon arrival at each substation the field operators systematically switched on each of the circuit breakers powering up sections of the power grid. The shortest full blackout period lasted approximately 1 hour, with the full power restoration taking approximately
6 hours. In attempting to restore their control system infrastructure, Oblenergo engineers found most operator workstations were unable to be booted up and those that they could, they found operators were unable to login with their credentials and where they could login successfully, many of their remote sites remained unable to be monitored let alone controlled remotely. The attack had rendered them remotely blind and completely remotely incapacitated. So how
¶ How Did They Get In?
did they get in? First of all it, started with a spear phishing attack focused on IT staff and sys-admins, (that's system administrators) that worked for several companies throughout the Ukraine that supported the Ukraine electricity network. They delivered an Email with a malicious attachment that appeared to have come from Ukraine
energy ministry. 6 of the Ukraine power companies were targeted with an old-fashioned Word or Excel document with a macro embedded in them, that once the user opened and enabled macros, delivered the Black Energy 3 payload. In this case the attachments were predominantly Excel files. Black Energy 3 then allowed the installation of key loggers to capture credentials, allowing further system access laterally until reaching the domain controllers. This began more than 6 months before the
actual attack would take place. None of the companies had any capability to continually monitor their control system network logs. The Oblenergos actually had significant logging enabled on most of their systems, server and network devices, however it was not centrally aggregated. Moreover none of it was checked manually or semi-automatically. There was also no system network monitoring at all, with no tools installed for that purpose.
In addition, remote access to the control network was via VPN and that VPN had no multi-factor authentication. The Ukrainian power grid did have remote access for engineers and with the intention of being able to access monitor and remotely support the system, but it did not have MFA (or Multi-Factor Authentication). For those that don't know what MFA is the intention is simple
log in to a secure site or an access portal with a username and a password. The system will then send a notification, a message, SMS, text or similar, to a secondary device that is known to be owned by the user, to confirm that that specific login attempt was coming from that specific user. Once that user independently approves or accepts, access is finally granted. MFA, were it in place, would have alerted those with a compromised account that someone was attempting to remotely log in using
their credentials. How the hackers got the right credentials? Each company segregated their control network from their corporate network using a firewall. So the hackers spent months incrementally infecting more machines on the corporate side with a variety of different exploits until they were able to get access to the active
directory controller. Once in the AD controller they had access to the domain credentials, they obtained credentials for control systems engineers with remote access privileges and were able to access the control network at will without detection. They compromised a total of 17 operator work stations
across the organization's. They installed a variety of back doors including GCat, DropBear and Kryptik, to ensure that there were multiple paths available to get in, should one of them be discovered.
¶ Setting Up The Outage
So how they set up the outage itself, now that they're in? First of all the control room UPS's. All the control centers in the Ukraine as is standard practice worldwide, are fully backed up with uninterruptible power supplies, such that if the grid power goes out the control room will remain powered and functional. The same applies to the dial-in call centers. In the first step the hackers took out 2 of the 3 UPS's such that when the power was turned off to their own power grids those control
rooms would also be blacked out. Then they disabled remote SCADA commands. Serial to Ethernet converters are used for address translation between the SCADA system and the high-voltage circuit breakers and that was remotely upgraded to an invalid firmware revision, awaiting a remote reboot request, after which the device, being unable to boot because the firmware was invalid, was effectively bricked and completely
inoperable. The 2 devices that they targeted were the MOXA UC7408 LX-plus: that is a Universal Communicator with an IXP425. It has 8 configurable RS-232, -422 or -485 ports, 8 Digital Ins, 8 Digital Outs, and a dual LAN. Importantly it's running an embedded Linux version. MOXA is a company based in Taiwan and they make a large number of industrial communications products. There is scarcely a plant that I've worked on, that does not have something made by MOXA, somewhere in it.
The 2nd device was an iRZ RUH2 3G. This is a 3G router, industrially hardened. It supports dual SIM cards and 10/100 Ethernet and again runs an embedded Linux version. iRZ is a company based in Russia. So whilst they are popular in Europe, outside of Europe they are rarely found.
By disabling these with these firmware hacks, this meant that any recovery of SCADA workstations would be ineffective because there would be no communication converter between SCADA and the circuit breakers for remote operation or monitoring. Essentially they were cutting off the intermediary and severing that
link. Disabling the SCADA workstations. KillDisk on the operator workstations will overwrite the MBR (that's the Master Boot Record) as well as key system files rendering the computer inoperable and unable to reboot to recover. Effectively disabling the SCADA operator workstations completely. For several machines KillDisk was manually triggered, by whereas for others there was a time trigger delay to launch at about 5:00pm, an hour and a half into the start of the attacks.
Finally they disabled an RTU on an embedded PC. An ABB RTU560 CMU-02 is a chassis mounted daughter card that runs an embedded Windows CE. One of these, and only one was discovered also corrupted by KillDisk. There been
¶ What Wouldn't Have Stopped it
many suggestions as to how this could have been prevented, but before we talk about that a few things that wouldn't have stopped it. Application whitelisting, for example at the firewall would not have changed the outcome. Non-obvious passwords were actually used so that was not a case of having weak passwords that were easily guessed. They weren't. Significant logging did actually exist, and firewalls existed. Adding more firewalls would not have
changed the outcome. Before we talk about what could have been done to prevent this, let's look at the fallout from the incident. In total about 73MWh
¶ The Fallout
of electricity was not supplied during the blackout period. That represents a somewhat insignificant amount, representing a mere 0.015% of the Ukraine's daily electricity consumption. Fractionally speaking that's 1/6,600th of their total daily supply. There were no documented economic impacts, health or medical complications or spoilage losses as a result of this
blackout incident. The Obelnergos themselves however, they were more heavily impacted in terms of cost to rectify the mess left behind by the hackers. It took months to rebuild the destroyed work stations due to a lack of regular backups. It took months to replace Serial to Ethernet converters due to a lack of spare parts and a lack of backups with the MOXA and IRZ units completely unrecoverable, requiring physical replacement and complete
reprogramming from scratch. During the rebuilding time the system was mostly run manually, with a significantly increased staff presence on the ground at substations. The additional costs of this rebuilding have not been publicly released although parts and labor for the engineering rebuild were likely to be at least a $1/4M USD
in parts and labor. Whilst StuxNet demonstrated that hacking by well-funded and highly motivated hackers could physically damage equipment connected to a control system, interruption of a power grid on a scale like this had never been seen before. The Ukrainian companies were well practiced in manual override controls and given their staffing structure and training they were able to restore power and run the system manually very quickly and for several weeks until the remote control
infrastructure could be fixed. Other countries around the world have been progressively relying more and more on control systems infrastructure to operate their facilities remotely to reduce operating costs. In many cases local controls are also being sacrificed entirely to save on manufacturing and ongoing maintenance costs. In such cases an attack like this one would have been far more devastating.
Without sufficient manual override control or without enough trained personnel to operate substations manually the outage could have lasted for days, or even a week in some countries around the world. The ramifications of Black Energy have actually been a strengthening of the application of cybersecurity defense-in-depth architectures in utilities around the
world. Driving a huge spike in investment in software for monitoring logs, detecting abnormal network communication and the identification of data traffic specifically between SCADA systems and equipment controllers. Four months ago in mid-2019 the SEIA bill or Secure Energy Infrastructure Act in the United States passed through the Senate.
It mandates sufficient manual controls to ensure manual control overrides and procedures exist to operate the electrical grid infrastructure in the event of a failure or a compromise of automated control systems. The Act attempts to counter the growing trend of cost reduction by removing full manual controls in favor of automation and to mandate that those controls must be installed, inspired in large by the Ukraine attack in 2015. So what lessons
¶ Lessons Learned
have we learned from this? Social targeting is a big problem that a lot of people don't appreciate. When I used to work at Nortel Networks in the late 1990s organizational charts (or Org Charts) were relabeled as "Commercial in Confidence" documents because there had been several incidents of targeted poaching attempts from other Tech companies. In essence it was a map to the structure internally and those wishing to poach people would know straight who
to speak to. Today social media and the over-sharing of information has made it difficult for companies to protect this sort of information let alone from poaching attempts but from hackers as well, and today companies like LinkedIn make it incredibly easy for a hacker to craft a highly targeted phishing or spear-phishing Email to exactly the right group of people they need to get
access. As previously mentioned spear-phishing campaigns or attacks are the first volley from those trying to penetrate a network and they're becoming harder to detect and block, and are becoming far more frequent. If I'm a hacker and I want to disrupt a company that runs industrial equipment like electricity generation or electricity supply, Oil & Gas, mining, water supplies even hospitals. Many employ control systems engineers like me to maintain and support their control system
infrastructure. If I post that I'm a control systems engineer on LinkedIn, I'm working for the target company someone wants to disrupt, they then draw up a short list. Sometimes people over share the projects
that they've worked on. Perhaps they've worked on a safety system for an explosives plant, which might make it easier to target those people specifically in a spear-phishing attack, since those people are the most likely to have credentials that have access to those safety systems you're trying to
disrupt. From a shortlist constructed from LinkedIn it stands to reason that I would know other control systems engineers also working at the same company so now they're able to send an Email to me that at first glance appears to come from one of my co-workers someone that I know but has a malicious attachment that I'm less
suspicious of. So I open it unsuspectingly then they have access to my machine, and you might think LinkedIn restricts access to my list of connections if I select that, and if you're not a connection of mine then that's true (certainly the way I've got it set) but all it takes is for someone to pose as a recruitment consultant, a headhunter, an agent, and if you're in the market for a new job even if you're strict about only connecting with people that you've personally met, you might
just accept one or two requests like that and if one is malicious then they have everything they need with all of your connections to find who to target. The hackers know that the control systems engineers have elevated privileges in the control system and might even have remote access rights as well if that exists, so once they've got on to my machine a key logger lays-in-wait for me to log into remote access gateway to the control network and then they have my credentials and it's open season.
If that access portal is internet exposed they can get in and out anytime they want and if no one is cross-checking when I'm logging in and when I'm not, and if there's no other mechanism like Multi-Factor Authentication then it's an open door and they can poke and prod and look for weaknesses from within the system to the extent my credentials let
them. In the case of Black Energy the original attack wasn't quite as targeted but ultimately they still achieved their goal by gaining access to the domain controller, it just took a little longer. But undetected they could take as long as they liked. There have been a lot of reports digging into the incident and many lessons can be learned from it. ISA or IEC 62443-3-3 lists 51 system requirements that are recommended to improve resilience to cyber intrusion in Operational
Technology (or OT) systems. An International Society of Automation (or ISA) report highlighted 7 significant SR breaches in this incident which I'll cover only three of them that I consider key. 2.4: The transfer of malware between systems on the OT network clearly demonstrated that there were no substantial controls restricting file transfer throughout the control system network. It's common practice to have a so-called secure gateway for all file access between machines on an OT network.
That gateway or Dropbox is heavily scanned and logged and reviewed regularly. All other file transfer pathways are then restricted or disabled completely making it significantly harder to spread malware through the OT system if you do get in. People complain (that use the system every day) it's hard to get data in or out of the system without that additional intermediate step, but it does stop the hackers from easily cross-infecting machines on the
network. Point 5: Whilst a firewall existed lacking sufficient additional controls it was rendered mostly ineffective. The essence of this comment: for a firewall to be useful there also has to be strong authentication. Also detection via an automated or manual log review method of some kind would have uncovered the hackers activity quite easily. Item 6.2: Lack of overall network
monitoring. Without any monitoring tools the attackers could surveil the network extensively completely undetected for weeks and in this case, months. So what do we conclude from
¶ Conclusion
all of this? If you're working with critical infrastructure and there are remote access pathways to your control system infrastructure then be warned, you are or will be (someday) a target. If you're part of an IT or OT department, make sure you have a centralized log collection and analysis tool, and have eyes on glass looking over those logs for any suspicious activity because that is part
of your job. If you're able to install software tools that can monitor network traffic, learn standard behaviors and then report on exceptions or suspect traffic patterns, then invest in it and use it. If you have a remote access portal for goodness sake install Multi-Factor Authentication. We humans are sometimes too clever for our own good. We use our intelligence to build machines that require less and less work
for us to do. It's easier to monitor, reset, open and closed circuit breakers from a room hundreds of kilometers away. Why drive there when you can just click a button. It's easier and it's cheaper to build things with no physical buttons, no indicator lights, no manual overrides and that's fine until something goes wrong and you need them. Cyber-security is a never-ending tug of war between company convenience and the risk of cyber
inconvenience. The more convenient that we make it for ourselves, the more convenient we make it for hackers to make our lives inconvenient. In the case of Black Energy only 0.015% of the power grid was inconvenienced that evening. Next time though, the consequences could be much much more severe. If you're involved with OT
pay attention. Those people intent on wreaking havoc throughout the world, they might be watching you and waiting for you to choose that convenience and laziness over security. If you're enjoying Causality and want to
¶ Outroduction
support the show you can: by subscribing to the Premium site via Breaker or via Patreon. You can find details at https://engineered.network/ /causality with a thank you to all of our patrons and a special thank you to our Silver Producers Carsten Hanson, John Whitlow, Joseph Antonio and Kevin Kosh, and an extra special thank you to our Gold Producer known only as "r". Patron rewards include a name thank you on the website, a name thank you at the end of episodes and access to detail
raw show notes too. Premium rewards also include add free high-quality releases of every episode, so if you'd like to subscribe you can help make sure the show continues to be produced and above all else it's all really, really appreciated. Of course there's lots of other ways to help like favoriting this episode in your podcast player app or sharing the episode or the show with your friends or via social. Some podcast players that you share audio clips of episodes so if you have a favourite
segment, feel free to share that too. All of these things help others to discover the show and can make a big difference. Causality is heavily researched and links to all materials used for the creation of this episode are contained in the show notes. You can find them in the text of the episode description of your podcast player or on our website. You can follow me on the Fediverse @chidgey@engineered.space on Twitter @johnchidgey (all one word) or the network @Engineered_Net.
This was Causality. I'm John Chidgey. Thanks so much for listening.
[Music]
[Music]
[Music]