Podcast: Where Do Your Bits Really Come From?

Earlier this year I attended the Public Safety Canada Industrial Control System Security symposium in Charlottetown, PEI (FYI the PSC ICS events are outstanding – worth attending, even if you are not Canadian). While there, I had a chance to meet with an old friend, Andrew Ginter, Vice President of Industrial Security at Waterfall Security Solutions. We chatted about an issue I’ve been interested in – or, dare I say, obsessed with – for a while now: the software supply chain in ICS and how to ensure that it’s trustworthy. Our conversation was the basis for the podcast Where Do Your Bits Really Come From? Let me fill you in on some of the points we discussed.

We began by talking about the field itself: just how widespread is the issue of supply chain integrity? The problem is actually much larger and complex than I suspected when I started investigating it in 2017. Initially, I thought that the problem just affected the owners and operators of ICS assets: as the 2014 Dragonfly 1.0 attacks showed us, technicians in the field risk patching their control systems with harmful updates.

Dragonfly Attacks
The Dragonfly attackers penetrated the websites of ICS vendors and replaced legitimate software with packages that had trojan malware called Havex embedded in them. Customers downloaded and installed these infected packages, believing them to be valid updates.

However, the more people I spoke with about supply chain issues, the more I realized ICS vendors had their own challenges as well. It turns out that people view the problem differently depending on which vertical they’re in and what their role is – but the same basic problem still exists for all audiences.

So what exactly is the problem? Well, that’s pretty simple: How do we, as users, trust the firmware and software that we’re loading into our industrial control systems?

And to look at it from the other side: How do vendors know that the software out in the world associated with their organization hasn’t been tampered with or did, in fact, come from them?

When dealing with software, there are multiple issues that test our trust. The two examples I discussed with Andrew were counterfeiting (injection of malicious code into the supply chain) and our inability to know exactly what components make up a software package. This latter issue is complex because of the way we develop software today: most software projects include embedded third-party and open-source code. And that embedded code has its own third-party and open-source code embedded in it. So what happens if one of those subcomponent’s subcomponents has a vulnerability? Would the ICS vendor even know about the vulnerability? Would their customers know? Unfortunately, the usual answer is “No.”

Nesting DollsPhoto by Marco Verch / CC BY 2.0 / Cropped, small doll face altered

This may sound a bit gloomy but don’t despair: we talk about how the industry is making progress in this area. For example, code signing technology is useful to address the software tampering issue, though it won’t solve the problem on its own. Unfortunately, it is complex, it’s not widely used outside of IT software – AND malware writers have figured out how to use it to their advantage!

The key is in the ability to break down the libraries so that we can identify who built which pieces and generate a reliable Software Bill of Materials, so to speak. The solution, as is often the case, lies in having better knowledge and then being able to effectively share that knowledge with users.

It’s amazing how much ground you can cover during a single conversation. After talking extensively about the problem, I shared how we are using the information available to us to find a solution. I spoke with Andrew about aDolus, how it all started, and how our team is working to address the problem with the supply chain. Vendors are keen to come on board to help us with our solution.

I’m hoping you find the challenge of securing the supply chain as interesting as the aDolus team does. If so, jump over to the Waterfall Security Solutions podcast page and listen to the full podcast. If you stick around until the end, you’ll even get to hear where the company name aDolus comes from. Enjoy!

Who Infected Schneider Electrics’ Thumbdrive?

Infected USB Drive

On 24 August 2018 Schneider Electric issued a security notification alerting users that the Communications and Battery Monitoring devices for their Conext Solar Energy Monitoring Systems  were shipped with malware-infected USB drives.

First of all, kudos to Schneider Electric for alerting their customers and providing information on how to remedy the situation. According to Schneider, the infected files would not affect the devices themselves. Schneider also noted that the particular malware was easy to detect and remove by common virus scanning programs.

Provided that all of Schneider’s customers read these alerts, this should remain a minor security incident. Unfortunately, this is a big assumption. Due to the complexities of modern distribution channels, I’m pretty certain no one in the world knows if the Schneider notice is getting to the people who actually use the Conext product. It could be getting stuck on some purchasing manager’s desk, never to be forwarded to the technicians in the field. Or it could be languishing in the inbox of an engineering firm that is no longer working at the location where the Conext product is deployed. If ZDNet and CyberScoop had not reported on the story, it may have stayed off everyone’s radar.   Clearly, both vendors and asset owners need better ways of sharing urgent security information.

The Conext Battery Monitor from Schneider Electric
The Conext Battery Monitor
Source: Schneider Electric

But what is especially interesting is that the thumb drives were not infected at Schneider’s facilities.  They were infected via a third-party supplier during the manufacturing process. Like ALL major ICS vendors, the supply chain for Schneider hardware, software (and even the media upon which it is shipped) is exposed to many hands.

This situation highlights an alarming reality in the ICS world. Just because a digital file comes from a trusted vendor doesn’t mean you can trust all the other companies that touched that file.

Who knows which “third-party supplier’s facility” was involved in contaminating those USB drives?? Was it the USB manufacturer… or a duplication company… or even a graphics company who added some branding? Schneider Electric no doubt will be re-thinking that relationship, but the fact remains that they have to work with 3rd parties to get their products to market.

The worrisome question is, what other ICS vendors use that same third-party supplier? How widespread is the infection? It seems unlikely that Schneider Electric is this supplier’s only customer. Naming and shaming the supplier may be fraught with legal consequences (or perhaps they are still tracking down the specific vendor) so Schneider has remained silent for now on the source of the malware. That means all the other vendors out there and their customers may be exposed as well. Or not. We don’t know – and that is a problem.

One hopes that if other vendors have detected issues with their USB drives, they will follow Schneider Electric’s lead and issue prompt alerts. Some vendors are better than others at transparency and there will likely be some who choose to lay low instead to avoid bad publicity. It is a pity because vendors like Schneider are as much a victim in this scenario as the end users.

This is one of the reasons aDolus is developing a platform for ICS asset owners and vendors that offers an ecosystem of trust where they can verify software of, let’s call it “complicated origin” and ensure it hasn’t been tampered with BEFORE they install it. We’re also looking at ways vendors can get early warnings about security issues occurring at their client’s sites and not have to wait until hundreds or thousands of facilities have been infected.  

Interested in learning more about protecting yourself from compromised software? Let us know if you are an end user interested in validating ICS software or an ICS vendor interested in protecting your distribution mechanisms to ensure they are clean.

Follow Us:

Twitter
  LinkedIn

Scroll to top