The concepts of trust and security are different, but often confounded. They are similarly nuanced as the difference between threats and vulnerabilities, but I’ll save that for another day. The difference between trust and security was highlighted recently because it was discovered that there had been more than 5 million downloads of a cryptocurrency miner virus from DockerHub. A few years back, the XCodeGhost malware infected 39 iOS apps, including WeChat and something I use personally, CamScanner. This impacted hundreds of millions of users. These two incidents highlight a serious security problem—to feel confident in your software supply chain, you have to do both of the following:
- You must download a trusted thing.
- You must download from a trusted source.
Neither is sufficient by itself.
First let’s define what a trusted thing is—this is something that you identify as secure. For example, when you download an operating system ISO file, you expect it to be from the operating system vendor and hence secure. You have used this operating system for years, and the ISO has always worked well. So, you trust that if the ISO is genuine, it’s trustworthy. But, if you download it from an untrusted source, it might not actually be the real ISO. In this case, you think you are downloading something that you trust, but it’s actually not because the source tricked you.
Now, what’s a trusted source? This is a place that you normally download things from, or perhaps a place that looks legitimate. You analyze the website—there aren’t a million ads and popup boxes, the interface design looks professional. Or, perhaps you have used products from the company before, so you trust the brand. So, you decide that the source is trustworthy, and hence secure. But this isn’t always the case.
Trust is about perception, and just because you trust something, that doesn’t mean it is secure. Threat modeling is about aligning our perceptions with reality. That’s really what this post is about.
If you download an untrusted container image from a trusted source, you can still be hacked—for example, DockerHub Miner Virus. If you download a trusted code editor from an untrusted source, you can still be hacked—XcodeGhost. This is especially true with software that yoy use to build other software—code editors and container images are downloaded off the Internet and used in your networks to build and deliver other software. This opens an indirect attack vector, making it difficult to catch once deployed—which means you must be extra careful.
Historically, Apple, Google, and Red Hat have done a pretty good job of keeping malware out of the App Store, Android Store, and Red Hat Portal, respectively. Apple and Google both complete reviews of apps before they are accepted and redistributed. Red Hat does not accept submissions from third parties, but instead reviews upstream code for security issues before it productizes it. Apple and Google are the gatekeepers of third-party content, much as Red Hat is the gatekeeper for the open source code in its products. The app store and open source problem spaces are similar, and historically these solutions have worked fairly well—as long as you download the trusted thing from the trusted source. Got it? Good.
In the case of the DockerHub incident, it appears that these container images were uploaded by a community member, and they were available for about eight months before they were taken down. This is a challenging problem with community-built images. Anonymously generated container images are commonly used as an intermediate layer to build on and deliver application code. This is the same as downloading any freeware from a untrusted source and compromising your supply chain.
In the case of Apple’s Xcode, the download is fairly large, so some app developers in China decided to download the Xcode development environment from a mirror site in China. They did this because it was much faster than downloading from the US. Apple even had tools in place to prevent the installation of a noncertified copy of Xcode development environment, but the developers (in this case) ignored the warnings. So, developers in China ended up downloading and installing a hacked copy of Xcode that added malware directly into the apps they created. The worst part is that the App Store review didn’t catch the Trojan.
How does this apply to container images?
Well, this same problem is applicable to any container image you download and run. There are multiple attack vectors. Typically, with containers, people think about a direct injection where a hacker would add malware to an existing container image. This kind of attack would be easily detected with Notary and the Update Framework or Simple Signing. If a hacker added code to the official MySQL container image and redistributed it, these signing mechanisms would automatically detect it and prevent it from running.
Well, this sounds good, but Apple and Google have that too. What if instead somebody hacked the build system where these container images are produced and added malware before they were signed? Then these container images would be distributed around the world and verifying signatures would happily allow you to install them. What if, a hacker hacked your local Docker binary so that everything you built with it had malware? Again, this could be redistributed, and nobody would know the better. You would blindly sign it with your keys and tell the world that it was safe to consume (a nonrepudiation problem, my favorite term in security).
How to stay secure
This is where deep container inspection (DCI) comes in. Imagine if, before you pulled down a new container image, it was first scanned for malware. Imagine if you had a policy that would quarantine an image and mandate review by a forensic technician—this is the future of DCI.
With DCI, even if someone hacked the development environment or hacked the build environment, or even if an angry employee manually injected code into an official container image, it could still be detected by users (who rely on the certified image) before they deployed it locally.
There is a whole ecosystem of images scanning partners, including Black Duck, Sonatype, Anchore, and Twistlock. For defense in depth, I typically suggest a combination of tools and approaches. First, download your base images from a trusted source like the Red Hat Container Catalog. Then, verify the image signatures. Then, as you build on top of them, do more verifications. Red Hat’s OpenSCAP is good for content which is provided by Red Hat, but I recommend a mix of third-party tools to scan anything you download off the internet (an untrusted source).
This article is published as part of the IDG Contributor Network. Want to Join?
