If xz's backdoors are inevitable, how do we stay secure? The answer is: move faster!
As we see more and more software supply chain attacks, organizations are seeking ways to mitigate risks and respond promptly when vulnerabilities are disclosed. Regardless of the attack itself — in this case the latest upstream vulnerability in the xz library (CVE-2024-3094) — most of the events thereafter are good news: it was caught three days after it was merged into mainstream Linux distros, and the communities reacted immediately and patched the affected branches.
And yet, despite the good news for open source in the case of this attack, both among the security community but also among those concerned with their supply chain, a somberness is prevailing.
There is a lingering sense of concern among the security and open source communities. Not only because most find it inevitable to avoid any similar attacks in the future, but also because many think that such sophisticated attackers surely have performed this same technique in other popular open source projects at present. And worse, that they are still going unnoticed. The methodology is certainly adaptable and it seems clear that the resources and patience of the attackers know little if any limits.
The goal of this blog post is to provide some hope to those that care about software supply chain integrity and to outline why our Chainguard Images solution is the best answer to today’s challenges. This is because easy auditability of software inventories in conjunction with fast propagation of patched software are currently the best way we have to minimize the impact of vulnerable open source software.
Trust but verify: The key to a transparent image catalog
Chainguard Images provide a comprehensive set of bundled open source software. They can all be found in our directory. One of the strongest features of Chainguard Images — and in general regarding Chainguard’s philosophy of software bundling and distribution — is zero trust and verification.
While our customers and users put a lot of trust in us by relying on the software we build daily, we design our Images product to be continuously verified, attested, and auditable to maintain that trust. And we provide them with all the tools necessary to assess themselves, independently.
Whether it’s to check who signed the image, when was it built, what it contains, or where the components came from — any of these questions can be verified by our customers at anytime. Our software supply chain is transparent and enables everyone to measure the impact of any attack immediately and easily.
The high-quality Software Bill of Materials (SBOM) generated at build time, the public registry telemetry, and Supply-chain Levels for Software Artifacts (SLSA) attestations are the tools we provide our Images with so that our customers and users can verify everything about their Chainguard Images across their own supply chain. Pick any given Image and you’ll find all of them conveniently distributed through the main tabs of the listing itself in the public directory.
Here’s an example using the Golang image:
Supply chain agency: Auditability, inventory, and attestations
Traditional methods of generating SBOMs post-build, typically through software composition analysis (SCA) tools, can produce less accurate information, sometimes referred to as GuessBOMs. This is still contested, but at Chainguard we believe that build-time SBOMs are the only way to make SBOMs successful long term. That’s why we generate them at build time.
Generating SBOMs at build time allows for the inclusion of more detailed information about the software components, reflecting any changes made by the compiler or other tools during the build process. Let’s not forget that one of the many obfuscations the liblzma attacker introduced was precisely at this stage. As the NVD CVE describes:
"Through a series of complex obfuscations, the liblzma build process extracts a prebuilt object file from a disguised test file existing in the source code, which is then used to modify specific functions in the liblzma code. This results in a modified liblzma library that can be used by any software linked against this library, intercepting and modifying the data interaction with this library."
Building SBOMS with apko — Chaiguard’s image packager — and part of Wolfi — our container image toolchain — exemplifies how enabling the creation of container images with complete SBOMs at build time enhances the integrity and accuracy of the SBOMs produced.
As soon as the news broke, our customers and users knew that all affected versions of xz were 5.6.0+ and more recent. They were well equipped to immediately query their image catalog, audit it, and measure the exposure of their software by following these simple steps using Chainguard Images features:
List all the images and their details, including the dates and epochs
Extract the repository names, tags, and unique digests from all images
Download attestations for each repo, tag, and/or digest using the cosign
Filter out any image that didn’t contain the affected xz component
Have a clear view of affected images in minutes
Overall, Chainguard customers had all the information they needed within minutes to respond adequately.
Slow kills, speed is safe
Chainguard Images were basically immune to this vulnerability for two reasons: the backdoor was designed to target Deb and RPM packaging systems. Wolfi uses apk and thus, it wouldn’t have worked for any of our packages. But also, the exploit only affected versions of OpenSSH, which linked to libsystemd. This doesn’t happen in Wolfi’s OpenSSH package since it doesn’t link to libsystemd.
Despite all this and out of an abundance of caution, any image and package in the Wolfi distro that contained liblzma was forced to use version 5.4.6. Liblzma 5.6.0 and 5.6.1 versions were removed from our distribution. We still recommend Chainguard Images customers and users update to the most recent versions of Chainguard Images that were released, which removed the affected versions of liblzma (5.6.0 and 5.6.1) and were rolled back to version 5.4.6.
And this is the last, final, and critical point: Chainguard Images — or the custom toolchain underpinning their build, release, and distribution — were precisely designed, streamlined, and fine-tuned to tackle these situations. To propagate patched software at rapid speeds and distribute safe, secure, and minimal container images to our customers’ and users’ doorsteps within hours.
This is how it started:
This is how it went:
To learn more about Chainguard Images and how they can strengthen your security posture and fortify your software supply chain, check out our website for more information or reach out to us directly.
Ready to Lock Down Your Supply Chain?
Talk to our customer obsessed, community-driven team.