Chainguard’s response to CVE-2024-3094, aka the backdoor in xz library

On March 29, a compromise of SSH via the upstream xz/liblzma package was published. A backdoor was added to the upstream XZ Utils Data Compression Library project, which through a complicated sequence, would change the behavior of sshd. Chainguard has determined that our Images are not affected, however, out of an abundance of caution for our users and customers, we’ve withdrawn and revoked impacted liblzmapackages and rebuilt our Images with unaffected versions of liblzma.

‍

The details

The security breach involved the liblzmacomponent of the xz package. Looking through the code history, it appears the author of the malicious code orchestrated changes to compromise the build process of liblmza on some systems. Under a set of specific conditions, liblzmawill inject additional code. When sshd is linked against a compromised version of liblzma, its startup behavior is modified, presumably opening a backdoor in the pubkey login process. The author of the disclosure provided many more interesting details, we highly recommend the read.

‍

Red Hat assigned CVE-2024-3094.

‍

We investigate issues like this one – so you don’t have to

Looking through the details of the attack reveals a few prerequisites in order to be affected:

‍

✅ Use a recent version of liblzma (5.6.0+): Chainguard Images were shipping version 5.6.1, but we have rolled back to version 5.4.6 and removed 5.6.0 and 5.6.1 from our distribution out of an abundance of caution, however, we were not affected and customers and users remain safe from this attack. These Images are available now for our free tier users and customers.

‍

❌ Build liblzma on a Debian/RPM based x86_64 distribution. Our packaging of liblzma takes place on our “undistro” Wolfi, which was not specifically targeted by this malware.

‍

❌ Configure the build of OpenSSH to link to liblzma. Our packaging of OpenSSH is part of our open source Wolfi repo, and does not link to liblzma.

‍

Analyzing linked libraries on Chainguard Images’ version of sshdalso reveals the offending liblzmalibrary is not included. Meaning users and customers will not be affected by the SSH backdoor.

$ ldd $(which sshd)
	linux-vdso.so.1 (0x0000ffffbc13b000)
	libcrypt.so.1 => /usr/lib/libcrypt.so.1 (0x0000ffffbbfa0000)
	libcrypto.so.3 => /usr/lib/libcrypto.so.3 (0x0000ffffbbb90000)
	libz.so.1 => /lib/libz.so.1 (0x0000ffffbbb50000)
	libc.so.6 => /lib/libc.so.6 (0x0000ffffbb9b0000)
	/lib/ld-linux-aarch64.so.1 (0x0000ffffbc0fe000)

Recommendations for Chainguard Images customers

While Chainguard Images are not impacted by this vulnerability, the affected liblzma versions 5.6.0 and 5.6.1 that were used to potentially attack other Linux distributions might still be present in previous Image versions users and customers may have pulled. We recommend Chainguard Images customers and users update to the most recent versions of Chainguard Images that were released, which removed the affected versions of liblzma (5.6.0 and 5.6.1) and were rolled back to version 5.4.6.

‍

Chainguard Images customers and users can also leverage each Chainguard Image SBOM and our Chainguard Events API to surface where the affected versions of liblzma are present.

‍

To enable customers to track what Images have been pulled by their users, we've published a demo application that listens to Image pull events and records them in Google BigQuery. You can deploy this example directly, or fork it and make whatever modifications you want.

‍

After recording pull events, you can join the table with SBOM data to determine when Images containing certain packages (such as xz 5.6.x) are pulled, by whom, and from where.

‍

For customers that need additional support surfacing this information, please contact Chainguard Support.

‍

What’s next?

This attack is a long game on the part of the author. The malicious code appears to have been added by a core contributor of the upstream xz project among a large number of other changes. We expect this incident to raise significant attention on the xz project which may uncover additional findings. Chainguard will closely monitor any potential new findings and continue to keep our users safe.

‍

Get started with Chainguard Images

Chainguard Images are built to reduce the number of components inside, only using what is required to build or run an application. Because of these minimal and intentional design decisions we take, Chainguard Images do not include SSHor liblzmaby default, thus protecting our users and customers from upstream attack vectors if our solutions were to be targeted. Chainguard’s approach to rapid updates also plays an important factor in our vulnerability response time. Within a few hours, we were able to identify use of the affected liblzmapackage in Images, withdraw it from our environments and rebuild Images with the unaffected versions to protect our users and customers.

‍

Software supply chain security concerns will only continue to be heightened as new threats that have widespread impact on users emerge—like this one. If you are looking to strengthen your software supply chain security defenses, Chainguard is here to help. You can get started with Chainguard Images for free today to see for yourself how we're working to improve the container image landscape with a secure-by-default design. Our Images inventory is always expanding. If you need something you don’t see listed in our Directory, reach out to our team.

‍

Editor's note: This blog post was updated on April 3, 2024 to include additonal recommendations for Chainguard Images customers.