The importance of toolchain security in NIST's SSDF

What if I told you that you’re likely operating a system that executes large amounts of untrusted code every day, that this system most likely has access to production deployment credentials, and that your vulnerability scanners are almost definitely not configured to monitor it? Even worse, many of these systems treat remote code execution as a feature - not a bug! To top things off, attackers routinely target these systems, with supply chain attacks rising over 600% in the last year alone.

‍

No, I’m not talking about log4shell or ffmpeg, I’m talking about your build systems. Most modern programming language toolchains, such as Python and Ruby support dynamically executing code from third-party modules as part of the installation process as a feature. This can be used to configure compilation for native code to make installation easier, or it can be used to steal keys and other private data. We saw this happen to the Python ecosystem back in February and similar attacks have been conducted on NPM as well.

‍

I’ve often said “Treat Your Build Systems Like Production Systems”; in this case it means taking as much care in choosing, scanning, keeping track of, and updating your build tools as you do for production. This sounds like common sense, but is hard to implement in practice.

‍

Secure-by-default must extend to your toolchains

Last week, the Go toolchain, a set of the most used tools to build Go projects, reported several vulnerabilities in the toolchain that allowed remote code execution (RCE) during the build process by manipulating LDFLAG settings. Go is one of the only programming languages that considers this threat vector (RCE when building untrusted code) a vulnerability. These vulnerabilities are interesting for a few reasons:

• Go is a compiled language, so the compiler itself isn’t present in most runtime contexts, meaning most organizations won’t be able to check which version of Go was used to build their applications without more detailed inventory management or provenance (the history of a software application, including its origins, development, and distribution).

• Almost every other modern programming language, Python, Node.js, Ruby, .Net, and PHP, would all close this bug as “Working as Intended”, and not issue security fixes or CVE entries. Go actually treats these as vulnerabilities – which is how it should be.

‍

You might be thinking, “OK, I’ll just scan all my build systems too”. This is the obvious answer, but you’re in for a huge surprise. Build toolchains are massive when compared to their respective runtimes. Here’s a graph showing some typical comparisons of toolchain images vs. their stripped down production counterparts.

With that size, comes an increased attack surface. Here’s the vulnerability data:

This isn’t something you can solve just by stripping out unused components, you need a maintenance and patching process.

‍

What you need to know about SSDF & your toolchain management

And you’re not going to be able to wait forever to implement it. The new Secure Software Development Framework (SSDF) from NIST places toolchain inventory management and security front and center. (Check out Chainguard Academy’s easy to read SSDF guide.)

‍

In fact, just last week the Office of Management and Budget (OMB) updated its guidance on software development practices doubling down on using the SSDF as a north star for self-attestation requirements and offers important clarifications around who will be required to self-attest within the next year. Organizations that are currently delivering software to the federal government or to a vendor of the federal government that haven’t been paying attention to these requirements need to start asking themselves hard questions.

‍

If you’re not sure if you’re ready for these requirements, try asking yourself or your developers these questions:

• Can we check which compilers were used to build what’s running in prod today? What about the versions? Are there any known vulnerabilities in these?

• What build system was used for every artifact running in production? What version? Are there any known vulnerabilities in these?

• What mitigation strategies do we have in place today for compromised dependencies in our build systems? Do we have anything to detect persistence, like in the Sunburst attack?

‍

Slimming production images and runtimes is always a best practice, but it only gets you so far. Toolchains are just as critical to your overall security posture, and trimming them is much much harder. You’ll likely need a different strategy to keep these up to date, secure, and accounted for in your overall asset management systems.

‍

Let us help!

Now for some good news. We’ve spent a lot of time at Chainguard thinking about these requirements and building tooling, like Chainguard Images, that makes it easy to comply without dramatically changing the way your team builds software today.

‍‍

At the ground-level, CVE triage and vulnerability management is a major pain point for organizations who are going to have to comply with the self-attestation form. What can you do? Use hardened, minimal container images that are frequently rebuilt and contain a minimal attack surface. These are the core benefits of our Chainguard Images product, which are minimal container images that only contain what is required to build or run your application, delivering on average a 97.6% reduction in CVEs. Recent Chainguard Labs research found that popular container images used for most software built today, when not updated, accumulate one known vulnerability per day. If organizations are going to certify they comply with CISA’s self-attestation form and maintain the required software security posture, one vulnerability per day can cost infinite time and resources from developer and engineering teams to ensure an organization’s compliance. There is a better way, and Chainguard Images can help.

‍

Chainguard Enforce delivers full visibility into what’s in your software supply chain. For software integrity and verification, Enforce provides capabilities to create policies for requiring signatures and SBOMs for all software artifacts. The platform also allows you to improve time to remediation when new vulnerabilities are disclosed because you have visibility into where vulnerable software is running in your environments.

Let us help you fortify, comply and conquer these supply chain security requirements. Get in touch today.

‍

Looking to understand more about the current regulatory environment for software development, specifically the upcoming self-attestation requirements? Watch this session with Chainguard CEO Dan Lorenc and CISA fellow Chris Hughes: