The security costs of base image version loitering

Why Not Updating Your Base Image Means More Vulnerabilities

‍Just like some people don’t like change, some software teams don’t like updating their container base images. The base image is the first layer of container-based software applications. Updating a base image to a new version can sometimes break the software application, especially if there is inadequate test coverage. So some software teams come to prefer the status quo, essentially loitering indefinitely on a working base image version.

‍

Base image version loitering has a cost though: an ever-increasing accumulation of known software security vulnerabilities. Security teams can measure this cost by using container security scanners to analyze the version-by-version security costs of not upgrading. This blog post demonstrates this technique by analyzing several popular base images and finds that base image version loitering can sometimes increase an application’s vulnerability count by an order of magnitude.

‍

To avoid this accumulation of vulnerabilities, software teams ought to update frequently with small changes, a practice made easier by using Chainguard Images, and pair this approach with “testing in production” practices like canary releases. These techniques strike the right balance between staying secure and making sure production doesn't go down.

‍

The Costs of Version Loitering

To illustrate the security costs of not updating a base image to the newest version, this post quantifies the number of publicly reported security vulnerabilities in three popular base images (NGINX, Debian, and Node) across several versions, including the most up-to-date version and several older versions. The main analysis uses two different container security scanners (Trivy and Snyk) to ensure that the results don’t reflect the idiosyncrasies of one particular scanner. All scans were performed in late May or early June 2022.

‍

Figure 1 presents the main analysis. Each pane presents the results for a different base image. Within each pane, the horizontal bars are arranged top to bottom from the newest version to the oldest version. The two scanners are represented by different color bars.

‍

Figure 1. Vulnerability Count by Base Image Version and Security Scanner

Depending on the base image and the scanner, the results suggest that base image version loitering can increase vulnerability counts anywhere from approximately 200 to 2500 percent. The NGINX:alpine base image analysis, for instance, shows that failing to update the minor version (from 1.18.0 to 1.20.2) can lead to a significant increase in reported vulnerabilities, from less than a handful to, depending on the scanner, 30 or 50. There are similar, if less eye-popping, results for the Debian base image and the Node base image.

‍

In other words, not updating your base image might avoid breakage but also incurs a security cost, a cost that this analysis makes visible. To be sure, these results do not necessarily hold for all base images across all versions. Only a detailed analysis of the particular base image on which an application relies can a security team know the costs, but these results do suggest that base image version loitering can have steep costs.

‍

The Costs of Updating

So if loitering is bad for security, why do teams do it? The answer is straightforward: there is also a cost to updating. Every team has stories about making a seemingly innocuous change to a patch version of a dependency that brought down production. This understandably leads to scar tissue in the form of extra checks and barriers to performing updates. In the best case, these checks slow down the application of updates. In the worst case, software teams loiter on a version until security incidents or incompatibilities force their hand.

‍

This leaves us with a conundrum: updating breaks things and not updating leaves us vulnerable. The answer, perhaps unintuitively, is to continuously update, as in multiple times a day, if not more. The more software teams update, the better they get at it and the more they are able to deal with the issues that arise. Further, continuous updates contribute to build and code health, so the secure practice actually improves the team's velocity and confidence.

‍

There are several helpful techniques and a variety of tooling that can make continuous updating a viable option:

‍

• Use continuously updated container base images like Chainguard Images that aim for zero known vulnerabilities. This can vastly reduce the work required by teams investigating and mitigating vulnerabilities in images reported by security scanners.

• Invest in automated integration, system and unit tests. Tests are essential and should catch the vast majority of bugs.

• Look into techniques for high-velocity, high-assurance development, including those detailed in the Google Site Reliability Engineering book such as canary releases, feature flags, and rollbacks. These help ensure any issues are caught early and affect as few end users as possible.

• Employ observability tooling to identify problems or performance issues as soon as possible.

So stop loitering and start updating!

‍

You can get in touch with Chainguard and learn more about Chainguard Images by filling out this short form.