Building minimal and low CVE images for compiled languages

The first wave of containerization was a revelation. You could download third-party applications like Redis or nginx and have them running in seconds with no configuration. You could put your application on top of a base image like Debian or Ubuntu, ship it to people and be confident that it would work exactly the same for them as it did for you. We had something with the isolation and portability of a virtual machine (VM), but at a fraction of the size.

‍

The first wave was a long time ago though, and it's now possible to create containers a fraction of the size of containers from the first wave. This can be achieved by separating build and runtime concerns and using a stripped down base image — instead of a full Debian, Ubuntu, or RedHat base image — for deployment. In this blog we'll look at an example of this process with Chainguard Images and how the final result is much smaller and more secure.

‍

As part of our Learning Labs series, I created an exemplar Go application that runs a simple Go web server application. The starting Dockerfile looks like this:

FROM golang

WORKDIR /work

COPY go.mod /work/
COPY cmd /work/cmd
COPY internal /work/internal

RUN go build -o hello ./cmd/server

ENTRYPOINT ["/work/hello"]

This Dockerfile uses the golang image from Docker Hub, which is based on the Debian base image (via some intermediate images that add build tooling). This isn't a bad image and it's similar to a lot of other Dockerfiles out there.

‍

But we can do a lot better. There are two main issues with this build:

‍

1. It's large. At the time of writing, this results in an 892 MB image. The vast majority of this is coming from the base image and the build tooling, not the application itself. Docker Scout tells me there are 303 packages installed in this image, yet our running application only requires a handful of these.

2. ‍It has Common Vulnerabilities and Exposures (CVEs). Scanning with Docker Scout suggests there are 42 low vulnerabilities (Grype suggests a lot more — including critical and high vulnerabilities — but many of these have been marked as "won't fix" indicating the provider has investigated and assessed them as not relevant).

‍

Let's start with a really simple change. If we just change the FROM line at the top to use the free Chainguard Images for Go:

FROM cgr.dev/chainguard/go
…

If we build this Image you will find:

‍

• The size reduces from 892 to 775MB. Still a large image, but a significant saving for a one-line change.

• The CVE count goes to 0. For both Docker Scout and Grype scans. This is a big shift and can represent a big saving in effort if you previously tried to triage and remediate the results from vulnerability scanners.

‍

Another alternative is to use the Alpine variant of the image, which results in a large reduction in size. Do be aware that the Alpine variant is considered experimental and not supported by the Go project (see the golang image overview on DockerHub for more details).

‍

A better option to reduce the size is to use a multistage build, so the final image doesn't include all the build tooling such as the Go compiler. In our example, this would look like:

FROM cgr.dev/chainguard/go:latest-dev as builder

WORKDIR /work

COPY go.mod /work/
COPY cmd /work/cmd
COPY internal /work/internal

RUN CGO_ENABLED=0 go build -o hello ./cmd/server

FROM cgr.dev/chainguard/static
COPY --from=builder /work/hello /hello

ENTRYPOINT ["/hello"]

The basic idea is that we are keeping the same build as before, but adding on a second build where we copy out the build artifact — the server binary — and put it in the much smaller cgr.dev/chainguard/staticimage for running in production.

‍

We had to add the environment variable CGO_ENABLED=0to the build instruction in order to produce a static binary. Without this, Go will produce a binary that is dynamically linked against system libraries, including libc. Just using this environment variable is enough to produce a static binary in this case, but in other situations you might find you need to pass a few more arguments to get a completely static binary, as covered in this blog by Matt Turner.

‍

The result of this technique is pretty drastic. The size of the final image is reduced to just 8.51MB, yet the application works just the same as it did before. The reduced size doesn't just make the image easier to transfer and quicker to start, it also improves the security posture as we're no longer including software that isn't needed but could be leveraged by attackers (for example in a living off the land attack).

‍

The Chainguard Static Image used in the above build is an extremely minimal image. It does include a few things that are needed by the majority of applications, including TLS certificates for talking securely to other applications, a directory structure, and a defined user. The Image is so minimal that even the shell and package manager have been left out. This approach to minimalism is often called "distroless," after the Google project which started it. The Google Distroless project was started by Matt Moore and Dan Lorenc, who are now CTO and CEO of Chainguard, respectively.

‍

Static vs. dynamic

Static binaries are completely stand-alone binaries that include all libraries required to run them. Dynamic binaries, in contrast, require other libraries to be present on the system in order to run.

‍

The Chainguard Static Image does not include any libraries — not even the base libc library. This means it can only be used by compilers that are capable of creating completely static binaries. The above example used Go, but we can do something very similar in Rust by using the musl target:

$ rustup target add x86_64-unknown-linux-musl
$ cargo build --release --target=x86_64-unknown-linux-musl

Note, this will mean your application is linked against the musl C library. If this isn't what you want, we also provide base images that include a handful of common libraries such as glibc-dynamic and cc-dynamic. You may find that these work even better for your use case and still provide a very small base image.

‍

Going back to the Go example, it is possible to automate a lot of the image building process with the ko tool, which can build minimal images for most Go projects with next to no configuration.

‍

Get started with Chainguard Images today

Moving a project to use Chainguard Images is typically painless and is often just a one-line change that results in reduced size, vastly reduced CVE counts, and an improved security posture.

‍

Adopting a "distroless" approach where images are really pared down to the bare essentials can be a little trickier, but is often worth the effort, as it vastly reduces the size of the image and the attack surface. This also helps prevent future vulnerabilities — the less software in the image, the less chance of being affected by CVEs in the future. Contact us to learn more.