In this post, we are going to walk through how we make Chainguard Images reproducible without the pesky problem where the images show up as created in 1970.
A little bit of background for the uninitiated. Typically container images carry a “created_at” timestamp in their “config file” reflecting when the container image was built. However, since this config file’s hash is part of a Merkle tree, which allows the container to be pulled by digest and how git works, any non-determinism in this timestamp results in the digest for the image changing even if everything else remains the same.
To address this non-determinism, a common practice became to set the timestamp to what’s called the Unix Epoch, which is midnight of January 1st 1970. above, this has led to all sorts of “fun” UX problems over the years.
So why don’t we just use “source date epoch”?!?! It sounds great!
In fact, we do! When we perform our APK builds for Wolfi, and our downstream enterprise Chainguard APKs, we encode the timestamp of when we last changed our packaging configuration file as the “source date epoch”:
Since each package pins to a particular version of the software, it only changes when the source or configuration changes (similar to the Merkle tree above):
So what’s a “build date epoch” then…?
Things get a bit more complicated when we produce the Chainguard Images from these APKs because when we produce our Images we generally want to pick up the latest versions of these packages, often in order to fix CVEs. What this means is that unlike above, the timestamp of the configuration file does not reflect the timestamp we should use because the configuration file itself changes very rarely, but it will still continue to pick up newer packages than the configuration file itself. If the configuration file does change, then the image digest will change as well.
In essence, “build date epoch” enables us to achieve a transitive form of “source date epoch,” and because of this the image digests will largely change only when one of their packages has a new version.
Want to learn more about how we build Chainguard Images? Watch our live demo with Chainguard CEO Dan Lorenc or visit Chainguard Academy.
