Secure your software factory with melange and apko
The GitHub repo associated with this blog post can be found here.
This posted was updated on September 9, 2022 to remove instructions relating to APK repo indexes which are no longer necessary with the latest version of melange.
What is a “Secure Software Factory”?
For simplicity, let's imagine a typical brick-and-mortar factory that produces toys. A software factory is sort of like that, but at the end of the assembly line instead of boxes of stuffed animals we get container images. Whether or not our factory can be considered “secure” depends on how confident we are that those container images were not tampered with and are safe to use.
The term “Secure Software Factory” (SSF) is in reference to a whitepaper released earlier this year by the CNCF Security TAG. This document provides a reference architecture for system architects to use for designing build pipelines which mitigate numerous supply chain security risks. These risks are outlined in detail in another whitepaper by the same group titled “Software Supply Chain Best Practices”.
We wrote a little bit about how Citi is building a Secure Software Factory in another blog post.
Building a Secure Software Factory is no small feat. There are many, many ways in which bad things can make their way into your release artifacts during the build process. From compromised credentials to compromised build nodes, attackers can get very creative in these environments.
Even considering an attacker was never able to modify a release artifact, these artifacts may contain components subject to future vulnerabilities. Do you know every version of every component inside your artifacts to mitigate this later? (see “SBOM”)
So one might ask: where do I start? The answer to this could really apply in any situation where there is an insurmountable amount of work to be done: start with low-hanging fruit.
(For those unfamiliar, “low-hanging fruit” is defined by Merriam-Webster as “the obvious or easy things that can be most readily done or dealt with in achieving success or making progress toward an objective”)
Image build: the low-hanging fruit
One of the lowest hanging fruits on this tree of doom is the image build. What tools are you using to build your images? Are you using Docker? Dockerfiles? Where are your base images coming from? (Remember, it's All About That Base Image!)
In containerized environments, the majority of vulnerable components will either surface during the building of the image or the running of the image. If you’re using a vulnerable base image, you could be doing everything else right and still get burned.
So, sure, you can start writing all sorts of fancy security policies which might quarantine insecure images if/when a vulnerability is detected, but how about starting to produce secure images from day 1?
Essential machines for the factory: melange and apko
Every factory has machines, right? Some machines are outdated and inefficient, while others are new and state-of-the-art. We introduce you to 2 incredible machines that we believe belong in every Secure Software Factory: melange and apko. These 2 tools in combination provide for a reproducible, declarative approach to building OCI images. Did we mention, these machines are FREE!? (They’re open-source, of course!)
melange
melange allows you to build APKs using declarative YAML pipelines. “APKs” refer to .apk packages compatible with apk (the package manager used by Alpine), similar to .deb or .rpm.
apko
apko allows you to bundle a collection of APKs into an OCI image using a declarative YAML manifest.
Check out our blog post which originally introduced apko.
Nothing in your images but APKs
You’re probably familiar with Dockerfiles where you can “curl | bash” whatever you want. It depends on the context, but we are of the opinion that this type of thing is probably bad. Many Dockerfile-produced images also rely on a base image, which may or may not contain vulnerabilities.
OCI images are the required “shape” of things which run in containerized environments such as Kubernetes. However, there is no required “shape” things must be in order to make their way into an OCI image.
apko addresses this issue very simply: everything must be an APK.
This sounds neat in theory, but not everything you need is packaged as an APK, including your custom applications. This is where melange comes in.
Show me the code!
Alright then! Currently melange and apk require apk-tools on Linux to run properly, so for the sake of simplicity, this example will use docker to run container images which contain these tools and all their dependencies.
The source for the following example can be found here. If you want to follow along, the remainder of the steps will assume you’ve done the following:
git clone https://github.com/chainguard-dev/hello-melange-apko.git
cd hello-melange-apko/go/
Let’s build an APK for a custom Go application, a simple HTTP server that responds “Hello World!”. Here’s the main.go:
package main
import (
"net/http"
"github.com/gin-gonic/gin"
)
func main() {
r := gin.Default()
r.GET("/", func(c *gin.Context) {
c.String(http.StatusOK, "Hello World!")
})
r.Run() // listen on 0.0.0.0:8080
}
Note: this directory should also include valid go.mod/go.sum files which reference the dependency on the gin framework.
To package this application up as APKs (one for each target architecture), we can define a melange.yaml as follows:
package:
name: hello-server
version: 0.1.0
description: friendly little webserver
target-architecture:
- all
copyright:
- license: Apache-2.0
paths:
- "*"
environment:
contents:
repositories:
- https://dl-cdn.alpinelinux.org/alpine/edge/main
- https://dl-cdn.alpinelinux.org/alpine/edge/community
packages:
- alpine-baselayout-data
- ca-certificates-bundle
- busybox
- go
pipeline:
- name: Build Go application
runs: |
CGO_ENABLED=0 go build -o "${{targets.destdir}}/usr/bin/hello-server"
The 3 top-level sections in the file are:
package - metadata about the APK
environment - packages required for building this APK
pipeline - series of steps to run which ultimately populate ${{targets.destdir}} (the root of the APK)
As we will be signing our packages and repo index, first generate a temporary melange keypair:
docker run --rm -v "${PWD}":/work cgr.dev/chainguard/melange keygen
Next, run the following to generate the APKs:
docker run --rm --privileged -v "${PWD}":/work \
cgr.dev/chainguard/melange build melange.yaml \
--arch amd64,aarch64,armv7 \
--signing-key melange.rsa
Note: the –privileged flag is required currently due to the use of bubblewrap internally which requires various Linux capabilities.
If melange ran successfully, you should end up with the following:
$ tree packages/
packages/
├── aarch64
│ ├── APKINDEX.tar.gz
│ └── hello-server-0.1.0-r0.apk
├── armv7
│ ├── APKINDEX.tar.gz
│ └── hello-server-0.1.0-r0.apk
└── x86_64
├── APKINDEX.tar.gz
└── hello-server-0.1.0-r0.apk
These APKs are now ready to be installed on an apk-based distro like Alpine.
Next, let’s take a look at our apko.yaml, which will bundle our custom APKs into OCI images:
contents:
repositories:
- https://dl-cdn.alpinelinux.org/alpine/edge/main
- /work/packages
packages:
- alpine-baselayout-data
- hello-server
accounts:
groups:
- groupname: nonroot
gid: 65532
users:
- username: nonroot
uid: 65532
run-as: 65532
entrypoint:
command: /usr/bin/hello-server
Notice that we have defined a local repository at the path /work/packages. So we will mount the current working directory to /work so that apko is able to find the hello-server package.
Run the following to build an OCI image with the tag factory-demo exported to output.tar:
docker run --rm -v "${PWD}":/work \
cgr.dev/chainguard/apko build --debug apko.yaml \
factory-demo output.tar -k melange.rsa.pub \
--arch amd64,aarch64,armv7
A quick side note here: apko generated images are reproducible - run the same build with the same inputs and you will get bitwise identical output. This is something that is currently impossible to do with Docker. If you were to run the command above a second time using “output2.tar”, you should be able to verify these files are identical:
diff output.tar output2.tar && echo "Files identical"
If the apko build was successful, you should be able to load the tarball into Docker:
docker load < output.tar
Now we can run it!
docker run --rm -it --rm -p 8080:8080 factory-demo
Then in another terminal window, try to hit the server using curl:
$ curl -s http://localhost:8080
Hello World!
That’s it! If you were to use apko publish to push the image instead, apko will automatically attach an SBOM containing all of the APKs included in the image! (This is probably grounds for a whole other blog post).
But what if I don’t use Go?
That is no problem! APKs are general purpose that can be used to package really anything.
Just to prove that point, we put together duplicate demo apps that you could be used in place of the Go example, located in the same GitHub repo:
Just clone down the repo, enter the directory of the variant you wish to try, and you should be able to run the example commands from the previous section verbatim.
This is amazing!
You know what? We agree! We agree so much that we’ve started to put together a collection of GitHub Actions that will help automate all of these steps. Also, if you like melange and apko, throw ‘em a star on GitHub!
In a recent blog post, we introduced the new distroless project. All of the images produced by the distroless project are created using nothing but melange and apko!
So if you’re feeling like your software factory could use a bit of modernization, here’s a fantastic place for you to get started.
Ready to Lock Down Your Supply Chain?
Talk to our customer obsessed, community-driven team.