Zero-friction “keyless signing” with Kubernetes

In this post, you will learn about a new way of signing container images without needing to manage signing keys, including a demo of how easy it is to get started signing on Amazon EKS.

So what is “keyless signing”?

Keyless signing is a new signing technique where you never handle long-lived signing keys (ergo “keyless”). It builds around the ideas of Certificate Authorities (aka CAs) and certificate chains that we use to secure web connections today (see also WebPKI). During the keyless signing process a short-lived certificate is generated, and linked into the chain of trust by completing an identity challenge to confirm the signatory’s identity. These short-lived keys only live long enough for the signing to occur, and signature verification must verify that the certificate was valid when the signing occurred. To support policy enforcement, the certificate encodes the identity information from the challenge, so we know the identity of the signatory.

So why is “keyless” better than conventional signing?

One of the key things (no pun intended) we have been educating folks about as Supply Chain attacks surge is that you should operate your build environment like a production environment. It is a widely accepted production best practice to use “short-lived” credentials, but the typical signing process today involves checking out a long-lived signing key from a KMS (e.g. Vault), signing something, and then scrubbing any references to the long-lived key material. The use of long-lived signing keys makes this susceptible to “exfiltration attacks” where these long-lived keys are stolen and used to sign malicious things. These “exfiltration attacks” are even more damaging for signing keys than for credentials because authorization of credentials is often an “online” process, so revocation is “easy”; however, with signing keys verification of signatures often happens “offline” with a public verification key. Keyless enables this signing process to happen with exclusively short-lived credentials: it uses short-lived identity tokens to provision short-lived signing keys.

Enter Project Sigstore. The goal of Sigstore is to do for signing what Let’s Encrypt did for TLS. Sigstore’s Fulcio project lets folks operate a signing CA that issues short-lived certificates based on OpenID Connect (OIDC) identity challenges (similar to Let’s Encrypt’s ACME challenge protocol). The identity challenge flow can be completed two ways:

• The “human” way is to go through a web-based authorization (aka 3LO) flow (awesome demo).

• The “workload” way is to send an OIDC token (our focus here).

The Fulcio project README captures a key point about signing with short-lived keys:

Fulcio is designed to avoid revocation, by issuing short-lived certificates. What really matters for CodeSigning is to know that an artifact was signed while the certificate was valid.

This can be done a few ways:

• Third-party Timestamp Authorities (RFC3161)

• Transparency Logs

• Both (Fulcio's Model)

The Project Sigstore community also operates a public Fulcio instance to enable folks to sign public packages with verified identities. Fulcio supports a number of different sources of workload OIDC tokens (e.g. SPIFFE, Github, Google), but the one we want to talk about today is the latest: Kubernetes!

Kubernetes has a feature called Service Account Token Volume Projection that went stable in 1.20, which lets you project an OIDC token for the Pod’s service account into the container’s filesystem. On its own, this is a useful feature for on-cluster service-to-service authentication, but what about off-cluster authentication? Amazon (EKS) and Google (GKE) have taken this to a new level by making the “Issuer URL” for these tokens public (with AKS coming soon). This means that these tokens can be verified and used for off-cluster authentication (AWS, GCP)!

In addition to adding support for the Kubernetes OIDC tokens, we also configured the public Fulcio root to accept OIDC tokens from any EKS or GKE clusters’ issuer, so keyless signing has never been easier. Let’s walk through a little demo!

I am a BIG fan of Amazon’s new Graviton machines, so I spun up a new cluster with:

eksctl create cluster \
  --name ez-mode-signing \
  --version 1.21 \
  --with-oidc \
  --nodes 1 \
  --node-type m6g.xlarge  # Graviton rocks!

We are going to use Sigstore’s “cosign” tool to sign a container image in ECR, which will publish the signature alongside the image (look for .sig tags), so let’s create a service account with an IAM role that can write to ECR to do this:

eksctl create iamserviceaccount \
  --name ecr-pusher \
  --namespace default \
  --cluster ez-mode-signing \
  --attach-policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser \
  --approve

More information about the managed IAM role AmazonEC2ContainerRegistryPowerUser

Next let’s create a little signing “Job” that will invoke cosign, fill in < YOUR IMAGE HERE > with an image in ECR you want to sign:

apiVersion: batch/v1
kind: Job
metadata:
  name: check-oidc
spec:
  template:
    spec:
      restartPolicy: Never
      automountServiceAccountToken: false
      serviceAccountName: ecr-pusher
      containers:
      - name: check-oidc
        image: gcr.io/projectsigstore/cosign:1.3.0
        args: [
            "sign",
            # Upload to Rekor even though this is a private image.
            "--force",
            # Use workload identity to push to ECR
            "--k8s-keychain",
            # The image in ECR to sign.
            < YOUR IMAGE HERE >
        ]
        env:
        - name: COSIGN_EXPERIMENTAL
          value: "true"
        volumeMounts:
        - name: oidc-info
          mountPath: /var/run/sigstore/cosign
      volumes:
        - name: oidc-info
          projected:
            sources:
              - serviceAccountToken:
                  path: oidc-token
                  expirationSeconds: 600 # Use as short-lived as possible.
                  audience: sigstore

Once you have created this file with your image name, kubectl apply -f <file>. You can see a pod spin up and complete with kubectl get pods. Once it completes, you can use kubectl logs jobs/check-oidc to get the logs, you will see something like this:

Generating ephemeral keys...
Retrieving signed certificate...
Successfully verified SCT...
tlog entry created with index: 810351

If you have cosign on your machine, you can now verify your image’s signature with:

COSIGN_EXPERIMENTAL=true cosign verify YOUR IMAGE HERE

At the end of the JSON blob this emits you should see the identity of your workload, which was embedded in the certificate:

"Issuer": "https://oidc.eks.REGION.amazonaws.com/id/...",
"Subject": "https://kubernetes.io/namespaces/default/serviceaccounts/ecr-pusher"

On AWS, you can follow the instructions here to map between clusters and OIDC issuers.

At this point you hopefully understand what “keyless signing” is, and how simple it is to start signing your container images on EKS! Now that you know how to sign your container images on EKS, see Dan and Scott’s post on verifying your containers are signed before deploying them to your cluster (leave the verification-key secret empty to tell cosigned to do keyless verification!).