Life of a Sigstore signature

Based on a presentation at SigstoreCon 2022; Slides available here.

‍

Introduction

When you ask, “How does Sigstore work?” someone will often “helpfully” send you a graphic like this:

While visualizations like these are very useful once you already have your footing in the Sigstore ecosystem, we found that it’s actually quite difficult to get to that point. Even after we thought we understood Sigstore, trying to debug subtle issues quickly disabused us of that notion.

‍

The fundamental problem here is that it’s really hard to understand something that you can’t touch. While obviously a “new standard for signing, verifying, and protecting software” isn’t actually tangible, it doesn’t have the hands-on feeling that you get when you see an HTTP request in Wireshark or tcpdump for the first time.

‍

So we thought to ourselves: what would a “tcpdump” of Sigstore look like? In this post, we’ll show you! You’re encouraged to follow along at home, as well.

‍

We’re going to sign the file data.txt, with the following content:

‍

hello sigstorecon!

‍

And we’ll trace the flow of the above diagram as we do it! By the end, you should be able to see what each of those components are.

‍

Setup

To follow along, you’ll need some basic tools:

‍

• A standard build environment with git and make. Most of you will have this already.

• go 1.19 (other versions may work, but weren’t tested explicitly).

• jq for manipulating JSON data.

• step for inspecting X.509 certificates (you can do the same with openssl, but we like step for its ergonomics).

‍

You should be familiar with the basic usage of Sigstore, including Cosign, but it’s okay if you’re shaky on some of the details of how it works.

‍

Following the life of a Sigstore signature

To sign the data, we’ll use Cosign, as usual:

export COSIGN_EXPERIMENTAL=1
cosign sign-blob \
    --output-certificate crt.b64 \
    --output-signature sig.b64 \
    data.txt

Easy enough. This will invoke the “keyless signing” flow and pop open a browser, which you can use to authenticate:

Then, you can verify the signature against my identity (in this case, I logged into Sigstore with my Google account):

export COSIGN_EXPERIMENTAL=1
cosign verify-blob --certificate crt.b64 --signature sig.b64 \
    --certificate-email zjn@chainguard.dev \
    --certificate-oidc-issuer https://accounts.google.com \
    data.txt

There’s one problem, though. Out of the box, Cosign will give you the certificate (crt.b64) and signature (sig.b64). (Don’t worry, we’ll go over these in detail later). But there’s a bunch of other stuff in the diagram (the certificate request, the HashedRekord, the LogEntry, the JWT) that’s missing.

‍

This is where tcpdump would usually come in, but the connections between Cosign and the Sigstore infrastructure are encrypted (which is good!). There are tricks to get around this, but the simplest is to just have Cosign itself print them out. This is easy enough to do by patching Cosign.

‍

While the patch is pretty simple (about 20 lines, just dumping data to files at the appropriate places), you can also just use my fork of Cosign that does this for you.

git clone -b lifeofa https://github.com/znewman01/cosign
cd cosign && make && cd ..

Then we can run it!

COSIGN_EXPERIMENTAL=1 \
./cosign/cosign sign-blob \
    --output-certificate crt.b64 \
    --output-signature sig.b64 \
    data.txt

We now have a directory chock full of interesting Sigstore tidbits: cert-req.json, crt.b64, data.txt, hashedrekord.json, jwt, logentry.json, sig.b64. Soon, you’ll know what they all are.

‍

Step 1: OIDC

Sigstore uses OpenID Connect to verify your identity. You can think of this as the protocol that enables “log in with Google.” The first step in the diagram is the “OIDC dance,” a silly name for a relatively complicated set of steps. In the dance, my browser pops open, I click “allow,” and Cosign gets a token that allows it to convince Sigstore that I am really zjn@chainguard.dev.

‍

What is this token? It’s a JSON Web Token (JWT, pronounced “jot”). A JWT is really just three base64 blobs, the first two of which encode JSON data, separated by a dot. We can parse it as follows:

echo "HEADER";
cut -d. -f 1 jwt | base64 -d | jq
echo -e "\nPAYLOAD"
cut -d. -f 2 jwt | base64 -d | jq
echo -e "\nSIGNATURE (redacted)"
cut -d. -f 3 jwt | tr 'a-zA-Z0-9_-' 'a'

The full result is in this Gist , but we’ll look at the highlights here. First is the header:

{
  "alg": "RS256",
  "kid": "0670f2c4d2c19fdf7486d96eccf02a9d4d26bd9b"
}

This header just tells Sigstore how to verify the token. Sigstore will use a 256-bit RSA public key (the algorithm alg) published by the identity provider to verify the payload. The key id (kid) is a hint to Sigstore that helps it pick which public key to use when there’s more than one.

{
  "iss": "https://oauth2.sigstore.dev/auth",
  "federated_claims": {
	"connector_id": "https://accounts.google.com",
	[...]
  },
  "email": "zjn@chainguard.dev",
  "aud": "sigstore",
  "iat": 1666309800,
  "exp": 1666309860,
  [...]
}

The payload is the interesting part! First, look at the issuer (iss). You might think that it would be Google. But Sigstore uses Dex, a federated OIDC identity provider, as part of its infrastructure. Dex is a proxy that sits in between a traditional OIDC provider, like Google, and Sigstore. It translates from a Google OIDC token to one that Sigstore understands. Different identity providers each have their own quirks, and Dex smooths those over, giving Sigstore consistency.

‍

That’s what federated_claims is about: the connector_id indicates which upstream identity provider (Google, in this case) the original token came from.

‍

Then, we get to the email which, unsurprisingly, contains the email of the account used to request the token.

‍

The remaining fields try to minimize the blast radius of a leaked token. The issued time (iat) and expiry time (exp) dictate when the token is valid. In this case, it’s only for 1 minute. The audience (aud) says “anybody who isn’t Sigstore should ignore this token.” This means that even if someone steals the token, all they can do with it is get certificates from Sigstore, not read your email—and they can only do this for a minute!

‍

Despite what I just said about the security features of the OIDC token, I’m not brave enough to share a real, signed token with you. In theory, there’s no risk. But in practice, JWT validation errors abound, including failures to validate both the aud claim and the exp claim.

‍

Step 2: Fulcio

The next step is to make a temporary public/private key pair and get a certificate. This certificate says, “the person with this public key is zjn@chainguard.dev.” Anybody can make a certificate, but you usually only respect certificates signed by somebody that you trust—a “certificate authority” (CA).

‍

Fulcio is the CA in the Sigstore ecosystem. It issues certificates automatically when it sees valid OIDC tokens. If somebody trusts Fulcio, they can use certificates that it issues to verify that a signature on a particular artifact belongs to you.

‍

Sigstore certificates are in X.509 format, much like the certificates you use to connect to your bank and other web sites.

‍

Requesting a certificate

Once we have the OIDC token in hand, we generate a key pair (very similarly to ssh-keygen). We then format a request for a certificate to send to Fulcio:

jq '.' cert-req.json

{
  "publicKey": {
    "content": "MFkwEwYHKoZIzj0CAQYIKoZIzj0[...]",
    "algorithm": "ecdsa"
  },
  "signedEmailAddress": "MEUCIQDBo6190831GsVZNDKG8gm[...]",
  "certificateSigningRequest": null
}

The full request is in this Gist. It contains a public key along with a signature over the email in the OIDC token (by the corresponding private key); this proves that you control the public key. Alternatively, you can send an PKCS#10 Certificate Signing Request (the missing certificateSigningRequest field), but this is what Cosign does today.

‍

The issued certificate

After we send this request, along with the OIDC token, we get a certificate in return! Fulcio is convinced by the OIDC token that I am indeed zjn@chainguard.dev and is happy to give me the certificate. It’s a base64-encoded and PEM-encoded X.509 certificate, which we’ll investigate using step:

base64 -d crt.b64 | step certificate inspect

You can see the full output in this Gist, but here are some highlights:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 319504054616999573366109868063946115769004630327 (0x37f70eac2af16f7c708881885b1c0f7cfd06a137)
    Signature Algorithm: ECDSA-SHA384
        Issuer: O=sigstore.dev,CN=sigstore-intermediate

It starts with standard certificate data: this is issued by sigstore.dev.

Subject Public Key Info:
    Public Key Algorithm: ECDSA
        Public-Key: (256 bit)
        X: 7f:6f:9f:60:d7:20:16:2c:67:[...]
        Y: 26:26:9f:e1:1c:c9:17:0c:82:[...]
        Curve: P-256

This is the key we just generated for Fulcio (trimmed for space), corresponding to the MFkwEwYH[...] string in the certificate request above.

Validity
    Not Before: Oct 20 23:50:00 2022 UTC
    Not After : Oct 21 00:00:00 2022 UTC

The certificate is valid for only 10 minutes! This is very short for a certificate. The short validity period means that even if the key that we just generated is compromised, it quickly becomes useless.

X509v3 extensions:
    X509v3 Subject Alternative Name: critical
        email:zjn@chainguard.dev
    1.3.6.1.4.1.57264.1.1:
        https://accounts.google.com

This is the identity in the certificate. In this case, Fulcio has signed a note saying “Google told me that the person controlling this public key is zjn@chainguard.dev.” The long string of numbers is an object identifier (OID), used to ensure that extensions to X.509 don’t conflict; here, Sigstore has registered one for the OIDC identity provider behind the certificate.

‍

Step 3: Submitting to Rekor

Earlier, we made a big deal about how the certificate we got from Fulcio was only valid for 10 minutes. But we want artifacts to be valid for more than 10 minutes!

‍

In Sigstore, we separate the lifetime of an artifact from the lifetime of a certificate. To do this, we require a trusted party to provide timestamps, which we call Rekor. Rekor also stores metadata about our artifact signature, which allows for users to search for artifacts, such as those signed by specific individuals.

‍

Preparing the Rekor entry

Rekor doesn’t store the data that we’re signing, for a few reasons. First, that data can be prohibitively large, and may be private. Second, users may want to be able to search Rekor using artifact metadata. This metadata is unverified (Rekor doesn’t check it) but can still be useful for locating artifact metadata.

‍

Instead, Rekor stores different “pluggable types.” The most common one is a HashedRekord, which Cosign uses. Others include RPMs, which describes RPM packages and includes package header data, or in-toto attestations, which can be used as part of an in-toto layout. Here’s our formatted HashedRekord (trimmed; full output at this Gist):

jq '.' hashedrekord.json

{
  "apiVersion": "0.0.1",
  "spec": {
    "data": {
      "hash": {
        "algorithm": "sha256",
        "value": "2e07a09219d831fd0193e6176d3[...]"
      }
    },
    "signature": {
      "content": "MEQCIBegNVAfmVLratPX81Eb821[...]",
      "publicKey": {
        "content": "LS0tLS1CRUdJTiBDRVJUSUZJQ0F[...]"
      }
    }
  },
  "kind": "hashedrekord"
}

Some of these things look should look familiar (the diff commands won’t print anything, because the inputs are the same):

diff <(jq -r '.spec.signature.publicKey.content' hashedrekord.json | tr -d '\n') \
     crt.b64

diff <(sha256sum data.txt | awk '{print $1}') \
     <(jq -r '.spec.data.hash.value' hashedrekord.json)

diff sig.b64 \
     <(jq -r '.spec.signature.content' hashedrekord.json| tr -d '\n')

Submitting the Rekor entry

Cosign then sends that HashedRekord to Rekor. In return, we get a SignedEntryTimestamp, which is a signed receipt from Rekor saying that it has incorporated our entry:

jq '.' logentry.json

The full output is in this Gist; we’ll go through the highlights:

{
  "body": "eyJhcGlWZXJzaW9uIjoiMC4wLjE[...]",
  "integratedTime": 1666309800,
  "logID": "c0d23d6ad406973f9559f3ba2d1[...]",
  "logIndex": 5526036,
  [...]
}

And indeed, the entry has made it into Rekor!

‍

It’s the same data that we sent up:

diff <(jq -r '.body' logentry.json | base64 -d | jq -S) \
     <(jq -S '.' hashedrekord.json)

The verification information is used to verify that this entry came from Rekor:

{
  [...]
  "verification": {
    "inclusionProof": {
      "checkpoint": "rekor.sigstore.dev - 2605736670972794746\n1362606\nVfFiKYaBgKCOR8zL0meDALlhti[...]\nTimestamp: 1666309800864291232\n\n— rekor.sigstore.dev wNI9ajBEAiBJihQuWQ1esL/vLgo[...]\n",
      "hashes": [
        "35a3138a718675d79cf067372d6[...]",
        "9884533ab96cbf8708424c68e4b[...]",
        [...]
      ],
      "logIndex": 1362605,
      "rootHash": "55f16229868180a08e47cccbd26[...]",
      "treeSize": 1362606
    },
    "signedEntryTimestamp": "MEUCIQD7qTkluFaxRpjfgVSxvU9[...]"
  }
}

We mostly care about the signedEntryTimestamp field. This is a signature from Rekor over all the non-verification data in the entry, and lets us know that the timestamp can be trusted and that Rekor has seen the HashedRekord.

‍

The rest of the information can be used if you want to “trust but verify” Rekor. Rekor is a transparency log, so you can audit the entries using cryptography. The inclusionProof helps us check that the entry is actually in the Rekor log!

‍

Step 5: Publishing your artifact

Now that you have your artifact, signature, certificate, and Rekor entry, you can distribute them so that end-users can verify your signature.

‍

If you’re signing a container image stored on an OCI registry, you’re in luck! Cosign will store the signature and other metadata right on the registry, next to your original image.

Otherwise, you can distribute this the same way you would distribute the artifact on its own. For instance, you could publish it on GitHub releases, or using a language package manager.

‍

Verification

There are many steps involved in verifying a Sigstore signature. This process might have seemed complicated before, but hopefully now it’s clear why we need to do all of these things:

1. Check that the certificate is signed by Fulcio.

2. Check that the subject of the certificate matches the expected signer (such as zjn@chainguard.dev).

3. Check that the Rekor entry is signed by Rekor.

4. Check that the timestamp on the Rekor entry is during the validity period of the certificate.

5. Check that the Rekor entry refers to the same data that you’re trying to verify.

6. Check that the signature verifies against the public key from the certificate.

Conclusion

Sigstore might seem pretty complicated. This piece demystified Sigstore by breaking it down piece-by-piece. Then, you can actually see what’s going on by looking at all of the messages involved, and it turns out that they all actually make sense.

‍

If you’ve made it this far, you have a solid understanding of Sigstore, and are definitely an intermediate-to-advanced user! We’d love to hear from you:

‍

• Check out the Slack community!

• Give us feedback! Sigstore is on GitHub and you should let us know if something is still confusing, or if you’re having trouble using it in your situation.

• Contribute! The Sigstore community is very welcoming, and if you’d like to help out, you can check out issues with the “good first issue” tag; if you need guidance, just comment and we can find a mentor.