sigstore, the local way

If you've been following the Chainguard blog, you might ask yourself: how do I run the open-source sigstore stack on my machine?

‍

While sigstore is often deployed using Kubernetes, it is flexible enough to run nearly anywhere: from a Raspberry Pi to an IBM mainframe. This article will demonstrate how to build the sigstore stack (cosign, rekor, fulcio) on your machine and use it to sign and verify container signatures without ever leaving localhost.

‍

Prerequisites Consult the documentation for your favorite package manager to install:

• The Go Programming Language (v1.16 or higher: confirm by running go version)

• The MariaDB SQL server

• The Git distributed version control system

• OpenSC for managing PKCS#11 security tokens

• SoftHSM for implementing a PKCS#11 storage interface

Examples:

• The Go Programming Language (v1.16 or higher: confirm by running go version)

• The MariaDB SQL server

• The Git distributed version control system

• OpenSC for managing PKCS#11 security tokens

• SoftHSM for implementing a PKCS#11 storage interface

‍

These dependencies may also be downloaded and installed from their respective websites.

‍

Level I: Keyed signing with a local registry

First, we will use cosign, sigstore's container-signing tool, to sign a locally published container using a locally maintained key pair. Here are the commands we will execute, along with where they will access data from:

1.1: Running a local registry

sigstore can sign containers stored within any container registry. To keep our demonstration local, we'll install a simple registry using Go. Open a terminal and run:

go install github.com/google/go-containerregistry/cmd/registry@latest

As we begin launching several services into the foreground, now is a great time to begin a shell multiplexer such as tmux or screen, or at least a terminal that supports tabs. Start the registry service:

$HOME/go/bin/registry

The terminal will now quietly hang until a request arrives. Start a new terminal session, and let's move on!

‍

1.2: Pushing an unsigned image to the local registry

So that we have a target container to sign, we will now build a sample container and upload it to the local registry. First, install the ko container builder:

go install github.com/google/ko@latest

Download rekor, sigstore's tamper-resistant ledger software:

mkdir -p $HOME/sigstore-local/src

cd $HOME/sigstore-local/src

git clone https://github.com/sigstore/rekor.git

Build and push an image containing the rekor CLI to our local registry:

cd $HOME/sigstore-local/src/rekor/cmd

KO_DOCKER_REPO=localhost:1338/demo $HOME/go/bin/ko publish ./rekor-cli

1.3: Keyed-signing with cosign

Now we get to use cosign, sigstore's container-signing tool, to sign the container we just published. Install the latest cosign release:

go install github.com/sigstore/cosign/cmd/cosign@latest

Create a local key pair using any password. For simplicity, I suggest an empty one:

cd $HOME/sigstore-local

$HOME/go/bin/cosign generate-key-pair

Sign the published container using the local private key:

$HOME/go/bin/cosign sign --key cosign.key \
  localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9:latest

Use cosign to verify that the published container matches the local public key:

$HOME/go/bin/cosign verify --key cosign.pub \
  localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9

The output for a successful verification will look like this:

Verification for localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9:latest --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- The signatures were verified against the specified public key
- Any certificates were verified against the Fulcio roots.
[{"critical":{"identity":{"docker-reference":"localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9"},"image":{"docker-manifest-digest":"sha256:3a46c2e44bfe8ea0231af6ab2f7adebd0bab4a892929b307c0b48d6958863a4d"},"type":"cosign container image signature"},"optional":null}]

Congratulations! You have just signed your first container! To sign other artifacts such as binaries, see Working with other artifacts.

‍

Level II: Certificate Transparency with Rekor

So far, verification has relied on a single mutable source of truth: the container registry. With Rekor, we will introduce a second immutable source of truth to the system:

2.1: Creating a database backend with MariaDB

While Sigstore can use multiple database backends, this tutorial uses MariaDB. Once you've installed the prerequisites, run the following to start the database up locally in a locked-down manner:

‍

• Arch Linux: sudo mariadb-install-db --user=mysql --basedir=/usr --datadir=/var/lib/mysql; sudo systemctl start mariadb && sudo mysql_secure_installation

• Debian | Ubuntu: sudo mysql_secure_installation

• Fedora: sudo systemctl start mariadb && sudo mysql_secure_installation

• FreeBSD: sudo sudo service mysql-server start && sudo mysql_secure_installation

• macOS: brew services start mariadb && sudo mysql_secure_installation

• OpenBSD: doas mysql_install_db && doas rcctl start mysqld && doas mysql_secure_installation

Follow the prompts from mysql_install_db, answering N to change the root password, and Y to everything else. Afterward, run the database creation script:

cd $HOME/sigstore-local/src/rekor/scripts

sudo sh -x createdb.sh

2.2: Installing Trillian

Trillian provides a tamper-proof append-only log based on Merkle Trees using a gRPC API. Trillian stores its records in the MariaDB database we previously created. Install Trillian:

go install github.com/google/trillian/cmd/trillian_log_server@latest
go install github.com/google/trillian/cmd/trillian_log_signer@latest
go install github.com/google/trillian/cmd/createtree@latest

Start the log_server, which provides the Trillian "personality" API, used by Rekor, and the Certificate Transparency frontend.

$HOME/go/bin/trillian_log_server --logtostderr \
  -http_endpoint=localhost:8090 -rpc_endpoint=localhost:8091

Start the log signer, which periodically checks the database and sequences data into a Merkle tree:

$HOME/go/bin/trillian_log_signer \
  --logtostderr --force_master --http_endpoint=localhost:8190 \
  --rpc_endpoint=localhost:8191

The Trillian system is multi-tenant and can support multiple independent Merkle trees. Run this command to send a gRPC request to create a tree and save the log_id for future use:

$HOME/go/bin/createtree --admin_server localhost:8091 \
  | tee $HOME/sigstore-local/trillian.log_id

2.3: Installing Rekor

‍The Rekor project provides a restful API-based server for validation and a transparency log for storage. Install it from source:

cd $HOME/sigstore-local/src/rekor

go install ./cmd/rekor-cli ./cmd/rekor-server

Start rekor:

$HOME/go/bin/rekor-server serve --trillian_log_server.port=8091 \
  --enable_retrieve_api=false

Upload a test artifact to verify that Rekor is functioning correctly:

cd $HOME/sigstore-local/src/rekor

$HOME/go/bin/rekor-cli upload --artifact tests/test_file.txt \
  --public-key tests/test_public_key.key \
  --signature tests/test_file.sig \
  --rekor_server http://localhost:3000

2.4: Verifiable signing with Cosign & Rekor

Upload a signature for our image, using the local key pair we created in step 1.3.

COSIGN_EXPERIMENTAL=1 $HOME/go/bin/cosign sign \
  --key $HOME/sigstore-local/cosign.key \
  --rekor-url=http://localhost:3000 \
  localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9

Verify the container against the mutable OCI attestation and the immutable Rekor record:

COSIGN_EXPERIMENTAL=1 $HOME/go/bin/cosign verify \
  --key $HOME/sigstore-local/cosign.pub \
  --rekor-url=http://localhost:3000 \
  localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9

Success looks like this:

Verification for localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9:latest --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- The claims were present in the transparency log
- The signatures were integrated into the transparency log when the certificate was valid
- The signatures were verified against the specified public key
- Any certificates were verified against the Fulcio roots.
[{"critical":{"identity":{"docker-reference":"localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9"},"image":{"docker-manifest-digest":"sha256:35b25714b56211d548b97a858a1485b254228fe9889607246e96ed03ed77017d"},"type":"cosign container image signature"},"optional":{"Bundle":{"SignedEntryTimestamp":"MEUCIG...yoIY=","Payload":{"body":"...","integratedTime":1643917737,"logIndex":1,"logID":"4d2e4...97291"}}}}]

🍦 Take an ice cream break if you got this far. You earned it!

‍

Level III: Keyless signing with Fulcio

Fulcio is a sigstore component that issues code-signing certificates based on an authenticated OpenID Connect identity. These certificates are short-lived, lasting only 20 minutes!

‍

Cosign has experimental support for using Fulcio to generate signing certificates, saving users the headache of managing certificates. As our goal is to run everything locally, we have to stand up substantially more architecture to provide secure keyless signing:

3.1: Install Fulcio

As we're going to use some experimental features in this tutorial, you will need to install Fulcio from HEAD:

cd $HOME/sigstore-local/src

git clone https://github.com/sigstore/fulcio.git

cd fulcio

go install .

3.2: Configure SoftHSM

SoftHSM implements a cryptographic store accessible through a PKCS #11 interface. You can use it to explore PKCS #11 without having a Hardware Security Module. For this demo, we will configure sigstore to reference tokens in $HOME/sigstore-local/tokens:

‍

Create your first HSM token, setting both PINs to 2324.

softhsm2-util --init-token --slot 0 --label fulcio

3.3: Create a CA certificate with OpenSC

Many tools need to be informed about the SoftHSM installation location, which is different in each operating system. Run this command to set and reveal the library location:

export HSM=$(find /usr/local/lib /usr/lib /usr/lib64 \
  /opt/homebrew -name libsofthsm2.so | head -n1)

file $HSM

Create a configuration file for the pkcs11 crypto library using the user PIN code you specified in the previous step:

mkdir -p $HOME/sigstore-local/config

echo "{ \"Path\": \"$HSM\", \"TokenLabel\": \"fulcio\", \"Pin\": \"2324\" }" > $HOME/sigstore-local/config/crypto11.conf

Generate a new key pair that is stored directly in the HSM. When prompted for a PIN, use 2324:

SOFTHSM2_CONF=$HOME/sigstore-local/softhsm2.conf pkcs11-tool \
  --login --login-type user --keypairgen --id 1 --label PKCS11CA \
  --key-type EC:secp384r1 --module=$HSM

Create a local CA root certificate:

cd $HOME/sigstore-local

SOFTHSM2_CONF=$HOME/sigstore-local/softhsm2.conf $HOME/go/bin/fulcio \
  createca --org=acme --country=USA --locality=Anytown \
  --province=AnyPlace --postal-code=ABCDEF \
  --street-address="123 Main St" --hsm-caroot-id 1 --out ca-root.pem

3.4: Install the Certificate Transparency Frontend

The ct_server is an RFC6962-compliant certificate transparency log that stores the code-signing certificates issued by Fulcio.

go install github.com/google/certificate-transparency-go/trillian/ctfe/ct_server@latest

Next, create a private key for the front end to use for signing certificates. For the password, I suggest using 2324 again, but you may use anything 4-characters or longer:

cd $HOME/sigstore-local

openssl ecparam -genkey -name prime256v1 -noout -out ct_unenc.key
openssl ec -in ct_unenc.key -out ct_private.pem -des
openssl ec -in ct_unenc.key -out ct_public.pem -pubout -des

rm ct_unenc.key

Store the password as a shell variable:

export PASS= <the password you just used>

Look up the Trillian log ID we previously created and set the LOG_ID variable to the resulting value:

Look up the Trillian log ID we previously created and set the LOG_ID variable to the resulting value:

Populate the Certificate Transparency configuration file:

printf "config {
  log_id: $LOG_ID
  prefix: \"sigstore\"
  roots_pem_file: \"$HOME/sigstore-local/ca-root.pem\"
  private_key: {
    [type.googleapis.com/keyspb.PEMKeyFile] {
      path: \"$HOME/sigstore-local/ct_private.pem\"
      password: \"$PASS\"
    }
  }
}" | tee $HOME/sigstore-local/ct.cfg

Start the certificate transparency server:

$HOME/go/bin/ct_server -logtostderr \
  -log_config $HOME/sigstore-local/ct.cfg \
  -log_rpc_server localhost:8091 \
  -http_endpoint 0.0.0.0:6105

3.5: Installing Dex for OpenID authentication

Dex is a federated OpenID Connect Provider, connecting OpenID identities from multiple providers. We are going to use Dex to provide GitHub authentication. To build it from source (Note: BSD users should use gmake instead of make):

cd $HOME/sigstore-local/src
git clone https://github.com/dexidp/dex.git

cd dex
make build
cp bin/dex $HOME/go/bin

For this demonstration, we'll use GitHub as an OpenID provider. Visit GitHub: Register a new OAuth Application, and fill in the form accordingly:

‍

Application Name: My Local Sigstore Adventure Homepage URL: http://localhost/ Authorization callback URL: http://localhost:5556/callback

‍

When you click Register Application, it will output a client ID. Save it to your environment:

export GI_ID= <your id>

Click the Generate a new client secret button, and copy the long alphanumeric string it emits into your environment:

export GI_SECRET=<your client secret>

Populate the Dex configuration:

printf "issuer: http://localhost:5556
storage:
  type: sqlite3
config:
  file: ./dex.db
web:
  http: 127.0.0.1:5556
frontend:
  issuer: sigstore
oauth2:
  responseTypes: [ "code" ]
staticClients:
  - id: sigstore
    public: true
    name: sigstore
connectors:
  - type: github
    id: github-sigstore-test
    name: GitHub
    config:
      clientID: $GI_ID
      clientSecret: $GI_SECRET
      redirectURI: http://localhost:5556/callback
" | tee $HOME/sigstore-local/dex-config.yaml

Start dex:

$HOME/go/bin/dex serve $HOME/sigstore-local/dex-config.yaml

3.6: Setting up Fulcio for keyless signatures

Populate the Fulcio configuration:

printf '{
  "OIDCIssuers": {
    "http://localhost:5556": {
      "IssuerURL": "http://localhost:5556",
      "ClientID": "sigstore",
      "Type": "email"
    }
  }
}' > $HOME/sigstore-local/config/fulcio.json

Start Fulcio:

cd $HOME/sigstore-local

SOFTHSM2_CONF=$HOME/sigstore-local/softhsm2.conf \
  $HOME/go/bin/fulcio serve --config-path=config/fulcio.json \
  --ca=pkcs11ca --hsm-caroot-id=1 \
  --ct-log-url=http://localhost:6105/sigstore \
  --host=127.0.0.1 --port=5000

3.7: Local Keyless Signing

Now it is time for the finale: keyless signing! In an internet-connected environment, the command-line to sign a container is relatively simple:

COSIGN_EXPERIMENTAL=1 cosign sign  <image location>

However, since we will be using our local environment, we need to override the public-key certificate used to verify identities and the endpoint locations. Run this command to sign the image we pushed locally:

SIGSTORE_CT_LOG_PUBLIC_KEY_FILE=$HOME/sigstore-local/ct_public.pem \
COSIGN_EXPERIMENTAL=1 $HOME/go/bin/cosign sign \
--oidc-issuer=http://localhost:5556 \
--fulcio-url=http://127.0.0.1:5000 \
--rekor-url=http://localhost:3000 \
localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9

When you run that command, your browser will open to the local Dex instance, which will prompt you to authenticate using GitHub. Afterward, verify the certificate:

SIGSTORE_ROOT_FILE=$HOME/sigstore-local/ca-root.pem \
  COSIGN_EXPERIMENTAL=1 \
  $HOME/go/bin/cosign verify \
    --rekor-url=http://localhost:3000 \
    localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9

Among other things, the output will include the OIDC issuer (our local Dex URL) and the e-mail address you authenticated to Github with. Here is an example:

Verification for localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9:latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The claims were present in the transparency log
  - The signatures were integrated into the transparency log when the certificate was valid
  - Any certificates were verified against the Fulcio roots.

[{"critical":{"identity":{"docker-reference":"localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9"},"image":{"docker-manifest-digest":"sha256:dae1e7cdb03fc6f16e3a48111634f0c5fc28752229da2f96a6a4d03f60d6
d609"},"type":"cosign container image signature"},
"optional":{"Bundle":{"SignedEntryTimestamp":"MEUCI...Og4=","Payload":{"body":"...","integratedTime":1644977812,"logIndex":3,"logID":"9d0dd...a86d"}},
"Issuer":"http://localhost:5556","Subject":"blog@chainguard.dev"}}]

😊 Pat yourself on the back! You made it through the tutorial!

‍

Where to next?

If you encounter any problems or would like to learn more about sigstore, see:

‍

• Official Sigstore Documentation

• The original sigstore-the-local-way tutorial - complete with a restore script

‍

Happy artifact signing, everyone!