A Fulcio Deep Dive
Sigstore is hard to describe briefly, but I prefer “The Let's Encrypt of Code Signing". This is meant to be a very ambitious North Star for the project, and comes with the greatest possible respect for the folks behind Let's Encrypt. I like to reflect a bit on the impact Let's Encrypt had on the Internet by remembering what the Internet was like before it. Web Certificates were expensive, annoying, and hard to set up, so TLS was uncommon. Personal websites never had it, company websites struggled to use it correctly.
Now you can add a single line to a yaml file and get TLS for free. Most PaaSes do it automatically. Let's Encrypt disrupted the internet and the industry by realizing that developers like to do the secure thing! It just has to be easy. The impact is pretty easy to see by comparing LE’s growth stats to those across the Internet, as reported by Chrome.
Designing a new system that allowed certificates to be issued automatically, for free wasn’t easy and didn’t happen overnight. This involved a new IETF standard (ACME), client tooling (Cert Bot), and professionally run, Internet scale infrastructure. By a non-profit organization Easy!
Let's Encrypt did this by changing a few major things:
- Automatic rotation
- Short validity
- A new standard
These all go hand in hand. Free, automatic certs are ripe for abuse, so LE only grants them for a short period of time in case something bad happens. This short validity period would be error prone and toilsome, so great automation was necessary. Finally, ubiquitous automation requires well understood standards to build tooling around. These factors work together to make LE work.
This post is about Fulcio though! Fulcio is very similar to Let's Encrypt - it's a free, automatic certificate authority (CA)! There are two big differences in the certificates Fulcio creates: the Subject and the Key Usage.
Fulcio certs are designed for signing code, not web encryption.
Key reuse is hard to get right without taking great care, so individual certificates normally are only valid for a few particular use cases, which are indicated in the cert as
KeyUsage block in an LE certificate compared to a Fulcio one.
# Let's Encrypt X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication # Fulcio X509v3 Key Usage: critical Digital Signature X509v3 Extended Key Usage: Code Signing
These differences indicate that Fulcio cannot be used for TLS certificates (they wouldn’t be trusted anyway), but they can instead be used to sign code.
The second difference is around the
Certificates are used to bind identities to cryptographic keys, and the
Subject is another term for identity.
Read this blog post from Mike Malone if you want to get a deeper understanding of these terms and certificates in general, but that’s all you need to know for this post.
An identity can represent anything - it can be a server, a person, an IP address, a piece of code, a piece of hardware, a URL, etc
In TLS, these are typically domain names.
One of the
SubjectAltName’s for the certificate at google.com is
DNS:*.google.com, for example.
This means it’s valid for all subdomains above google.com.
Instead of DNS hostnames, Fulcio issues certificates to a few different types of identities related to software supply chains. Today, we use the following standards:
- People! Via email addresses based on OpenID Connect
- Workloads, based on the SPIFFE SVID specification
Fulcio also supports a few custom identity schemes we designed specifically for supply chain security - GitHub Action invocations and Kubernetes Service Accounts.
Fulcio gives certificates out over an API based on information in the request. But it can’t just hand any certificate to any person, that would make it easy to impersonate people or services, making the entire system worthless. Fulcio has to do some type of validation that the caller should be able to retrieve a certificate for that system, this validation is called a “Challenge”.
Let’s Encrypt uses the ACME protocol to validate that the caller owns the DNS domain for a certificate before issuing one. ACME has a few built-in challenges you can read more about, but they typically require the user to perform some basic proof, like putting a random string on the website in a particular HTML block or in a DNS entry.
For Fulcio, we tried to reuse existing standards wherever possible, and it turns out that there’s already a widely adopted method for validating identity, called OpenID Connect! This is the system that allows you to log into sites using a third-party login, you might recognize it in browser pop ups like this:
OIDC has a few main flows that allow it to be used for both machines and people, and Fulcio supports many of these so it can be used almost anywhere. The basic premise is that you log in to an Identity Provider somehow. This could be in a browser with a username and password, or it could be a service using ambient credentials or a long-lived token. After authenticating with this provider directly, you receive an Identity Token, which you can hand to other services for validation. These services validate that the token is correctly issued by the provider, and valid for this use case.
Fulcio is one of these relying services, so it effectively exchanges OIDC identity tokens for code signing certificates. Here are a few examples of how the fields get set in a certificate based on token types:
X509v3 Subject Alternative Name: critical URI:spiffe://oidc.dlorenc.dev/tekton/taskruns/runs/1
X509v3 Subject Alternative Name: critical URI:https://github.com/dlorenc/hello-ko/.github/workflows/build.yaml@refs/heads/main
X509v3 Subject Alternative Name: critical email:email@example.com 220.127.116.11.4.1.57264.1.1: https://accounts.google.com
Other Cool Things!
Fulcio is a single CA today that can be used as a shared trust root. Different individuals and organizations can rely on Fulcio to correctly authenticate and issue signatures for each other that can be validated offline.
This does require some trust in Fulcio to correctly verify user IDs and operate securely, but much of this is mitigated through the use of Transparency logs. Users don't need to fully trust Fulcio, because they can verify it! Every certificate issued by Fulcio is present on a public log. This log can be verified and searched, so users can easily obtain a list of every certificate issued on their behalf.
If Fulcio messed up, or someone’s credentials got stolen - it would be on the public record for all to see, allowing for a fast and complete resolution.
As described above, the Fulcio CA can issue code signing certificates based on verified identities in SPIFFE SVIDs.
An SVID contains a few parts:
The domain portion here allows for federation, allowing organizations to use the public Fulcio trust root for their own SVIDs.
To configure Fulcio to federate to a specific domain name, we use a set of config files and a standard pull request based flow.
Here’s the setup for my personal test domain:
url: https://oidc.dlorenc.dev contact: firstname.lastname@example.org description: "test address for federation!" type: "spiffe"
Any other domains can be added as well! To prevent namespace collisions, Fulcio ensures that SVIDs begin with the federated domain name here, which is where the OIDC Discovery Endpoint is hosted. The overall flow looks like this:
One of the main benefits to signatures is that they can be verified in an offline context. All you need is the public key and you can check a signature without making any network requests. The use of transparency logs can make that harder by requiring that the users perform an inclusion proof to ensure something is present in the log.
Fulcio still supports offline verification though, with the use of “stapled” inclusion proofs. When a certificate is first issued and entered into the log, Fulcio returns a “Signed Certificate Timestamp”, or SCT alongside the certificate. This signature serves as a proof that the certificate has made it into the Certificate Transparency log, and can be verified offline. If clients verify these SCTs and audit Fulcio regularly, users can trust that certificates are present in the log without needing to make an actual request to the log!
Keyless signatures powered by Fulcio aren’t completely keyless of course - they’re just a clever tactic to make it easier to manage keys for most users. This is all based on the Fulcio service managing a single key for everyone, which acts as a shared trust root for the entire community.
I’ve already discussed how we mitigate the technical one using transparency logs, but placing this key in the hands of any one single person or company is another type of risk! Thankfully the Sigstore community worked to mitigate this one too using The Update Framework and a distributed, community-based key ceremony.
Five Key Holders are responsible for managing the signing material for the entire Sigstore project, including our transparency logs, binary release keys, and Fulcio itself! A majority of the Key Holders are required to make any changes to the keys used, and we manage expiration and rotation using TUF. Key Holders rotate out as well, and we make sure a majority of the individuals never work for the same organization. The initial ceremony was live streamed and verified in real time, and is now available on Youtube.
Hopefully this post provides an overview of the power of Fulcio, Sigstore's free, community-driven Code Signing Certificate Authority. Our goal is to make Fulcio as easy to use from as many different environments as possible, so developers have to excuse to not sign their code!
If you're interested in running your own instance of Fulcio, either in a subordinate or independent configuration, Chainguard can help! Please reach out to us over Twitter or email and we can help you get started.