The end of GitHub PATs: You can’t leak what you don’t have
TLDR: This post outlines how we were able to replace our usage of long-lived GitHub Personal Access Tokens (PATs) with short-lived credentials across several GitHub organizations managed by Chainguard. To eliminate our need for these long-lived credentials, we created Octo STS to act as a “Security Token Service” (STS) for GitHub credentials.
One of our mantras at Chainguard is: treat your build systems like production systems. Really this sentiment applies not just to your build system, but to your entire development platform, which for a great many people is GitHub.
At Chainguard, we are continuously exploring ways to improve our security posture (e.g. ephemerality, and minimalism). We have A LOT of automation interacting with GitHub. In some places, we leverage GitHub Actions, which already supports short-lived tokens for basic same-repository operations. However, we also have a large amount of automation that these tokens do not work for, including:
Actions that create Pull Requests (needs a PAT to trigger presubmit GitHub Actions)
Actions that interact across repositories (unsupported by built-in permissions)
Actions that interact with the organization
Automation that is not running on GitHub Actions
For these use cases, the only viable credentials were long-lived:
Create a dedicated GitHub App (the private key is long-lived)
Create a “deploy key” (the private key is long-lived)
Create a Personal Access Token (PAT)
We had automation using a mixture of these long-lived credentials across many places in our codebase. To make matters worse, the toil of producing and managing the rotation of these long-lived secrets meant that developers would start to reuse existing tokens rather than go through the hassle of creating new ones, which violates the principle of least privilege (and increases the blast radius of a leaked token).
Credential leaks are one of the most common ways systems are compromised, and long-lived credentials are at the heart of that. These types of attacks have led to breaches at large enterprises like Mercedes and Toyota. At Chainguard, we typically avoid long-lived secrets like the proverbial plague, but with GitHub it seemed we were stuck with them.
Until Octo STS.
To eliminate our need for these long-lived credentials, we created Octo STS to act as a “Security Token Service” (STS) for GitHub credentials. The idea of an STS is largely inspired by the cloud providers like Amazon Web Services (AWS) and Google Cloud Platform (GCP), but other services have them too, including Chainguard. An STS exchanges a short-lived third-party token for a short-lived first-party token, after checking that the caller has permission to make the exchange.
Using AWS's STS, you can exchange a GitHub token for an AWS token to deploy resources from CI, without granting the workflow long-lived credentials. This is precisely what Octo STS does, but for GitHub — it allows users to federate third-party OpenID Connect (OIDC) tokens for GitHub credentials. As Dino A. Dai Zovi, Head of Security at Cash App, puts it — these types of “federation [are] a security super-power,” and we couldn’t agree more.
Octo STS even lets GitHub workflows use OIDC to obtain short-lived credentials to authorize GitHub requests, eliminating the need for long-lived PATs.
In short: GitHub didn’t expose an STS, so we went ahead and built one.
Establishing trust
The cornerstone of these types of federation is the trust policy. How does an STS know which third-party identities to hand first-party credentials to? To establish this trust in Octo STS we have users check-in trust policies under .github/chainguard/{foo}.sts.yamlin the repository where access is needed. A simple policy for federating a GitHub Action’s identity looks like this:
# Match these literal values in the respective claims
issuer: https://token.actions.githubusercontent.com
subject: repo:chainguard-dev/bar:ref:refs/heads/main
permissions:
contents: read
issues: write
These trust policies can get much more sophisticated, but let us first see how we would leverage our starter policy.
Federating with Octo STS
At a very high-level, the way Octo STS works is outlined in this sequence diagram:
Now suppose the policy from the previous section were checked into github.com/chainguard-dev/fooas .github/chainguard/baz.sts.yamlthen in github.com/chainguard-dev/barwe could author an actions workflow with this:
permissions:
id-token: write # Needed to federate with octo-sts
steps:
# Federate our workflow's identity token for a token that we can use
# to interact with chainguard-dev/foo
- uses: octo-sts/action@6177b4481c00308b3839969c3eca88c96a91775f # v1.0.0
id: octo-sts
with:
scope: chainguard-dev/foo
identity: baz
# Use the resulting token with the GitHub CLI (for example)
- name: Use the token with gh
env:
GITHUB_TOKEN: ${{ steps.octo-sts.outputs.token }}
run: |
gh repos list
… and just like that, we have enabled cross-repository access without a PAT.
But wait, there’s more!
Since Octo STS is built around OIDC federation, we can federate OIDC tokens from anywhere, not just GitHub Actions. For example, to allow a Google Service Account to federate you might use a trust policy like this:
# Match these literal values in the respective claims
issuer: https://accounts.google.com
subject: "1234567890" # This is the "unique id" of the Google Service Account
# (optionally) match the "email" version (beware: the values are regular expressions)
claim_pattern:
email: baz@blah\.iam\.gserviceaccount\.com
...
With this policy in place, a Google Service Account can POST its token to Octo STS and ask for a token to interact with the GitHub repo, and Octo STS will generate one.
Another neat trick: these trust policies can take advantage of claim_pattern to restrict which workflows can federate using the workflow_ref claim, which is super helpful for pursuing least privilege.
One more use case
The last use case mentioned in the beginning that we haven’t yet covered are things that need organization-level or multi-repository access. For these cases, we use the .githubrepository to hold our trust policies (e.g.). Trust policies in this repository can specify an additional field with the list of repositories to which access should be granted:
# Grant this identity
issuer: ...
subject: ...
# These permissions
permissions: ...
# On these repositories
repositories:
- foo
- bar
A note on permissions
Octo STS has to be installed into your organization (possibly scoped to a few repositories) for us to be able to hand out credentials that can access things within your organization. Due to how permissions for GitHub Apps work, Octo STS has to have a superset of the permissions it is capable of handing out. Right now the list is what we (Chainguard) needed to get off of PATs, but this will necessarily expand over time as users want to federate for additional permissions (reach out if there is an additional permission we don’t support).
The App itself uses exactly one permission: contents: read, which it needs to fetch trust policies from the repositories as described above. All other permissions exist so that users can request them for their federated tokens.
Wrapping up
Octo STS eliminated our need for an entire class of long-lived credentials, thereby eliminating our exposure to this attack vector. Across Chainguard’s GitHub organizations, we have been able to effectively eliminate our usage of PATs, several “deploy keys,” and at least one bespoke GitHub App. Wolfi and Chainguard are now more secure because of it.
The one long-lived key for Octo STS itself sits safely in a KMS system accessible to only the GitHub App for signing (not raw access) without setting off alerts.
Continuously hardening Chainguard’s own software supply chain as we become a part of many other organizations’ supply chains is critical. We take a defense-in-depth approach to protecting all of our internal controls and systems, conduct bi-annual pen tests with third parties, live and breathe the principles of minimalism and ephemerality, and leverage automation for alerting, detection and response. The only way to fix weak links in the global software supply chain is to start with the most secure foundation possible, and that’s exactly what Chainguard is committed to being for our users and customers today and into the future.
To install our Octo STS app, get started here. To learn how Chainguard can help your organization’s software supply chain security strategy, reach out to our team.
Ready to Lock Down Your Supply Chain?
Talk to our customer obsessed, community-driven team.