An Ode to Defense in Depth

On March 14th, 2025, a supply chain attack was performed that compromised the tags on the tj-actions/changed-files GitHub repository to point them all at a malicious commit that augmented the behavior of the action to dump the action runner’s memory, which includes any secrets that runner might hold.  This action is used across tens of thousands of GitHub repositories, including some of Chainguard’s (unaffected) and some of our customers’ public GitHub repositories.  The goal of this attack was a simple credential dump, which the attacker could then use to move laterally into other systems.

This behavior was thankfully quickly caught and flagged by StepSecurity, but any repositories that simply referenced this action by tag (e.g., tj-actions/changed-files@v2) were immediately affected, and numerous other repositories received and merged pull requests from dependabot and similar automation like Renovate updating even pinned versions of this action to the updated digest.

At Chainguard, our team sprung into action and quickly identified that we were unaffected by this through a combination of good fortune and a defense in depth approach to GitHub security, which I will expand on later in this post.  Regardless, we took several immediate actions out of an abundance of caution:

• First, we immediately blocked dependabot updates from the affected repository, so that we wouldn’t receive an update moving us to the offending digest.

• Second (out of an abundance of caution), we temporarily suspended some of our own automation so that if an attacker were able to compromise an upstream project which we package, that they wouldn’t be able to spoof a release and have our systems attempt to qualify it.  Now that the upstream issue has been completed, we have cross-checked a list of affected repositories, and our automation has resumed.

• Third, we reached out to some of our own customers who we observed to have public repositories containing the malicious commit, and have been trying to reach out to as many open source maintainers as we can to raise awareness.

We applaud the StepSecurity folks for quickly catching this issue. 👏

One of our mantras at Chainguard (and a core tenet of supply chain security) is to treat your build systems like production systems. As part of this mantra, there are a number of precautions that we take to help us mitigate against precisely this kind of attack.

Where we got lucky: Our dependabot PRs were staged a couple hours before the attack began, so we had awareness of the ongoing attack before it impacted us, which gave us time to add additional mitigations (above).

Pinning actions to commits (a la immutability). This prevented us (and many others) from immediately picking up the offending change.  This type of attack clearly highlights the importance of doing this for all of your actions workflows.  However, this alone is not enough because many folks doing this were affected because they still merged dependabot updates to the bad commit!

Drop the default GitHub action permissions from read-write to read-only (a la minimalism).  This is basic “least privilege” and highlights a really poor default in GitHub.  Thankfully, our workflows use read-only tokens by default. If we were affected, we still wouldn’t have leaked a read-write token!

We don’t use GitHub Secrets for anything sensitive.  Instead, we lean on identity federation and accessing secrets from our cloud provider’s secret manager; we don’t have any real secrets stored in GitHub Secrets to be leaked.  We made this choice after determining that we couldn’t secure these secrets to our standards. Remember: you can’t leak what you don’t have!

We use workflow-scoped identity federation (a la minimalism).  The identity tokens issued by GitHub use the same subject across different workflows on the same branch, which is insufficient for restricting particular workflows to assume particular identities.  Our custom identity federation rules leverage additional claims so that we can scope which workflows can federate to an identity to a particular workflow’s token.  This way, if workflow X is compromised, it cannot assume an identity intended for workflow Y because it doesn’t have the necessary claims.

We avoid long-lived credentials like the plague (a la ephemerality).  The few credentials our workflows deal with are short-lived.  We make use of GitHub actions OIDC tokens (short-lived).  We federate those tokens for CSP credentials (short-lived).  We also federate those tokens for different GitHub tokens via Octo STS (short-lived).

We revoke what credentials we can (a la ephemerality).  For credentials that support revocation, we revoke them upon workflow completion.  GitHub actions does this natively for the tokens associated with a workflow.  Our Octo STS action revokes the credential upon workflow completion.

We disallow tag mutation in our GitHub orgs (a la immutability).  For our own actions, such as the Octo STS action, we have repository rules in place that disallow the mutation of tags at the organization level.  Tag mutation is a bad practice and at best a nuisance to downstream developers, but at worst (as we saw here) a supply chain risk.

We encourage the use of hardware security keys for GitHub keys.  For my own GitHub SSH credentials, I have to tap my Yubikey in order to push commits.  This adds a layer of resilience that prevents these kinds of credentials from being susceptible to exfiltration even if malware gains access to my laptop.

I hope this whole thing ends up being a near-miss, but it highlights the need to treat your build systems like production systems and implement a defense in depth approach.