Working as unexpected

TLDR: This is a tale of a “working as intended” branch protection bypass that allows for protected credential exfiltration. This is a security vulnerability that was reported to GitHub via HackerOne on February 2nd, 2024, and fairly quickly closed as “working as expected.” While GitHub may expect this behavior, it violates the principle of least surprise, and so I wanted to outline this vulnerable behavior so that folks don’t fall into the trap it creates.

‍

As part of hardening Chainguard’s security posture, I am continuously in pursuit of ways to leverage controls to treat GitHub like a production environment. On this particular excursion, I was exploring whether I could eliminate the ability to create new branches directly on our upstream repository with a wildcard branch protection *:

My hypothesis was that the above might prevent folks from creating new branches on our upstream repository (effectively that this would protect non-existing branches in addition to existing branches). It was simple enough to test with an experiment, and I was (partially) wrong. However, now that the branch exists, it (somewhat obviously) shows up as protected. Hmm…

‍

I found this intriguing because I had also recently been exploring the use of GitHub’s environments feature, which allows you to restrict the visibility of certain secrets to specific branches or to protected branches. Bear in mind, this feature is the most secure level of secret storage GitHub offers. You can configure it like so:

So my very next question was: If my new branch immediately becomes protected, could its workflows immediately be eligible to access these secrets? To test that theory, I crafted the following workflow to exfiltrate a secret I created in the mattmoor-testingenvironment above called NOT_A_SECRET:

on:
  push:
    branches:

name: example secret exfiltration

jobs:
  build:
    runs-on: ubuntu-latest
    environment: mattmoor-testing

    steps:
    - shell: bash
      run: |
        echo ${{ secrets.NOT_A_SECRET }} | base64
 

Somewhat unsurprisingly this worked:

The full reproduction steps I gave to GitHub are:

At first, I thought (likely similar to GitHub) that this was a vanishingly small niche. After all, how widely used are wildcard branch protection rules that folks expose secrets to? However, the more it marinated in my head, the more it concerned me, and a very plausible scenario emerged.

‍

Digging deeper

On previous open source projects I have worked on, it was pretty typical to create long-lived release branches from which we could cut patch releases (e.g. release-1.2). We protected these branches with a protection rule that applied to release-*. Now suppose that we wanted our release workflows to have access to sensitive secrets that only the release workflow should have, say for instance signing keys (e.g. terraform provider GPG keys, deb, rpm, apksigning keys). Gulp.

‍

Initially, I had also questioned the value of such an attack with excuses like: my branch was pretty obvious (it is protected, so I couldn’t just delete it to cover my tracks), the extra workflow was pretty obvious, and the way I exfiltrated the secret made it available to anyone (not just me). This also did not age well as it marinated in my head.

‍

For projects with many releases I could very likely hide my branch typo-style with something like release-1.02or release-.1.2, which would look like an innocent error. To hide the workflow, I could put the exfiltration itself into an existing workflow, so it blended into the other action executions of the project. Lastly, instead of using something as trivial as base64to avoid GitHub’s secret masking I could do something that made it only accessible to me: I could post it to a service I own. Or if the execution were somehow network jailed (not something Actions supports), then I could encrypt it with an asymmetric key included in the branch and log the encrypted value instead.

‍

But I’d have to be a writer on the repo. True, but this is also more common than you’d think. After all, GitHub locks up all kinds of useful permissions behind having this level of access, including things like being able to label issues, move them around project boards, or interact with milestones. In fact, the main feature that previously allowed me as a maintainer to sleep at night with a non-trivial number of repo editors was ironically … (drumroll) ... branch protections.

‍

‍Now here’s a piece I missed in my initial branch protection configuration, which helps to mitigate this, but does not completely close the hole. I’d missed this because of a subtle text difference between what is displayed for new branch protections (above) vs. existing branch protections (below):

I was looking at an existing branch protection when exploring these knobs, which indicates that anyone with the ability to write to the repo can still create branches. However, the left hand side indicates that only administrators can create new branches (progress, but ideally this would be configurable with):

Best practices for branch protections

If you are concerned about vulnerable behaviors in GitHub branch protections, here are some takeaways and best practices to consider: ‍

• Favor the use of repository rulesets (new) over branch protections (old), which can actually block administrators if they are not explicitly put onto the bypass list. ‍

• Only use wildcards in branch protections when absolutely necessary. ‍

• When using wildcard branch protections always restrict who can create matching branches (e.g. so that only admins can create release branches). ‍

• Only trust branch protections in environments (vs. concrete branches) when absolutely necessary. ‍

• If you do find yourself doing the above, consider requiring administrators to approve environment access. ‍

• Prefer the use of a cloud service provider’s secret store accessed via OIDC federation over GitHub’s built-in secret storage, and ensure that the federation rules are as restrictive as possible.

‍

As I mentioned earlier, GitHub marked this issue as working as intended, but (silver linings) it freed me to at least help educate you all to be on the lookout for vulnerable behaviors like this.

If you are interested in learning more about Chainguard’s approach to our own product security and internal practices with defense-in-depth principles, visit our Trust Center or learn more about our Octo STS project here.