Putting VEX to work

Software bills of material (SBOM) are an important part of supply chain security. Yet, the data they offer needs to be fine-tuned to be effectively actionable. Chainguard has been thinking about the problem and has created a new tool to help organizations achieve that important fine-tuning of SBOM data using VEX, the Vulnerability Exploitability eXchange.

‍

Know Before You Run!

At Chainguard, we are big fans of transparency in software. A trustworthy record of the components contained in your workloads is one of the first things you need to ensure a more secure supply chain, and when it comes to component disclosure, SBOMs are the way to go. These documents are now the standard format to inform which ingredients, like libraries and other dependencies, went into the mix that eventually became your application. While there are many types and varying qualities of SBOMs, the ideal future is one where we will have a comprehensive list of all software contained in that package you are about to execute.

‍

In a security context, the SBOM promise is that organizations will be able to instantly locate affected components when a new vulnerability is found simply by checking their SBOM inventory.

‍

While having a list of all software contained in an application package is great, the reality is that a vulnerability found in a component may not necessarily affect the application packaging it. Alerting on these false positives reduces the value of alerts themselves but it also means that as SBOMs get better and better, the frequency of these false positives increases, causing even more noise.

‍

There are many reasons why vulnerabilities may not affect a software application. For example, a project may work around an old, known but unpatched security issue by adding mitigations in the app code. Libraries sometimes have huge codebases and it could be the case that an application simply doesn’t trigger the parts of the code where the flaw lives or that there may not be a way for adversaries to trigger the vulnerable code.

‍

As a software developer, you want to let your users know when your application is safe from a serious vulnerability. This post explaining why Quarkus is not affected by CVE-2021-44228 is a great example of humans informing other humans that the Kubernetes Java framework is not affected by the log4j vulnerability. Unfortunately, scanners and automated policy tools do not care a lot for terse and convincing prose in blog posts. Even if you create a post like that, scanner noise will happen and it will lead to team fatigue. False positives will still wrongly trigger policy enforcement. Examples like these are why we need machine-readable means of informing our downstream consumers about the true impact of vulnerabilities.

‍

Enter VEX

VEX, short for Vulnerability Exploitability eXchange is a specification originally proposed by the SBOM team at the Cybersecurity and Infrastructure Security Agency (CISA). It defines machine-readable data that complements SBOMs to state the impact of security flaws in software. Using VEX, software publishers can hand their consumers information that helps them turn off those false alerts in their scanners. You can think of a VEX document as a negative security advisory.

‍

Chainguard’s CEO Dan Lorenc captured the spirit of VEX in a recent comic, it is worth checking it out but this is the relevant slide illustrating how VEX works:

The key principle in VEX is communicating what is called a product status - data describing how a specific vulnerability affects software. The currently defined statuses are simple to understand:

By sending these simple constructs downstream, software publishers can help tune if and when scanner alerts should fire. It is recommended that VEX impact statements include a justification when a vulnerability is deemed not to affect a piece of software clarifying why not.

‍

At the time of writing, VEX data can be defined in CSAF (Common Security Advisory Framework) and CycloneDX documents. Work is underway in the Software Package Data Exchange (SPDX) community to support it, too. Conveying VEX metadata in each of these formats has its own challenges. CSAF documents sometimes feel overly complex for the task. We in particular believe VEX data should be kept separate from SBOMs as it is produced by different processes, at different moments, and issued on a different cadence. Despite these challenges, VEX can be put to work today!

‍

vexctl: Chainguard’s VEX Tool

To help with VEX, Chainguard is today releasing vexctl a new CLI tool to enable software authors and consumers to work with VEX metadata. Using our new tool, users can apply documents to scanner results and turn off false positives. It also allows software publishers to publish vex data in containerized workloads.

‍

You can get vexctl today from https://github.com/chainguard-dev/vex

‍

Basic Operation

In its most basic form of operation, vexctl enables users to rewrite the output of a security scanner to remove vulnerabilities specified in VEX metadata as not impacting. Here is an example using Trivy:

‍

At present, Trivy detects five critical vulnerabilities in the official nginx image:

# Run Trivy, use jq to find the found CVEs:

$ trivy image nginx -s CRITICAL --format=sarif | jq -r '.runs[0].tool.driver.rules[].id'

CVE-2019-8457
CVE-2022-40674
CVE-2022-1586
CVE-2022-1587
CVE-2022-37434

Doing a quick search of the nginx source code, sqlite seems not to be used at all by the popular web server. However, if I were to package my application using nginx:latest as a base, scanners checking my image would fire on CVE-2019-8457. To give my users assurances that they are safe from said CVE, I could issue a VEX document. The relevant parts would look like this (simplified CSAF document fragment):

"vulnerabilities": [
  {
    "cve": "CVE-2019-8457",
    "notes": [
      {
        "category": "description",
        "title": "SQLite3 is vulnerable to heap out-of-bound read"
      }
    ],
    "product_status": {
      "known_not_affected": [
        "MYIMAGE"
      ]
    },

At this point, my users would have the security scan and a piece of metadata capturing a human assessment of the vulnerability’s (non-)impact. Now what? This is where vexctl comes into play!

‍

Using vexctl you can cross the data from the scanner and the VEX document to produce an “effective” report. To get it, you can simply pipe the Trivy report through vexctl:

$ trivy image nginx -s CRITICAL --format=sarif | vexctl filter - vex.csaf.json

Running the above would output the same Trivy report, but rewritten to remove the result specified as harmless in the VEX metadata. To verify this we can enumerate the CVEs in the report:

$ trivy image nginx -s CRITICAL --format=sarif | vexctl filter - vex.csaf.json | jq -r '.runs[0].tool.driver.rules[].id'

CVE-2022-40674
CVE-2022-1586
CVE-2022-1587
CVE-2022-37434

Currently, the tool understands scanner reports in SARIF (Static Analysis Results Interchange Format), we have plans to expand it to read data from the native formats of security scanners.

‍

Using vexctl, anybody who gets handed VEX data can use it to filter the results of scanners, effectively reducing noise and turning off unnecessary warning lights. But we need to make this practical, read on!

‍

Scaling It Up!

So we can now read and use VEX data. Great! But things in the real world are not that simple. In containerized environments, applications tend to change fast which means that new versions of the same image are released often minutes apart.

‍

Nobody wants to write new VEX metadata every time a new image is created. In the same sense, no one wants to look for new VEX documents with every version bump. Wouldn’t it be great if the container image itself could tell us which vulnerabilities do not apply to it?

Teaching images to express their own VEX metadata is the second feature in vexctl. Using a vex document as a source of rules, vexctl can create an attestation containing the VEX impact statements. The idea is that once a security team determines a vulnerability does not affect their product, they can create a golden rules file that can be applied to each new container image. If the impact status changes, simply modifying the rules file will cause new images will pick it up.

‍

Attestations of impact are signed with Sigstore and are in the standard in-toto format. They are attached in the OCI Registry to the container image they describe, this makes them easily locatable.

‍

To generate the attestation, simply point the vexctl attest subcommand to the image and the VEX file:

$ vexctl attest myimage@sha256:26fed2df… vex.csaf.json

That command will collect the required data from the image and the VEX document and start the usual Sigstore authentication flow which lets you prove your identity with a choice of authentication services like Google, GitHub or Microsoft. Once authenticated, your account will be used as the signing identity for the attestation. Ambient credentials are also supported if you are running in CD/CD environments.

‍

After vexctl attest is done, VEX data will be attested to and attached to the image.

‍

Using Images as the Source of VEX Data

Now, let’s see how to use the newly attached attestation. Once we have an attestation living next to an image, we no longer need the VEX document and we can use the image itself as the source of VEX information as we have now effectively taught it how to tell us about VEX impacts.

‍

In order to use images with VEX attestations, simply swap the vex document for the image reference in the command we ran earlier:

$ export IMAGE=myimage@sha256:26fed2df…
$ trivy image ${IMAGE} -s CRITICAL --format=sarif | vexctl filter - ${IMAGE}

This invocation is effectively the same as running vexctl with the CSAF file and should produce the same results.

‍

The main benefit of this approach is that images can get their VEX data attested and attached to them in the OCI registry. As images are often used as base images for derivative works, applications can collect the VEX data from an expected source, and since they are signed they ensure the right person is stating the impact of vulnerabilities.

‍

VEXing it Forward

As container images and their layers are reused and composed to new images, VEX documents can be combined to give the end user the most accurate view of the effective impact statements.

‍

For example, if you package your application using Chainguard Images, you’ll find a VEX document when our team determines them not to be affected by CVEs found by the most popular scanners. This means that your users can use vexctl in front of their scanners to turn off false positives.

‍

But there is more. It may be the case that your application based on one of our images may trigger a VEXed vulnerability. In that scenario, you may want to attach an attestation to your application with a status of AFFECTED. By combining the base image attestation and your own, vexctl can let your users know that a particular version of the app is affected, regardless of what the base image attestation says. vexctl is smart enough to sort vex data in order, so vex documents are applied in chronological order which should reflect how knowledge of the vulnerability has evolved.

‍

Towards the Future

VEX is still under heavy development and we expect things to evolve significantly as the program matures. We will regularly publish updates to help you navigate and make the most of VEX. As for vexctl, it is our intent to keep it up to date as the spec improves. If you have questions or comments please don’t hesitate to reach out to us.