Stemming the tide of false positive vulnerabilities

TL;DR: Many (most?) vulnerabilities are false positives: they aren’t actually exploitable. New research from Chainguard Labs proposes tooling to automatically identify false positives so that you only need to worry about vulnerabilities that matter.

‍

At Chainguard, we hear consistently from customers about the burden of managing vulnerabilities in software. Security teams feel obligated to account for every vulnerability, because they know that if they miss just one they could be hauled in front of Congress or fined. (Or maybe they’re just trying to get FedRAMP certified). But when building on top of default containers that might each have hundreds of vulnerabilities (with more every day), this task can seem Sisyphean.

‍

Fortunately, there’s an easy button: Chainguard Images (built on the open source Wolfi project), where we roll the vulnerability boulder for you. How? One trick is minimalism: by tailoring our OS to the container environment, we can keep images lean, mean, and vulnerability free. Another is doggedness: we put in the work to keep our dependencies on the bleeding edge, which lets us fix vulnerabilities before they’re published. But the only thing better than Chainguard doing this work for you is nobody having to do it at all (especially if it saves you a rebuild).

‍

Trevor Dunlap, a PhD candidate at NC State University, spent the summer at Chainguard working on automated tooling to identify false positives in vulnerability reports: cases where a software application depends on a library with a vulnerability, but that vulnerability is not exploitable. As you’ll see, this is promising technology that can save open source projects and the organizations that use them from a lot of toil addressing vulnerabilities:

‍

• Past research finds that only 15% of packages with vulnerabilities are actually used in production—and only 2% are exploitable.

• Tools like govulncheck exclude these, but rely on manually-curated metadata that’s absent from 63% of vulnerability reports.

• Our tool finds “vulnerable symbols,” which help rule out vulnerabilities, succeeding for 89.3% of tested vulnerabilities.

‍

False positive vulnerabilities

Your container image scanner may find hundreds of vulnerabilities, but do you really have to worry about them? By using runtime analysis, Sysdig finds in their 2023 Cloud-Native Security and Usage Report that only 15% of packages with vulnerabilities are actually used in production—and only 2% are exploitable! This suggests that the vast majority of vulnerabilities are false positives. But don’t rest too easy: an attacker only needs to find one vulnerability they can use. So even though the median vulnerability probably isn’t anything to worry about, as long as there’s one exploitable vulnerability you could be in trouble. Further, vulnerabilities can come from anywhere in the dependency chain:

So what can you do about this? Many organizations burn double-digit percentages of their engineers’ time painstakingly updating packages and triaging vulnerabilities in dependencies. Others keep a spreadsheet, and try to manually determine whether these vulnerabilities are exploitable. Besides being mind-numbingly tedious, this process is really easy to get wrong—and it just takes one mistake to sink you.

‍

The vulnerability management community has created Vulnerability Exploitability eXchange (VEX) to allow expressing “this vulnerability is a false positive” (along with a justification) in a machine-readable way. Chainguard works on one implementation, OpenVEX. But currently, populating VEX statements is a manual procedure. Can we do better?

‍

Case study: govulncheck

The Go language community was sick of this problem, and their solution is govulncheck. Go maintains a vulnerability database containing reports of vulnerabilities in Go modules. These database entries identify vulnerable functions, not just the entire package. Using this information, the govulncheck tool can determine whether those functions are present in your application, or whether they got optimized out by the compiler. It can then ignore those vulnerabilities in its ultimate report.

‍

This is extremely handy, but with two problems. First, it only works for Go. Second, it takes a lot of manual work: humans populate the Go vulnerability database (and the vulnerable function identification in particular) by hand. Our dream is to scale this technique to all languages (and even to cross-language applications and images) without adding manual work for each one.

‍

What’s in a vuln?

To scale false positive detection, we need to consider how a human handles reports of new vulnerabilities in a dependency. At a high level, the process has two steps:

‍

• Determine under what conditions a dependent application is vulnerable.

• Determine whether your application definitely satisfies, definitely does not satisfy, or may/may not satisfy those conditions.

‍

Of course, step (1) is impossible to do in full generality. But we can pick heuristics that help us reach verdicts in step (2). The obvious example—what govulncheck does—is reachability: if the application does not call a vulnerable function, the application is (in most cases) not vulnerable. Reachability is a great heuristic to automate. First, we can try to identify vulnerable functions automatically in a two-step process: finding vulnerability fixing commits, and looking at commit diffs to find vulnerable functions. Second, computers can automate call graph analysis to check reachability.

‍

Missing patches

Vulnerability databases have a field to identify the patch that fixes a vulnerability. This is a huge boon for reachability analysis: if the patch removes a vulnerability, call graphs that don’t include the changed code must not be exploitable. There’s just one problem: the majority (63% in GHSA) of vulnerability reports don’t include these patch links.

‍

Trevor’s past work with his NC State collaborators aim to identify these missing patch links (so-called “vulnerability fixing commits,” or VFCs) with a tool called vfcfinder, which uses modern AI techniques. In testing, vfcfinder found the true VFC 80.1% of the time, and 98.6% of the time the true VFC was among its top 5 candidates. The team was able to contribute VFC data to over 200 GitHub security advisories.

‍

Vulnerable function identification

With the patch that fixes a vulnerability in hand, Trevor turned this summer to identifying vulnerable functions. The Go vulnerability database presents a useful ground truth, as the Go security team has done this by hand. The tool uses CodeQL on the patch to identify the affected function. This may be private, so dependent packages don’t use it directly. The tool then uses CodeQL’s call graph analysis to find vulnerable entry points to the package:

Running on 149 vulnerability reports, this tool found all of the reported vulnerable entry points 89.3% of the time. In at least one case where its results differed from the upstream vulnerability database, the Go security team confirmed that the tool did the right thing, and the report was missing a function.

‍

What’s next?

The tool and analysis are available on GitHub. While preliminary efforts were limited to Go due to the availability of a ground truth dataset, in the future we hope to extend the analysis to other languages. Long-term, we hope that other language ecosystems adapt this technology, which will enable something like govulncheck everywhere! We also hope that these tools can make the Go security team’s life a little bit easier. You can contact the Chainguard Labs team about this research at labs@chainguard.dev.

‍

If you want to avoid having to deal with all of these vulnerabilities at all, Chainguard Images can keep you at “Inbox Zero for CVEs.” Our Images are constantly rebuilt to be on the bleeding edge. Further, Chainguard Images users can use the Wolfi advisory feed to stay at CVE Zero without even updating the images you use.

‍

This post is part of a series highlighting the work of Chainguard’s 2023 summer interns.