Vulnerability fixes in plain sight: How your scanners are missing hundreds of vulnerabilities

TL;DR — An analysis of over 600 projects in Wolfi found over 100 security fixes that do not have an associated CVE. We scanned thousands of past release notes from open-source projects, finding over a 100 mentions of vulnerability fixes without official publicly recorded CVEs. The implication for your organization? Keep your software updated. Updating your software does more than fixing known vulnerabilities; it also comes with vulnerability fixes you would often miss.

‍

Patching vulnerabilities is a race — your organization against attackers — where the difference between winning and losing hinges on speed. For example, the infamous Equifax breach was the result of a two-month delayed patch for the Apache Struts Remote Code Execution Vulnerability (CVE-2017-5638), which led to losses exceeding hundreds of millions of dollars.

‍

The race to patch vulnerabilities typically starts when a CVE and the fixed version of the vulnerable software is released. However, most organizations rely on alerts from security scanners, such as software composition analysis (SCA) tools, to learn about these vulnerabilities. These SCA tools, in turn, depend on the completeness of vulnerability databases like the National Vulnerability Database (NVD).

‍

This reliance of vulnerability scanners on the NVD raises a key question: Are vulnerability databases missing already discovered vulnerabilities? To answer this question, this analysis examined the release notes of open source projects packaged in Wolfi. We used a mix of regular expression pattern matching, large language models, and manual review to determine if the release notes discussed a vulnerability fix and to check if those fixes had an associated CVE.

‍

The main finding is clear.

‍

An analysis of over 600 projects in Wolfi found over 100 security fixes that do not have an associated CVE: Vulnerability databases are incomplete. These “missing CVEs” leave organizations in the dark since SCA tools and the databases underpinning them lack information on these silent vulnerabilities.

‍

What does this mean? It means that scanners that rely on vulnerability databases can leave you with a false sense of security. One way to combat this is to stay on the bleeding edge of software updates. Fortunately for users of Chainguard Images — our suite of minimal, hardened container images — this problem is minimized because over 80% of Wolfi package updates happen within 24 hours after the release of a new version upstream.

‍

The remainder of this blog discusses how we found vulnerability fixes missing their CVEs.

Vulnerability fixes in plain sight

Detecting vulnerabilities without CVEs is not a new concept. These are often referred to as silent vulnerability fixes. Among academics, finding silent vulnerability fixes has been a recent hot topic (Sawadogo et al., Zhou et al., Dunlap et al.). Researchers have found hundreds of silent vulnerability fixes within open-source software. Consequently, the debates about silent vulnerability patching as a practice are also not new.

‍

Prior work on this topic has often analyzed underlying code changes during commits, genuinely looking for "silent" vulnerability fixes. However, we had a hunch that you didn't need to look that closely. That vulnerability announcements are coming from the maintainers directly in release notes announcements — and didn't make it into the form of CVEs.

To find a vulnerability fix missing a CVE, this research started with the release notes for a target project, searching for 230+ keywords related to security concepts. However, keywords aren't enough. We need to identify the context in which those keywords were used (Mistral AI's LLMs are helpful here). For example, someone could say, "We didn't fix any security issues," versus "We fixed security issues." If we find a release note mentioning a security-related issue, we search within OSV (Google's Open Source Vulnerability Database) for a known fix based on the prior version.

‍

OSV allows us to expand our search beyond just MITRE’s CVE List accessed in the NVD, as it curates data from multiple sources (e.g. GitHub Security Advisories and the Go Vulnerability Database). If a match for that version appears in OSV, we assume the release note announcement has a CVE; otherwise, we believe the security fix to be missing a CVE. For thoroughness, we manually validated our results.

‍

In total, we scanned 600+ projects within the Wolfi OS ecosystem and their associated release notes, which accounted for 3,000+ releases. We found over 100+ release notes that discuss security-related fixes but do not have an associated CVE.

‍

Below, we provide a few examples of what we saw. We first show what a release note looks like for discussing a vulnerability fix with a CVE, followed by other security fixes that are missing CVEs.

‍

The clear vulnerability announcement:

Sometimes, project releases are pretty straightforward and tell you the associated CVE. For example, the redis/hiredis release note:

CVE-2021-32765 is mentioned in the release note, and OSV has the CVE for <=v1.0.0 for hiredis. This example is a clear announcement and should be detected by your SCA tool.

An announced security fix, but missing a CVE:

On the other hand, security issues are mentioned but do not have a CVE:

A lot of projects use dedicated security sections in their release notes. Does that mean a CVE is associated with the “SECURITY” section? Nope, I couldn’t find anything for Consul v1.16.4. In the pull, the developers mention the impact is minimal. The scenario here could be the developers didn’t think this was serious enough to request a CVE. Still, it was severe enough to mention directly in the “SECURITY” portion of the release notes and backport to other versions.

‍

The secret vulnerability fix?

Sometimes, a randomly dropped security hint can be in the release notes:

The mention of “and a security fix” at the end of the first sentence suggests a security bug previously existed. When following the “See the full whats new for a list of all the changes,” we couldn’t find any mention of a CVE or security fix. Also, there was nothing in OSV. Could this just be a misstep or an incorrect release note on the maintainers’ part? Potentially. We’re not sure, but if you’re interested in digging further, you can check out all the commits that make up version 2.1.4 of Pandas to see if you can find a security fix.

‍

Final thoughts

Vulnerabilities are constantly being found and fixed. Perhaps surprisingly, some vulnerabilities never receive a CVE. Many organizations consume thousands of open-source projects and only monitor CVEs based on what their security scanning tool tells them. Here's the issue: when a scanner isn't aware of a vulnerability, your organization won't be either. That's what we found here — over 100 instances of releases fixing vulnerabilities, but missing assigned CVEs.

‍

We're not pointing fingers here. Being an open-source maintainer is hard, thankless work, and we don't want to pile more work on their plates. The point isn't that maintainers are lazy in filing CVEs or that their judgment is poor, but that they don't sometimes, which can impact users. So, until vulnerability databases or scanners are reliably automatically populated by monitoring repository changes to detect security fixes, it's best to keep your software as up-to-date as possible.

‍

At Chainguard, we automatically monitor when new software package updates are released. The median time for a Wolfi package to update is only four hours after the upstream release. This often leads to updates before your scanner is even aware of a vulnerability. Do you dream of being able to remediate vulnerabilities that you didn't even know existed? Contact us to learn more.