How much time is wasted triaging known exploits?

‍

TL;DR: Our scan of 230 popular Bitnami container images revealed that seven percent contained CVEs listed in VulnCheck’s Known Exploited Vulnerability (KEV) catalog. After 10 hours of detailed triaging, we determined these CVEs were not exploitable in the specific contexts of these containers. The point? Triaging CVEs, even those that are known to be exploited, can be a giant time suck. Fortunately, our analysis of historical CVEs that affected Chainguard Images shows that, on average, we remediate them in less than three days. Zero CVEs, to a first approximation, mean zero CVE triage.

‍

Vulnerability (aka CVE) management is about assessing risks — asking, "Do these vulnerabilities harm us?" A vulnerability truly harms an organization if it can be actively exploited to damage systems, breach security, or disrupt operations. These exploits can be costly — according to IBM, the global average cost of a data breach in 2023 was over $4 million. However, determining if a vulnerability is harmful to your organization is a time-consuming task and requires specific expertise.

‍

To demonstrate what this looks like in practice, I focused on CVEs known to be exploited in the wild, using data from VulnCheck’s Known Exploited Vulnerability (KEV) catalog. I scanned 230 different Bitnami images in early April 2024 with Grype. On average, each container image contained 143 CVEs. Out of the total set of CVEs in the Bitnami ecosystem, three of those CVEs appeared in VulnChecks KEV catalog, affecting seven percent of the latest Bitnami images.

‍

I manually triaged these three CVEs to answer this question: "Do these known-to-be-exploited vulnerabilities matter?" The surprising answer? Not one of them could actually be exploited in these container images. I then examined how fast Chainguard Images have historically fixed CVEs.

‍

Our findings are two-fold:

1. We burned three hours of staff time per exploitable CVE triage: Assessing one CVE for exploitability in a container image took about three hours. Even if a CVE is in the KEV catalog, it doesn't automatically mean a container image is vulnerable to it. Unfortunately, it does mean your company will invest significant time and energy in triaging it, because each of these CVEs will be unique within their own environment.
   ‍
2. Chainguard Images fix CVEs, on average, in 2.5 days: Chainguard Images has addressed 10 CVEs from the VulnCheck KEV catalog, affecting 145 packages. All have either been fixed or identified as component mismatches.

‍

These findings beg the question: Why spend time triaging known exploited CVEs if Chainguard Images already has the fix?

‍

The remainder of this blog is really for tech enthusiasts and cybersecurity experts. I’ll briefly discuss the differences between vulnerabilities and exploits and introduce some databases for prioritizing vulnerabilities — specifically, the Known Exploited Vulnerability (KEV) catalogs. Finally, for those with too much time to kill, we'll explore triaging vulnerabilities in container images appearing in the KEV catalogs. 

‍

Vulnerabilities, exploits, and KEV

A vulnerability refers to a weakness in an information system, system security procedures, internal controls, or implementation that could be exploited or triggered by a threat source [NIST]. An exploit is a piece of code designed to take advantage of a vulnerability or security flaw [Trend Micro]. The relationship between vulnerabilities and exploits is a dependent one — exploits require vulnerabilities to function. However, not every vulnerability will necessarily be exploited. Transforming a vulnerability into an exploit requires specific skills and circumstances, making some vulnerabilities more critical than others.

‍

To help organizations better prioritize these threats, the Cybersecurity and Infrastructure Security Agency (CISA) introduced a Known Exploited Vulnerability (KEV) catalog in 2021. This catalog lists vulnerabilities that are known to have been exploited. Additionally, in early 2024, VulnCheck released a similar KEV catalog. This updated catalog offers additional features such as exploitation timelines and prioritization advice, and it includes over 2,000 entries — double the number found in CISA’s original KEV catalog.

‍

Despite these efforts, both catalogs likely underrepresent the actual number of exploited vulnerabilities. This underrepresentation stems from the fact that not all breaches are reported or detected. Nevertheless, these catalogs play an essential role in cybersecurity, enabling organizations to identify and prioritize threats more effectively, thus enhancing their overall security posture.

‍

The detailed pain of triaging

Here, we discuss the details of triaging vulnerabilities in base container images in practice. This process is time-consuming, taking us, on average, three hours to complete per CVE. 

‍

After scanning 230 of the latest Bitnami container images with Grype, I found that three CVEs also appeared in the VulnCheck KEV dataset. I then selected an image containing one of those CVEs to understand what the triaging process actually takes.

‍

The process of how I planned my triaging efforts:

• Understand the CVE: Collect details on the severity and affected versions. Find proof of concept exploits.
  ‍
• Locate the vulnerability in the container: Use scanning tools to identify affected components.
  ‍
• Assess exploitability: Evaluate potential attack vectors and risks. 

DISCLAIMER: Just because I did not find these three CVEs exploitable in the example Bitnami images doesn't mean they are secure in another container.

‍

CVE-2019-6110: The first three hours of time I’ll never get back

This CVE shows up in an exploit known as “SSHtranger Things.” The underlying vulnerability could allow an attack to alter what the client sees from a Man-in-the-middle attack. This CVE appears in seven of the Bitnami latest images.

‍

‍

We’ll focus on the Bitnami/git image within this example. The below image is a scan from Grype.

‍

‍

Two details caught my eye:
‍

• OpenSSH version match differences: The CVE mentions OpenSSH version 7.9, however the Grype match is on OpenSSH version 9.2. From the Grype match, it appears a non-vulnerable version is used.
  ‍
• Severity rankings differences: NVD ranks the CVE severity as medium, however Grype uses the Debian severity of negligible or unimportant.

‍

CVE-2019-6110 is matched in four image components, all related to SSH. Interestingly, the package match is “1:9.2p1-2+deb12u2,” which is version 9.2 of SSH. However, the original CVE and CPEs mentioned that the vulnerability was fixed in version 7.9 of OpenSSH. 

‍

Next step, let’s confirm SSH version 9.2 is used in the image:

‍

‍

Confirmed. The latest Bitnami/git image uses a newer non-vulnerable version (at least for the specific version of CVE) of OpenSSH.

‍

Why was this matched by Grype if the OpenSSH versions are different? Grype also consumes data from the Debian security tracker and matches on the CPE: 1:9.2p1-2+deb12u2. Let’s check what upstream maintainers in OpenSSH say for CVE-2019-6110:

‍

“We don't consider the report relating to stderr to be a vulnerability - lots of stuff depends on stderr being present (e.g. login warning banners that some people inexplicably love) and it's impractical for scp to selectively process them. The machine you just logged into can print junk to your screen, so what?”

‍

But what’s that old saying in security? Don’t trust, verify. {Insert an additional hour of time trying to get this exploit to work. No success.}

‍

CVE-2019-6110 conclusion for Bitnami/Git: Not impacted. Component mismatch versions and inability to get an exploit working within this environment concludes to the final not impacted decision.

‍

CVE-2022-0543: The next three hours of my life wasted

This CVE relies on an exploit of a local Lua sandbox escape impacting Debian releases of Redis, which would allow a remote code execution. The CVE shows up in three of the latest Bitnami images (redis, redis-cluster, and redis-sentinel).

‍

‍

This one seems promising since Bitnami is based on a minimalist version of Debian, Minideb. We’ll focus on the bitnami/redis:latest image. A portion of the Grype results for Bitnami/redis:

‍

‍

Key observation:
‍

• Binary Redis match: Grype matched on the binary of Redis for version 7.2.4. Seems promising so far.

‍

However, this vulnerability is a combination. It requires Redis, Debian, and Lua. First, I’ll confirm the container is using Debian by running:

‍

‍

Yep, Debian version 12 (bookworm)! 

‍

Let’s look back at the CPEs from the NVD CVE to determine the impacted Redis and Debian versions:

‍

‍

All versions of Redis (wonderful) and Debian 9.0, 10.0, and 11.0. I don’t care about the Ubuntu CPE because the image is Debian-based.

‍

The Bitnami image is running Debian version 12. However, Grype's output matched on the Redis CPE, causing this alert overall. Now, I’m a bit doubtful this vulnerability will impact this image since it’s a mismatch of Debian versions.

‍

After some digging online, I found an additional PoC walkthrough for CVE-2022-0543 showing that the exploit relies on the Lua library file /usr/lib/x86_64-linux-gnu/liblua5.1.so to be functional. 

‍

Let’s check if the file is within the image:

‍

‍

The necessary file doesn’t appear to be within the container. 

‍

CVE-2022-0543 conclusion for Bitnami/Redis: Not impacted. The underlying necessary file to carry out the exploit isn’t present in the image container due to the newer version 12 of Debian that Bitnmai uses.
‍

CVE-2023-44487: Final three hours of my life down the drain

This is a powerful DDoS attack. Due to this vulnerability, Google reported mitigating attacks with over 398 million requests per second. It appears in ten of the latest Bitnami images.

‍

‍

137 reference links! An absurd number of “helpful” reference links.

‍

I’ll focus on the bitnami/git:latest image. A portion of the Grype results for Bitnami/git:

‍

‍

Observation from the Grype scan:
‍

• Standard Go library match: A match appears within the standard Go library. However, is the vulnerability related to an HTTP/2 protocol? This indicates that this match is probably due to the standard Go library having a dependency on the HTTP/2 protocol somewhere. But where is a Go component within the image?

‍

Let’s dig deeper into the Grype JSON report to find the actual match details:

‍

{
 "matchDetails": [
  {
   "type": "cpe-match",
   "matcher": "go-module-matcher",
   "searchedBy": {
    "namespace": "nvd:cpe",
    "cpes": [
     "cpe:2.3:a:golang:go:1.19.8:-:*:*:*:*:*:*"
    ],
    "Package": {
     "name": "stdlib",
     "version": "go1.19.8"
    }
   },
   "found": {
    "vulnerabilityID": "CVE-2023-44487",
    "versionConstraint": "< 1.20.10 || >= 1.21.0, < 1.21.3 (unknown)",
    "cpes": [
     "cpe:2.3:a:golang:go:*:*:*:*:*:*:*:*"
    ]
   }
  }
 ],
 "artifact": {
  "id": "6f46f0c385b7ecf3",
  "name": "stdlib",
  "version": "go1.19.8",
  "type": "go-module",
  "locations": [
   {
    "path": "/usr/bin/git-lfs",
    "layerID": "sha256:b7d44b"
   }
  ]
 }
}

‍

Location of the vulnerability from the Grype output:
‍

• The vulnerability is associated with a file located at /usr/bin/git-lfs and is a go-module. This is notable because Git LFS is an extension for Git that allows users to manage large files separately from the main Git database, but it seems the vulnerability detection is associated with the underlying Go version used by Git LFS itself? 

So we know the match is due to a vulnerable version of Go located in the go-module Git LFS. We can run a tool called GoVulnCheck to perform a reachability analysis for this CVE to determine if Git LFS calls a known vulnerable function for this CVE. However, we must first check if the CVE is within the Go vulnerability database.

From a manual search, it looks like both GO-2023-2102 and GO-2023-2153 are the related GoVulnDB Ids with CVE-2023-44487. 

Steps to run GoVulnCheck:
‍

• Identify the version of git-lfs running in the container (it’s v3.3.0). 
  ‍
• Locally clone the git-lfs repository to run GoVulnCheck.
  ‍
• Then validate the results of GoVulnCheck. 
  ‍

After doing each of those I get the following results:

‍

‍

None of which were for GO-2023-2102 or GO-2023-2153. 
‍

CVE-2023-44487 conclusion for Bitnami/Git: I’m about 75 percent sure this vulnerability isn’t exploitable within the Bitnami/git:latest image based on missing calls to the vulnerable functions identified by the Go security team with GoVulnCheck. Additionally, a lot of Go apps have used a version of the standard library affected by this. But relatively few Go apps use the standard library's HTTP2 server — so if they aren't using it, there's no HTTP2 serving vulnerability.
‍

Final thoughts

Overall, that was painful. I spent around three hours on each CVE, digging to the root of understanding if a particular vulnerability affects an image. The original expectation is that if the CVE were in KEV, those container images would also be exploitable. Surprisingly, the answer was no; they weren’t exploitable. But I still had to go through a rigorous evaluation to be sure — just as you’ll have to do when CVEs show up in your container images that also appear in a KEV catalog.

So this brings us back to the original question: Why spend time triaging known exploited CVEs if Chainguard Images already has the fix? Chainguard will do the heavy lifting for you. We fix CVEs fast. In the past few months, we have, on average, fixed CVEs in approximately 2.5 days. That’s the time from when a CVE is published to when we push a fix in Wolfi. 

Want to learn more about Chainguard Images and how they can help reduce the time your team spends triaging and investigating pesky CVEs? Reach out to our team today.