Can auto-patched container images pass the zero CVE challenge?
TL;DR
Auto-patching 20 popular container images with copacetic reduced the count of CVEs by, on average, 8%.
The big picture
Past analysis from Chainguard suggests that many popular container images have hundreds of known common vulnerabilities and exposures (CVEs). Software developers and security engineers are therefore understandably interested in ways to reduce this CVE count, a source of toil and risk that depletes valuable productivity and resources. One potentially promising method for reducing CVEs is using an “auto-patching” tool like copacetic, an open source tool for updating vulnerable packages in a container.
In a nutshell, copacetic is a patching engine. The results from a vulnerability scan tell copacetic which packages are vulnerable. Trivy is copacetic’s default vulnerability scanner, though there is plugin support available for adding additional third-party scanners. Packages with a newer, non-vulnerable version are then patched via buildkit. At that point, copacetic creates an additional “patch layer” on top of the existing image. See figure 1 for an explanatory diagram.
Copacetic uses distro package managers to apply patches, creating limitations on which images can be patched. Firstly, copacetic can only patch images that use certain package managers. Additionally, copacetic can only apply patches provided by the distro, not those patches provided at the application level.
Copacetic and CVEs
To test the effectiveness of copacetic as a vulnerability reduction tool, the following experiment was performed:
Use grype to find the number of CVEs and known fixes for:
The original image (Original)
The image after being patched by copacetic (Copacetic)
The image after being patched by updating packages (Update)
The Chainguard Image equivalent (Chainguard)
Step c was added to examine if simply performing a manual update of all operating systems packages produced similar CVE reduction percentages compared to copacetic.
The results are provided in Figure 2.
Out of the six images with fixable CVEs reported by grype, copacetic was able to patch two images: caddy and traefik. For the remaining images, copacetic reported that no patches were available and did not generate a new image to be scanned, so the original number of CVEs was kept. Overall, copacetic reduced CVEs by 8%. Updating packages also reduced CVEs by 9%. Chainguard Images were able to reduce the CVEs by 99% for all 20 images.
Copacetic reduced CVEs in the traefik image from 10 to two versus 10 to six with a simple update. Both methods resolved and missed the same CVE patches; updating packages, however, introduced four CVEs not found in the original scan of the traefik.
Additionally, only in the mongo image did updating packages outperform copacetic; update packages led to 44% reduction in CVEs versus a 0% reduction via copacetic. This is likely due to the fact that trivy missed the CVEs during the initial scan, which resulted in copacetic not being provided with the associated vulnerability information.
We also investigated the role of the base distro when reducing CVEs with copacetic. Using a total of 40 various Alpine, Debian, and Red Hat based latest images, copacetic did not generate any patched images despite the majority of images containing numerous vulnerabilities. This in part is due to the fact that unlike grype, trivy does not include GitHub Security Advisories in its reports that are ingested by copacetic. However, updating images also resulted in a negligible reduction of CVEs of less than 1% across all 40 images.
Our research finds that neither copacetic patching nor simple updates produced dramatic CVE reduction. Only Chainguard Images were the only instances where a 99% reduction of CVEs was achieved.
Conclusion
Overall, copacetic does reduce CVEs in containers, but it is limited by the quality of the ingested vulnerability reports and the availability of patches in the container’s underlying linux distribution. We’d love to hear from you if you are leveraging copacetic to help further our research efforts on the benefits impact of the tool — reach out today!
Ready to Lock Down Your Supply Chain?
Talk to our customer obsessed, community-driven team.