Zero CVE metrics on Cloud Run

At Chainguard, we make extensive use of Cloud Run, which powers the majority of our production environments. To surface metrics, we expose them via OpenTelemetry (aka OTel) and leverage the managed service for Prometheus to slurp those metrics into Stackdriver. However, when we started to integrate this, the first step was to add the OTel sidecar into our cloud run services:

+     - image: "us-docker.pkg.dev/cloud-ops-agents-artifacts/cloud-run-gmp-sidecar/cloud-run-gmp-sidecar:1.1.1"
+       name: collector

As good Chainguardians, of course the first thing we did was to run a CVE scan of this image:

grype us-docker.pkg.dev/cloud-ops-agents-artifacts/cloud-run-gmp-sidecar/cloud-run-gmp-sidecar:1.1.1
 ✔ Scanned for vulnerabilities     [51 vulnerability matches]  
   ├── by severity: 8 critical, 9 high, 23 medium, 0 low, 0 negligible (11 unknown)
   └── by status:   29 fixed, 22 not-fixed, 0 ignored 
NAME                                              INSTALLED             FIXED-IN    TYPE       VULNERABILITY        SEVERITY 
busybox                                           1.36.1-r15            1.36.1-r16  apk        CVE-2023-42366       Medium    
busybox                                           1.36.1-r15            1.36.1-r19  apk        CVE-2023-42365       Medium    
busybox                                           1.36.1-r15            1.36.1-r19  apk        CVE-2023-42364       Medium    
busybox                                           1.36.1-r15            1.36.1-r17  apk        CVE-2023-42363       Medium    
busybox-binsh                                     1.36.1-r15            1.36.1-r16  apk        CVE-2023-42366       Medium    
busybox-binsh                                     1.36.1-r15            1.36.1-r19  apk        CVE-2023-42365       Medium    
busybox-binsh                                     1.36.1-r15            1.36.1-r19  apk        CVE-2023-42364       Medium    
busybox-binsh                                     1.36.1-r15            1.36.1-r17  apk        CVE-2023-42363       Medium    
github.com/Azure/azure-sdk-for-go/sdk/azidentity  v1.4.0                1.6.0       go-module  GHSA-m5vv-6r4h-3vj9  Medium    
github.com/docker/docker                          v24.0.9+incompatible  25.0.6      go-module  GHSA-v23v-6jw2-98fq  Critical  
github.com/hashicorp/go-retryablehttp             v0.7.4                0.7.7       go-module  GHSA-v6v8-xj6m-xwqh  Medium    
github.com/mostynb/go-grpc-compression            v1.2.2                1.2.3       go-module  GHSA-87m9-rv8p-rgmg  High      
github.com/rs/cors                                v1.10.1               1.11.0      go-module  GHSA-mh55-gqvf-xfwm  Medium    
go.opentelemetry.io/collector/config/configgrpc   v0.94.0               0.102.1     go-module  GHSA-c74f-6mfw-mm4v  High      
go.opentelemetry.io/collector/config/confighttp   v0.94.0               0.102.0     go-module  GHSA-c74f-6mfw-mm4v  High      
libcrypto3                                        3.1.4-r6              3.1.6-r0    apk        CVE-2024-5535        Critical  
libcrypto3                                        3.1.4-r6              3.1.5-r0    apk        CVE-2024-4603        Medium    
libcrypto3                                        3.1.4-r6              3.1.6-r0    apk        CVE-2024-4741        Unknown   
libssl3                                           3.1.4-r6              3.1.6-r0    apk        CVE-2024-5535        Critical  
libssl3                                           3.1.4-r6              3.1.5-r0    apk        CVE-2024-4603        Medium    
libssl3                                           3.1.4-r6              3.1.6-r0    apk        CVE-2024-4741        Unknown   
openssl                                           3.1.4-r6              3.1.6-r0    apk        CVE-2024-5535        Critical  
openssl                                           3.1.4-r6              3.1.5-r0    apk        CVE-2024-4603        Medium    
openssl                                           3.1.4-r6              3.1.6-r0    apk        CVE-2024-4741        Unknown   
ssl_client                                        1.36.1-r15            1.36.1-r16  apk        CVE-2023-42366       Medium    
ssl_client                                        1.36.1-r15            1.36.1-r19  apk        CVE-2023-42365       Medium    
ssl_client                                        1.36.1-r15            1.36.1-r19  apk        CVE-2023-42364       Medium    
ssl_client                                        1.36.1-r15            1.36.1-r17  apk        CVE-2023-42363       Medium    
stdlib                                            go1.20.12                         go-module  CVE-2024-24790       Critical  
stdlib                                            go1.20.12                         go-module  CVE-2023-24531       Critical  
stdlib                                            go1.20.12                         go-module  CVE-2024-24791       High      
stdlib                                            go1.20.12                         go-module  CVE-2024-24784       High      
stdlib                                            go1.20.12                         go-module  CVE-2023-45288       High      
stdlib                                            go1.20.12                         go-module  CVE-2024-24789       Medium    
stdlib                                            go1.20.12                         go-module  CVE-2024-24787       Medium    
stdlib                                            go1.20.12                         go-module  CVE-2024-24785       Unknown   
stdlib                                            go1.20.12                         go-module  CVE-2024-24783       Unknown   
stdlib                                            go1.20.12                         go-module  CVE-2023-45290       Unknown   
stdlib                                            go1.20.12                         go-module  CVE-2023-45289       Unknown

I’ll note that as of my original drafting of this post (2024/04/24), the above image was built and published 24-48 hours prior, and still had a large number of CVEs:

It is still the latest available version.

Since Chainguard already curates an OTel sidecar as part of our Directory, we dug into the source code for the sidecar to see if we could replicate its behavior by relying on our own image and avoid the CVEs:

grype chainguard/opentelemetry-collector-contrib:latest
 ✔ Scanned for vulnerabilities     [0 vulnerability matches]  
   ├── by severity: 0 critical, 0 high, 0 medium, 0 low, 0 negligible
   └── by status:   0 fixed, 0 not-fixed, 0 ignored 
No vulnerabilities found

It turns out that we can leverage the existing sidecar for this purpose, and thankfully Cloud Run supports public DockerHub images (where we have a subset of our images), so we just need to feed the sidecar the appropriate configuration, and this can be passed as an environment variable.  We deploy all of our services via Terraform, so we use this little snippet (roughly):

    containers {
      image = "chainguard/opentelemetry-collector-contrib:latest"
      // config via env is an option; https://pkg.go.dev/go.opentelemetry.io/collector/service#section-readme
      args = ["--config=env:OTEL_CONFIG"]
      env {
        name  = "OTEL_CONFIG"
        value = file("${path.module}/otel-config/config.yaml")
      }
    }

The OTel configuration is loaded from a file here and inlined at deploy time, but this could also be outlined and shared via something like Secret Manager as well.

Ironically, I think that the entire Google sidecar image could be reimplemented with zero CVEs and no code based on the above pattern with the following Dockerfile:

FROM chainguard/opentelemetry-collector-contrib:latest
CMD ["--config=env:OTEL_CONFIG"]
ENV OTEL_CONFIG "<contents of the config file>"

Hopefully, you find this useful, and hopefully it reduces any toil you face in your own pursuit of zero CVEs. If this piques your interest about achieving zero CVEs, reach out!