Reducing vulnerabilities in Backstage with Chainguard’s Wolfi

Karl Haworth, is a Principal Staff Engineer on American Airlines Developer Experience products. His team is leveraging Chainguard Images to harden their software and strengthen the security of their software supply chain. Karl recently created an alternative, secure image for the Backstage open-source framework using Changuard’s wolfi-base image. In this guest post, Karl explores the decision to use wolfi-base and the benefits he has seen in reducing vulnerabilities and shrinking the overall Backstage attack surface.

‍

American Airlines is committed to enhancing the developer experience by implementing a frictionless self-service platform to create delightful developer experiences. By doing so, the developers can deliver value sooner to our customers. In order to achieve the stated goal, we adopted Spotify’s Backstage open-source framework for accelerating the development of our internal developer portal Runway. This marks my third endeavor at establishing a developer portal which has been successful due to a heavy focus on InnerSource practices. Backstage has proven instrumental in streamlining the development process, thanks to its community-driven foundation as a solid base to build upon.

‍

While Backstage prioritizes security, concerns arise when utilizing the base image, node:18-bookworm-slim, as it contains 74 vulnerabilities ranging in severity. This has prompted us to assess the security implications when using the docker build command and Dockerfile within the Backstage framework, as the default added an additional 330 vulnerabilities. I attempted using updated node and Debian images, but they also yielded similar results, prompting me to seek out different solutions.

‍

As I searched for answers, I came across Wolfi, Chainguard’s open-source undistro, and as I delved deeper, I noticed methods that appeared different from what I had encountered previously. Melange, apko, ... say what?! Continuing to dig, I observed enhancements in terms of minimizing packages, reinforcing supply chain security, decreasing image sizes, and reducing vulnerabilities, all while maintaining compatibility with glibc.

‍

Integrating Wolfi into Backstage

After finding a potential solution, it was time to evaluate how Wolfi could assist us by refactoring the Dockerfile. While the process appeared simple, I encountered issues due to some packages missing or incomplete, causing the application to not run smoothly. After some diligent searching and using apk search in the registry, I managed to resolve the missing packages. After investing an hour or two, I finally had a functional image.

‍

For instance, the standard Backstage Dockerfile comes with the libsqlite3-dev package by default, which is nonetheless absent in the Wolfi registry. To tackle this issue, I initiated a wolfi-base container on my local setup using a docker run command and then searched for the necessary package using apk. I iteratively minimized the package name until I successfully obtained a result that met my requirements. Below are the commands I employed for this task.

❯ docker run --rm -it cgr.dev/chainguard/wolfi-base:latest
16cab1bb5835:/# apk update
fetch https://packages.wolfi.dev/os/aarch64/APKINDEX.tar.gz
 [https://packages.wolfi.dev/os]
OK: 52397 distinct packages available
16cab1bb5835:/# apk add libsqlite3-dev
ERROR: unable to select packages:
  libsqlite3-dev (no such package):
    required by: world[libsqlite3-dev]
16cab1bb5835:/# apk search libsqlite3-dev
16cab1bb5835:/# apk search sqlite3-dev
16cab1bb5835:/# apk search sqlite3
apr-util-dbd_sqlite3-1.6.3-r0
soci-4.0.3-r2
sqlite-3.45.1-r0
sqlite-dev-3.45.1-r0
sqlite-libs-3.45.1-r0
sqlpad-7.4.1-r3
tcl-8.6.14-r0
16cab1bb5835:/#

Although I hadn't noticed any immediate benefits or improvements after the Dockerfile refactor, I still felt confident that swapping out the base image was a good decision. It was time to launch grype! The analysis revealed fewer packages, decreased executables, and only 15 vulnerabilities. Upon further inspection, six vulnerabilities were linked to the base image, while the remaining vulnerabilities stemmed from the libraries used in node. While 15 vulnerabilities may be considered low, I was determined to do better. Upgrading from node 18 to node 20 significantly reduced vulnerabilities at the OS level to just one. Fixing the remaining vulnerabilities involved making changes to the packages used in node.

‍

Achieving the desired objectives involved decreasing the number of vulnerabilities from 404 to just a few. 🤯 The successful reduction of the attack surface further solidified our goals, and we also increased speed in addressing packages due to Wolfi's efficient update and patching system.

‍

Additionally, Backstage has placed significant emphasis on being secure by default, but I noticed a crucial aspect was lacking. Integrating a hardened image contributes effectively to the entire ecosystem and aligns with the project's current objectives. It is imperative that no organization compromises security while leveraging open source, and this pull request is instrumental in reinforcing a secure by default approach.

‍

Benefits of using Wolfi base with Backstage

Working at American Airlines, I am grateful for the opportunity to give back to the open-source community through the appropriate channels. Recently, I was able to submit a Pull Request to Backstage, receive and incorporate feedback, and successfully had my changes integrated.

Following the integration, I had the privilege of sharing my success story with the Backstage community. The Backstage maintainers grasped the issue at hand, appreciated the contribution, and discussed potential plans to consider alternative options based on community input.

‍In retrospect, when reflecting on the modifications made, I aimed to maintain the flow and commands as closely as possible to prevent any maintenance complexities. The outcome closely resembled the original setup with a few tweaks to include additional or modified packages, as well as adjustments to users and permissions.

‍

To get started with swapping your default base to Wolfi, the Backstage Deployment Documentation talks about the contributed Wolfi alternative along with the Dockerfile contents on GitHub.

‍

By transitioning to a Wolfi base, we were able to save numerous hours spent triaging vulnerabilities, decrease the total number of vulnerabilities to a manageable level, establish supply chain verification, shrink our attack surface, and accelerate the process of regularly applying patches and upgrades.

‍

I am eager to engage in further discussions regarding establishing a fortified base as the standard option in Backstage with additional input from the community.

‍

The Chainguard team has been a great partner in supporting this work and helping to work through any technical hurdles or questions pertaining to Wolfi and wolfi-base. For additional information on how to make the most of the wolfi-baseimage, refer to the Chainguard Image Directory.