The Dirty Secret of Cybersecurity Standards

And Other Notes from Chainguard’s First NIST Comment

Chainguard recently made its first public comment on a software security standard. When the National Institute for Standards and Technology (NIST), a U.S. government agency that creates technology standards, requested feedback on its cybersecurity framework, we decided to weigh in. So did over 100 other organizations and individuals.

To save you from the toil joy of reading these comments, this post summarizes Chainguard’s submission and, based on reading the other submissions, notes a dirty secret of information security standards.

Chainguard’s Comment

We decided to spare the reader a blow-by-blow, paragraph-by-paragraph examination of the standard, and instead offered a perspective informed by a commitment to open source security and making the world’s software supply chain secure by default. Our full response can be found on the NIST website.

First, we suggested that NIST’s cybersecurity framework focused excessively on the perspective of one single company trying to secure itself, which obscures the importance of collaborating on and implementing security fixes for the overall open source software ecosystem. Such a perspective is — in our opinion — too narrow, and projects and organizations that are helping to improve the security of the open source ecosystems on which they depend should be further championed. For instance, the open source digital signature project Sigstore and the activities of Shopify exemplify the ecosystem approach that we encourage more companies to adopt.

Second, cybersecurity frameworks like NIST’s ought to emphasize software integrity as a foundational prerequisite for software security. While proposals such as those related to software bills of materials (SBOMs) were popular among the many comments provided to NIST, measures that solely focus on inventorying and assessing software components overlook the fact that there have been tens, if not hundreds, of reported instances of software tampering. SBOMs are helpful, but SBOMs alone won’t stop tampering.

Third, the Chainguard comment encouraged the removal of anti-competitive practices in the cybersecurity software industry such as DeWitt clauses. These clauses prevent analysts and researchers from benchmarking software, including software security tools, without the permission of the tool creator. DeWitt clauses stifle the spread of knowledge of what security tools work and which ones don’t, reducing the security of all software, including open source.

The Dirty Secret of Cybersecurity Standards

While reading the submissions, it occurred to me that the dirty secret of the cybersecurity standards industry is that there is no consensus on how to objectively determine what security controls ought to be included in any standard. This does not mean that creating security standards is without merit, but it does mean that asking for feedback on standards leads to lots of conflicting advice based on relatively little evidence. The Internet Security Alliance comment, in fact, makes a related point.

For instance, one comment calls for adding “threat hunting,” or active searching for compromised devices on a company’s network, to NIST’s cybersecurity framework. But how should NIST (or any other standards body) decide if this control is worth adding to the framework? The NIST cybersecurity framework, like most frameworks, is silent on the matter. Consequently, it’s hard to assess many of the specific proposals made to NIST because, to quote the Big Lebowski, “Yeah, well, you know, that's just, like, your opinion, man.”

Adding security controls to a standard doesn’t have to be merely an opinion-based endeavor. Analyzing past attacks and what controls do and don’t help is possible—see one suggestive example here that investigates the SLSA framework—though it’s not easy. Other systematic approaches are also possible but are clearly currently under-utilized.

Until the Next Public Comment

Our commitment to open source software security and making the world’s software supply chain secure by default means that this won’t be our last public comment on software security standards. Until then…

John Speed Meyers is a security data scientist at Chainguard. If you are interested in learning more about Chainguard, our products, our services, and our commitment to open source software security, please leave us a note and we’ll get in touch.