Principles for secure software distribution: Lessons from leaked Android platform signing keys

On November 30, 2022, the Android Security and Privacy Team project published a vulnerability report as part of the Android Partner Vulnerability Initiative, which tracks security issues for Android Original Equipment Manufacturers (OEMs). In this post, we cover:

‍

• What exactly happened?

• How bad is this vulnerability, and how will we recover?

• The secure software supply chain principles and technologies that would have helped, and how we can apply them in other contexts.

‍

What happened, and what next?

Android phones ship with a “platform” certificate as part of the device firmware. The signing keys for these certificates belong to OEMs, which use them to ship device updates. Any application signed using the keys in this certificate gets root-level privileges on the operating system. Łukasz Siewierski, a Google security researcher, discovered malware in the wild which was signed using several of these “platform” signing keys.

‍

Worse, these keys belong to several distinct vendors! This means that malicious actors have obtained access to signing keys from multiple vendors, and are using them to create malware which has escalated privileges when run on the corresponding device models.

While this attack is quite bad, there are a few ways in which impacted parties got lucky. First, only specific models of phone are affected. (However, at this time there is no public list of which ones.) Second, most users get applications only from the Google Play store; they’d need to sideload a malicious application to be affected—not a likely action for the average user. Finally, OEMs can push over-the-air device updates which rotate these keys, fixing the vulnerability.

‍

The best defense is a good offense: planning for compromise.

A common reaction to situations like this is to tell vendors very firmly that they should implement stronger security protocols. There are productive ways to do this, like writing and enforcing requirements (see the CA/Browser Forum’60+ pages of rules for certificate authorities), but such strategies will only go so far.

‍

Instead, we must accept that incidents will happen and plan accordingly. Thankfully, security engineers and researchers have been thinking hard about how to mitigate the likelihood of, detect, and respond to attacks for decades (excessive pessimism being the main thing that security professionals are good at). These ideas fall into the following categories.

‍

Reduce single points of failure. Traditionally, trust in signed software relies on a single party, who blesses an artifact as “good.” This makes any compromise of that source critical. Instead, more nuanced definitions of what needs to be true in order to trust an artifact can reduce danger. A simple example is requiring multiple signatures: an application might be trusted only if it’s signed by two distinct parties. This system is resilient to either one losing their key.

‍

We can take this further in a number of ways. For instance, thresholds: three parties who sign the binaries, and any two of them suffice. Or the parties can have roles: applications must be signed by any of the development leads and all of the members of the security team.

‍

Replacing the single certificate with a complicated policy about which applications to trust under which circumstances can be unwieldy, and approaches that hard-code such policies aren’t flexible enough to accommodate the wide range of OEMs which might want to have an array of configurations. The in-toto project addresses exactly this issue: once configured, it becomes simple to express any desired trust policy as a “layout."

‍

‍Transparency. It should be nearly impossible for attackers to strike unnoticed. One approach is transparency: forcing all activity, by actors both good and bad, to be public.

To do this, the Android ecosystem must create a shared, public log (backed by tamper-evident cryptography) of all signatures on Android applications (or other ecosystems). Then, when verifying a signature on an application, clients must check that the signature is in the log, and reject any signatures which are not. Even if an attacker does obtain a signing key, their malware is only useful if they publish the signature.

‍

This furnishes two benefits. First, detection: cross-referencing the log signatures from a particular OEM’s platform certificate against a list of the updates that OEM has published will show discrepancies in the event of a breach. Second, analysis: after an attack, this log can help determine how many unauthorized signatures exist, and the timeframe in which they were made (telling us when the incident occurred).

‍

In fact, there is a transparency log for Android binaries! Unfortunately, it’s not widely used—yet. Hopefully, this incident will prove a turning point for binary transparency adoption.

‍

Survivability. Even with compromise less likely and easier to detect, we must still be ready for when it inevitably happens. When a system is able to recover from a particular compromise, that compromise is survivable. This encompasses both operational concerns (a team should have playbooks in place for hypothetical scenarios, and perform regular drills) and system design (it should be possible).

‍

It just so happens that the Android platform signing ecosystem can fix this issue: OEMs can push an update replacing these certificates. However, this is fragile: vendors may first need to re-sign any existing platform releases. Further, users need to actually update their phones to be safe!

‍

An ideal system should build the ability to survive these sorts of attacks into the distribution and verification infrastructure itself. There are a number of challenges involved: revocation and rotation of keys, subtle “rollback” or “freeze” attacks where attackers replay signatures that were valid at a previous point in time, and issues of consistency.

‍

Fortunately, a project called The Update Framework (TUF) addresses these challenges. It provides a specification for a software update system following these principles, and recommendations for key management that minimize risks by putting the most important keys offline (for instance, in a safe deposit box). Further, TUF allows you to push rotations proactively, rather than hoping that users update before they’re exposed to an attack.

‍

Conclusion

As attackers increase their sophistication, our defensive technologies for software signing must grow more sophisticated as well. This post focused primarily on the Android ecosystem in light of recent events, but the lessons learned apply to all systems for distributing software securely, including the internal software supply chain of any organization.

‍

To protect yourself, it’s best to use tools with these principles built-in. For instance, Sigstore has transparency as a fundamental component, and uses TUF to manage its own root of trust, ensuring that if the worst were to happen, the project can safely recover.

‍

At Chainguard, we are building solutions that allow you to specify explicit policies for checking that software is trusted, ensure violations and unusual behavior are easy to detect, and enable efficient remediation after an incident. When we build in security this way, thoughtful design up-front results in products that make it easy to build your software supply chain following these core principles.