Cryptographic Signing for Containers

Introduction

In May of 2021, the United States Executive Branch released an Executive Order on Improving the Nation’s Cybersecurity. One of the key focus areas for this order was enhancing software supply chain security, with requirements around addressing supply chain risks by, in part:

• Securing development environments with strong access controls
• Using automated code scanning to find and remediate vulnerabilities in software
• Maintaining trusted supply chains by ensuring the integrity of code
• Keeping accurate and up-to-date data on the provenance (i.e. origin) of software components

This blog focuses primarily on helping customers understand software supply chain security in the context of integrity and provenance—specifically, how cryptographic signatures can be used to simplify the process of ensuring the integrity of container images as they move through your software supply chain. We will also discuss how signing can help organizations validate their container images are coming from a trusted publisher, and how signing can be integrated with code scanning and approval workflows to facilitate a secure software supply chain.

To be successful, signing and verification should be easily implemented and integrated with DevOps processes, ideally not placing undue burden on development teams to manage cryptographic keys and certificates. While this blog primarily covers signing container image manifests and related artifacts, cryptographic signatures can also be used to sign/verify documents, authentication tokens, software packages, and more.

Today, building containers involves creating an image and putting it in a registry such as Amazon Elastic Container Registry Public (ECR Public), or Amazon ECR private registry; developers can then deploy containers from these repositories. Developers use code pipelines to build and deploy their container images. Building integrity verification for open source container images (as well as images built locally) into your CI/CD pipeline can reduce the risk of software supply chain attacks and provide continuous confidence to businesses using these container images across multiple business units.

Put simply, we will examine the questions:

1. What are cryptographic signatures, and how can they be used in a container build pipeline?
2. How can organizations use signing to ensure that their container images are approved for use and have been verified as meeting their security standards?
3. How can developers use signing to verify the container images they create haven’t been tampered with after they’re vetted and approved for use?

What is a cryptographic signature?

To start, let’s define some terms and concepts—what is a cryptographic signature and what benefit does it provide?  As you read this blog in your web browser, you’ll find a lock icon in your browser’s address bar, indicating the connection to this page is secured by transport layer security (TLS). Your browser automatically trusts the certificate used to establish the TLS connection, as the certificate has a verifiable chain of trust that extends up to a trusted root certificate that is installed in all major operating systems and web browsers—in this case, the root certificate for Amazon. A similar trust mechanism is used to verify the identity of the person or organization who creates a cryptographic signature of a container image.

A typical cryptographic signature consists of (at least) two components: an encrypted hash of the data to be signed, and a certificate that provides identity information about the signer. In the simplest terms, a hash is a one-way (i.e. irreversible) function that takes data of any size and outputs a fixed length, unique string. Hashes are deterministic, meaning the same data will always produce the same hash, and any change to the data (however small) will produce a different hash from the original. This latter point is important when we consider that we are verifying the integrity of a container image: if the image has changed at all, the resulting hash of that image manifest will also change.

The hash of our original file would be encrypted with the private key of the signer, which should be treated with care: this private key uniquely identifies the signer and ensures the signature cannot be generated by anyone other than the signer. Take a look at Figure 1 for a view of a generic signing process in action:

Figure 1. Generic signing process

Often the signature will also include a timestamp to indicate exactly when the signature was created. You can imagine signing a contract on paper: you include some information about yourself, perhaps your full name and address, as well as your written signature and the date when you signed the contract.

In cryptography, this signature can then be checked against the artifact being validated to tell if the data has been modified since it was signed by decrypting the signature with the public key (typically included in the certificate along with the signature) and comparing the resulting decrypted hash to a hash of the original file. If the two hashes do not match, the original data has been changed in some way since it was signed. We can see this mechanism in action in Figure 2 below:

Figure 2: Generic signature verification process

As alluded to previously, the signature can also provide what’s called non-repudiation: the idea that the signer cannot deny having signed the file. Access to the signing key is needed to generate the signature. In practice, this means that someone verifying the signature of a file can be reasonably sure the file was actually authored or signed by, for example, Amazon Web Services.

This mechanism is useful when we look at securing container images in our software supply chain, because we can sign the container images we create ourselves; or we can download images from a public image repository, run static code analysis scans to verify the images don’t contain known vulnerabilities, and/or configure them to meet our organization’s security standards. Next, we can sign the container image manifests to indicate they are trusted and approved for use in our organization.

Some organizations may even want to create multiple signatures on an image, allowing them to retain the signature from the public registry where they originally pulled the image, or enabling developers on different teams (in the same organization) to sign an image to indicate it is approved for their specific use case.

Integrating signing and verification into the container build process

When building a container image, the vast majority of customers will start with a base image pulled from a public container registry, store it in a private registry, and build upon it to suit their specific use case or workload. Customers may also host their own private container registry with pre-approved images that can be similarly customized to meet their needs. But in either case, the process would be the same—developers pull the image from an image registry and configure it to suit their application’s requirements or organizational standards.

Organizations will typically have specific requirements for how their container images are built and secured. This may mean checking for known vulnerabilities and Common Vulnerabilities and Exposures (CVEs), running approved static code analysis tools against the image, or maybe disabling certain services/packages that the organization deems as insecure. For example, AWS customers will often use ECR Enhanced Scanning (an integration between ECR and Amazon Inspector) to scan container images for known vulnerabilities and misconfigurations.

From now on, we’ll refer to this process of checking for vulnerabilities and configuring the image to meet your requirements as hardening. Images that have been hardened and approved for use in your organization can be referred to as verified or approved images.

However, organizations need mechanisms for ensuring only approved images can be promoted to different environments and eventually deployed for use in production, and that those approved images are not tampered with after they’ve been marked as approved.

Restricting who can push images to a registry is one method for ensuring that only trusted entities are able to import images to your registry; but this does nothing to help others verify the integrity of a container image after it has been pushed to the registry. If you need to investigate potential image tampering, you need to analyze registry logs to see who imported the image in question. As container images are often reused in various environments and by different development teams, a robust logging infrastructure is required to get a comprehensive view of an image’s lifecycle. Also, identities and access controls do not cross organizational boundaries. But a signature associated with an image manifest will travel with the image, even if it leaves your organization. Solving for these factors without using signing can add significant complexity, especially for large organizations with many developers and projects.

You can also use image tags to denote that an image is approved for use in your organization (e.g., myimage:approved), but again: tags don’t provide a method to verify if the image has been tampered with after it has been approved for use. Also, any developer who can push an image to the registry can also tag images, meaning there’s no simple method to ensure only specific developers or teams are able to mark an image as approved. In contrast, using signatures means that only the developers or teams with access to a valid signing key are able to generate a signature that indicates the image is approved.

Similarly, a simple mechanism for verifying the integrity of a container image is to compare the image manifest digest in the registry to a hash of the image manifest after pulling the image from the registry. In this case, we can be sure the image integrity is maintained, but there is no non-repudiation. Non-repudiation requires proof that the image manifest was signed by a specific entity, which cannot be provided by a hash that can be generated by anyone. Cryptographically signing an image manifest ensures that only those with access to the signing key can generate signatures and indicate an image is approved for use—and it allows us to verify the integrity of images even after they’ve been pushed to a public or private registry.

Customers may choose to utilize a combination of access controls on the registry, image tagging, and hash verification in order to provide integrity verification and non-repudiation: but implementing this effectively is complex and it is easy to make mistakes. Cryptographic signatures allow for integrity verification and non-repudiation, and provide defense in depth when combined with standard access controls, tagging, and monitoring.

In summary, we can use signatures to solve two challenges: denoting that a container image has went through our organization’s hardening process and is now approved for use in our organization, and providing a mechanism for consumers of the image to verify its integrity after it is approved. After the build or hardening process is complete, we can cryptographically sign the image manifest with our organization or development team’s private key to state: “this container image has been deemed secure (by the signer) and is ready for use.” This signature can be used in approval workflows as well as to verify the image’s integrity.

As different teams within the same organization may have unique security standards and risk posture, some customers may even attach multiple signatures to the same container image manifest, to denote that their specific team or department has gone through the hardening process and are approving this image for use in their pipeline. Referring to Figure 3 below, we can see a high-level container build pipeline diagram and where signing and verification can be implemented:

Figure 3: Where do signing and verification integrate?

Time changes all things…

Maybe you’ve already identified a problem with this approach: the hardening process is a point-in-time configuration. What is deemed as secure can change over time as new vulnerabilities are found, or your organization’s security standards are modified. This is why we often add timestamps to signatures—to be clear about when this code or image was signed, and differentiate older signatures from newer signatures.

This also introduces a potential need for revocation: the ability to say “this signature is no longer valid.” Revocation may be necessary because new vulnerabilities were discovered in previously trusted software packages or images, or because the private key of the signer was compromised. In the case of key compromise or overly permissive access to a signing key, revocation can prevent entities from assuming the identity of the original signer in order to distribute vulnerable code or container images that contain malicious code.

A key compromise does not necessarily mean that all software signed with the compromised key is vulnerable, only code signed after the key was compromised is unambiguously at risk. However, consider a situation where a software provider uses the same root of trust (e.g. signing key) for several subsequent versions of the same container image, but later discovers that their signing key used to sign the image manifests was compromised.

The software provider might issue a revocation for signatures created with that key, in order to prevent a malicious actor from using their signing key to impersonate them and distribute vulnerable or malicious code. But when consumers of these images from a public registry see that the signing certificate has been revoked, they may decide nothing signed by that key is trustworthy and also discontinue use of previous image versions that are not technically at risk. The pros and cons of revocation is a complex discussion that we won’t cover fully here, but it is worth being aware the functionality exists.

SBoMs, provenance, and software supply chain security

Software Bill of Materials (SBoM)

Another important concept in software supply chain discussions is the idea of a Software Bill of Materials (sometimes called a Systems Bill of Materials) or SBoM. You can think of an SBoM as a recipe or a list of the components that make up a whole product.

Consider this example: you run a bicycle shop that puts together bikes and sells them. You have a list of the components you need to purchase in order to build the bike: you need wheels, handlebars, a frame, a chain for the tires on the bike, etc. Maybe you discover a new manufacturer that makes a chain that is significantly cheaper and appears to be high quality. You start using chains from this manufacturer to build your bicycles and sell them to your customers.

A year later, you hear complaints that the chain was actually defective and has been breaking—causing failures and physical injury! You want to notify your customers who purchased a bike with this chain that they can come in and have it replaced before any damage is done. But how do you know which customers purchased a bike built with this defective chain?  Hopefully you have a bill of materials that includes all the components you used to build the bikes before you sold them. Then you can consult your purchase orders to see which bikes contain that chain in the list of components. Similarly, customers who see a recall notice for the defective chains could use the provided bill of materials to independently verify if the bike they purchased is at risk.

This applies to software as well. We may include software packages in our code or container images that we originally believed to be secure, but we later find is flawed or even malicious. It is important to have an SBoM associated with software or container image builds that provides a clear indicator of the software components in the build.

Think back to the hardening process. There may be software packages that were originally approved for use in your container images that we later find have newly discovered vulnerabilities: we need to use the SBoMs to determine which container images contain the flawed or malicious code so we can fix the vulnerabilities and/or revoke the signatures on those images and artifacts.

Provenance

Let’s briefly examine another important concept: provenance. Provenance refers to a place of origin or history of ownership. It’s often used when purchasing art or antiques for the buyer to better understand who discovered or created the item, as well as the custody of the item over time.

We can apply the same logic to building software: developers need to understand the provenance, or history, of the software packages and artifacts they use as components in their application or service. Software provenance can be understood as a strategy for providing verifiable attestation of the origin of code running in your environments. In other words, all running code should be tied to a specific set of commits that come from a known and verifiable author who has the appropriate permissions. Naturally, you want to ensure the integrity of provenance information to prevent falsification by malicious actors.

Cryptographic signatures are an important control necessary for ensuring the integrity and non-repudiation of a project’s SBoM, associated design artifacts, and provenance information—not just the image manifest.

Signatures, approval workflows, and secure SDLC

Let’s put all these concepts together and imagine a hypothetical container image build pipeline at a large organization with many development teams and a private image registry. The first development team to use the private registry will start by pulling an existing base image from a public registry, after using the image’s signature to validate the integrity of the base image and to confirm it is signed by the expected publisher. The original signature would also be pulled into the developer’s private registry along with the base image. Next, they will run vulnerability and code scanning tools on the image and modify it to meet their security standards and design needs.

After the developers create their container image using the public image as a starting point, they should sign the new image manifest with their team’s signing key, and push the signature and the associated signed artifacts (e.g. SBoM) in their private registry to live along with the container image they just built. The original signature pulled from the public registry is still kept in the private registry along with the original base image, but now the new image manifest has a signature noting that this specific development team has created the image and approved it for use in the build environment.

After the build phase is complete, the developers want to begin testing, but first they must pass an approval workflow to promote their new image into the testing environment. They might need to perform a code review or run a series of scans on their image before it can be used for testing. Afterwards they could sign the image again with another key to indicate it has undergone this process. The development pipeline can verify this signature is in place as part of the code promotion process, and prevent builds that do not have the appropriate signature from moving on to higher environments.

The organization may have many of these control gates, or approval workflows, as images move from build all the way to production, but the container image signatures can be used throughout the process to validate the image at points determined by the organization’s software development lifecycle (SDLC) procedures.

Conclusion

In this blog post we’ve defined cryptographic signatures, stated why container image signing is valuable, and how image signing fits into your software supply chain and CI/CD pipeline. We also introduced some important concepts like signature revocation, SBoMs, provenance, and approval workflows. Lastly, we walked through a basic example of integrating image manifest signatures into your software development lifecycle.