Cosign image signing In AWS CodePipeline

In this post we are going to show you how to integrate sigstore’s Cosign with AWS CodePipeline. Cosign is an open source project for container signing, verification and storage in an OCI registry. It can be utilized in various CI/CD pipelines like here in GitHub actions. Today we are going to show how to do just that in AWS CodePipeline.

For our example we will use Hashicorp’s Terraform. Terraform creates all the AWS Resources necessary to run the CodePipeline. The terraform code lives in the example repo in GitHub. In order to do this we will be provisioning:

• CodePipeline

  • Artifact S3 Bucket

  • IAM Role

  • IAM Role Policy

  • AWS CodeCommit Repo

• CodeBuild Project

  • IAM Role

  • Artifact S3 Bucket

  • Cloudwatch Log Group and Steam

• ECR - Container Repository

• KMS - Asymmetric key used for cosign key signing

AWS CodePipeline Overview

CodePipeline is AWS’s managed continuous delivery service. It automates the release pipelines for all kinds of applications as well as infrastructure changes. CodeBuild is their managed continuous integration service that compiles source code, runs tests, creates containers, and just about anything that is needed for your software pipeline. CodeBuild is serverless, all you need to do is provide the environment and the build spec. Let’s see how we can wire all that together to build and sign containers.

CodePipeline is made of stages, which have actions and actions are completed by providers. CodeBuild is a provider for CodePipeline. You pass information between CodePipeline stages using input and output artifacts.

In the example below we have a Source stage, which is the application’s source code, be it s3, AWS CodeCommit or GitHub. For this example our source is in AWS CodeCommit and is also provisioned by Terraform. When CodePipeline detects changes to the CodeCommit repo the latest changes will be downloaded and stored to the working artifact S3 bucket, this is the output artifact for the source stage and will be in the input artifact for the build stage. All of this is configured in the Stage’s action configuration highlighted below:

stage {
name = "Source"

    action {
      name     = "Source"
      category = "Source"
      owner    = "AWS"
      provider = "CodeCommit"
      version  = "1"
      output_artifacts = [
      "source_output"]

      configuration = {
        BranchName = aws_codecommit_repository.devsecops.default_branch
        RepositoryName = aws_codecommit_repository.devsecops.repository_name
      }
    }
}

Our next step is to use the CodeBuild provider to build the container and sign it.

stage {
  name = "Build"

  action {
    name     = "Build"
    category = "Build"
    owner    = "AWS"
    provider = "CodeBuild"
    input_artifacts = [
    "source_output"]
    version = "1"

    configuration = {
      ProjectName = "${var.name}-codebuild-BUILD"
    }
  }
}

CodePipeline can have as many stages as needed for your pipeline. For now, we only have the two stages. Let’s see how CodePipeline and CodeBuild work together.

AWS CodeBuild Overview

In our example CodeBuild uses containers for working environments. This is defined in the environment section of the CodeBuild project.

environment {
    compute_type                = "BUILD_GENERAL1_LARGE"
    image                       = "aws/codebuild/amazonlinux2-x86_64-standard:3.0"
    type                        = "LINUX_CONTAINER"
    image_pull_credentials_type = "CODEBUILD"
    privileged_mode             = "true"

    environment_variable {
      name  = "ACCOUNT_ID"
      value = data.aws_caller_identity.current.account_id
      type  = "PLAINTEXT"
    }

    environment_variable {
      name  = "COSIGN_ROLE_NAME"
      value = aws_iam_role.codebuild.name
    }
  }

CodeBuild is set up using a Build spec, which is a yaml file that runs commands defined in stages. The Buildspec yaml has four phases, install, pre_build, build and post_build.

source {
     type      = "CODEPIPELINE"
     buildspec = file("${path.module}/../BUILD-buildspec.yml")
 }

The build spec creates this view in the console, with a list of all the runs for this specific CodeBuild Project.

Before we can use cosign we need an AWS KMS key for signing.

Generate KMS Key

Cosign supports the AWS Key Management System to store the private key for signing containers. As with the other AWS resources, terraform generates the KMS key to be used later in the cosign CodeBuild post_build phase.

WARNING: Make sure to set the key_usage to SIGN_VERIFY, otherwise cosign will fail.

resource "aws_kms_key" "cosign" {
  description             = "Cosign Key"
  deletion_window_in_days = 10
  tags = {
    env = var.name
  }
  key_usage = "SIGN_VERIFY"
  customer_master_key_spec = "RSA_4096"
}

resource "aws_kms_alias" "cosign" {
  name          = "alias/${var.name}"
  target_key_id = aws_kms_key.cosign.key_id
}

In our next section we walk through the build spec which installs cosign, builds, pushes, signs and verifies the container.

Install Cosign

We're going to install Cosign as a CodeBuild provider for a Signing action. This will generate the signed container and store it in the AWS ECR repository. CodeBuild comes with various containers to use and the phases allow us to customize the environment to meet our needs. Below we install the AWS cli, golang 1.17 and Cosign. You could create custom images to speed up build times, AWS has a blog post about it.

NOTE: Most of the commands below are using Make so that they work locally as well as in the AWS CodeBuild. I am a big fan of the Three musketeers pattern.

install:
    commands:
      - yum update -y && yum -y install curl jq python3-pip python3-dev perl-Digest-SHA && pip3 install --upgrade awscli
      - export PATH=$PWD/:$PATH
      - curl -o go1.17.5.linux-amd64.tar.gz https://go.dev/dl/go1.17.5.linux-amd64.tar.gz -vvv -L
      - shasum -a 256 -c <<< 'bd78114b0d441b029c8fe0341f4910370925a4d270a6a590668840675b0c653e  go1.17.5.linux-amd64.tar.gz'
      - rm -rf /usr/local/go && tar -C /usr/local -xzf go1.17.5.linux-amd64.tar.gz
      - export PATH=/usr/local/go/bin:$PATH
      - go version
      - go install github.com/sigstore/cosign/cmd/cosign@latest

In this version we installed cosign via go, but we very well could have downloaded cosign from the GitHub releases.

We can see this phase in action in the above screenshot of CodeBuild Console.

The pre-build stage allows us one final phase before we build the container. As with docker hub, we need credentials to the AWS ECR repository.

pre_build:
    commands:
      - make ecr_auth

Now that we have the credentials to ECR we can build the container.

CODEBUILD_RESOLVED_SOURCE_VERSION is a prebuilt CodeBuild environment variable that will produce the sha of the commit that kicked off the pipeline.

build:
  commands:
    - make docker_build VERSION=$CODEBUILD_RESOLVED_SOURCE_VERSION`

When the container is built we can push it to the ECR registry and then sign and verify it.

CodeBuild Signing Containers

In the post build phase, we push the container to the AWS ECR registry. Then sign the container with the kms key and for good measure verify that it was signed.

post_build: commands: - make docker_push VERSION=$CODEBUILD_RESOLVED_SOURCE_VERSION - make sign VERSION=$CODEBUILD_RESOLVED_SOURCE_VERSION - make verify VERSION=$CODEBUILD_RESOLVED_SOURCE_VERSION

Here are the Makefile targets, so you can see the full cosign commands. Cosign supports using aliases instead of having to gather the KMS id.

sign:
  cosign sign --key awskms:///alias/$(NAME) $(DOCKER_ECR_IMAGE)

key_gen:
  cosign generate-key-pair --kms awskms:///alias/$(NAME) $(DOCKER_ECR_IMAGE)

verify: key_gen
  cosign verify --key cosign.pub $(DOCKER_ECR_IMAGE)

Success!

Below we can see the final phase of the CodeBuild run signing and verifying the container.

Here is the view of the AWS ECR console with the signature and the container images. ECR also has a CVE scanning ability but our signature is just that, a signature and not an actual container.

Summary

In this post we showed you how to install, configure and run sigstore’s Cosign inside an AWS CodePipeline. If you want to review and run through this example yourself all of this is put together as a demo in this accompanying repo. fork it and let us know your experience with Cosign and AWS CodePipeline.

If you are interested in getting involved, or learning more about sigstore, please reach out via slack, email, or join the weekly community call.