Elastic Stack container images signed with Sigstore!

By

Maxime Greau

illustration-stack-modernize-solutions-1689x980.png

Software supply chain attacks keep increasing. That's why this topic is a top priority for security leaders.

In that regard, this blog post highlights a new capability of signing the Elastic Stack container images with Sigstore to:

  • Secure the Elastic Software Supply Chain workflow
  • Provide an easy and standard way for Elastic users to verify the provenance of the Elastic container images before deploying them to any infrastructure and therefore protect against supply chain attacks
  • Meet regulatory and compliance requirements

What is Sigstore?

Sigstore is an OpenSSF project, supported by Chainguard, Red Hat, and Google among others, that provides a new standard for signing, verifying, and protecting software easily, leveraging features like keyless signing workflow. Elastic images are signed with cosign, which is part of the Sigstore project. Cosign supports container signing, verification, and storage in an OCI registry.

Which Elastic Stack versions are signed with Sigstore?

Elastic started publishing ELK container images back in 2016 with 5.0. In May 2023, 8.8.0 introduced the first ever signed Elastic Stack container images.

Making sure an image has been signed by Elastic is as simple as installing the cosign application and executing the following command:

$ cosign verify --key https://artifacts.elastic.co/cosign.pub \
                  docker.elastic.co/elasticsearch/elasticsearch:8.8.1

The command prints the check results and the signature payload in JSON format:

Verification for docker.elastic.co/elasticsearch/elasticsearch:8.8.1 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The signatures were verified against the specified public key
[
  {
    "critical": {
      "identity": {
        "docker-reference": "docker.elastic.co/elasticsearch/elasticsearch"
      },
      "image": {
        "docker-manifest-digest": "sha256:27cb808b1029ac75718a12ac16f2c09b0cda6469146b6039fd3573fc2f0711d3"
      },
      "type": "cosign container image signature"
    },
    "optional": {
      "Bundle": {
        "SignedEntryTimestamp": "MEUCIQDSDY3XrFURA5DO5fJ36WZfKf1ejaPlASgLn6tMXEHlDwIgKQPXXgNOasuXOSRRjeNdl0L028n/Yx3yMETYWNxthzg=",
        "Payload": {
          "body": "eyJhcGlWZXJzaW9uIjoiMC4wLjEiLCJraW5kIjoiaGFzaGVkcmVrb3JkIiwic3BlYyI6eyJkYXRhIjp7Imhhc2giOnsiYWxnb3JpdGhtIjoic2hhMjU2IiwidmFsdWUiOiIwNWNhZDk4MmYxMDQ3OTk0OGY3Zjk5NDQyNGEwNWQ5ZDZkZDM5ZDAyZWJmMzNjY2QzMTVlNDUwNmJkOGE4NzY2In19LCJzaWduYXR1cmUiOnsiY29udGVudCI6Ik1FVUNJRkVWZGFJRWcyR1RVQ0l3RkhYeHdxd0kyZGtlazZMbjFXTEFvcHowM0hQMEFpRUE2ZnpDaHpuLy96cGZqYUtCSG1adkgvREhuZzRHMVlKMGltbUNZL20zUWFvPSIsInB1YmxpY0tleSI6eyJjb250ZW50IjoiTFMwdExTMUNSVWRKVGlCUVZVSk1TVU1nUzBWWkxTMHRMUzBLVFVacmQwVjNXVWhMYjFwSmVtb3dRMEZSV1VsTGIxcEplbW93UkVGUlkwUlJaMEZGY1ZaMlRtUlJkR1JrZEdWdGRtWmpWV1V5VGpCbloxZ3ZjSFJxYVFwRlZYRjRlakp3UkZVM1ZWYzFiVE53WkcxSU1UTnJUVXR3ZURselJqUjJWVFZLVDJVM1ZYSXJSazVJVERkaFlXaE1hbWRIWXpBNGRXUkJQVDBLTFMwdExTMUZUa1FnVUZWQ1RFbERJRXRGV1MwdExTMHRDZz09In19fX0=",
          "integratedTime": 1686211821,
          "logIndex": 23111241,
          "logID": "c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d"
        }
      },
      "tag": "8.8.1"
    }
  }
]

Note that we do set the tag attribute with the release version when signing the images (for example 8.8.1) in order to facilitate the verification command.

Verifying signature using the image digest

It’s recommended as a best practice to pull and reference a container image via its digest value, which is technically immutable, contrary to the image tag (even though we do treat tags as immutable internally). 

Using the crane tool to interact with remote registries and images, it’s easy to get the digest value of any image without pulling it:

$ crane digest docker.elastic.co/elasticsearch/elasticsearch:8.8.1
sha256:27cb808b1029ac75718a12ac16f2c09b0cda6469146b6039fd3573fc2f0711d3

Then, verifying the signature via the digest is simple, as follows:

$ cosign verify --key https://artifacts.elastic.co/cosign.pub \
                  docker.elastic.co/elasticsearch/elasticsearch@sha256:27cb808b1029ac75718a12ac16f2c09b0cda6469146b6039fd3573fc2f0711d3

Elastic container registry support

Our Elastic container registry is available at docker.elastic.co; it is an OCI-Compliant registry and therefore is compatible with cosign. But we had to update our container-library UI to make it compatible with the cosign signatures format. As a reminder, cosign takes the SHA256 checksum of an image and pushes the signature as a tag to the registry in the following format: sha256-<sha256_checksum_of_image>.<sig>. These tags are hidden from the UI.

Checkout this blog post to learn more about the upcoming OCI v1.1 specification and the cosign support that will solve this .sig tag workaround.

Third-party container registries support

The Elastic Stack images are also available from the AWS ECR registry and the Docker Hub Elastic repository. We leverage the cosign copy command to copy our multi-architecture images and signatures from the Elastic registry to these third-party registries with one simple command while also ensuring the digests are preserved. 

Using the crane tool, it’s easy to compare the digests of these images:

$ crane digest docker.elastic.co/elasticsearch/elasticsearch:8.8.1
sha256:27cb808b1029ac75718a12ac16f2c09b0cda6469146b6039fd3573fc2f0711d3

$ crane digest docker.io/elastic/elasticsearch:8.8.1
sha256:27cb808b1029ac75718a12ac16f2c09b0cda6469146b6039fd3573fc2f0711d3

$ crane digest public.ecr.aws/elastic/elasticsearch:8.8.1
sha256:27cb808b1029ac75718a12ac16f2c09b0cda6469146b6039fd3573fc2f0711d3

Then, the cosign verify command produces the same output for both AWS ECR and Docker Hub hosted images:

$ cosign verify --key https://artifacts.elastic.co/cosign.pub \
                  public.ecr.aws/elastic/elasticsearch@sha256:27cb808b1029ac75718a12ac16f2c09b0cda6469146b6039fd3573fc2f0711d3 | jq .


Verification for public.ecr.aws/elastic/elasticsearch@sha256:27cb808b1029ac75718a12ac16f2c09b0cda6469146b6039fd3573fc2f0711d3 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The signatures were verified against the specified public key
[
  {
    "critical": {
      "identity": {
        "docker-reference": "docker.elastic.co/elasticsearch/elasticsearch"
      },
      "image": {
        "docker-manifest-digest": "sha256:27cb808b1029ac75718a12ac16f2c09b0cda6469146b6039fd3573fc2f0711d3"
      },
      "type": "cosign container image signature"
    },
…
]
$ cosign verify --key https://artifacts.elastic.co/cosign.pub \
                  docker.io/elastic/elasticsearch:8.8.1@sha256:27cb808b1029ac75718a12ac16f2c09b0cda6469146b6039fd3573fc2f0711d3 | jq .


Verification for index.docker.io/elastic/elasticsearch@sha256:27cb808b1029ac75718a12ac16f2c09b0cda6469146b6039fd3573fc2f0711d3 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The signatures were verified against the specified public key
[
  {
    "critical": {
      "identity": {
        "docker-reference": "docker.elastic.co/elasticsearch/elasticsearch"
      },
      "image": {
        "docker-manifest-digest": "sha256:27cb808b1029ac75718a12ac16f2c09b0cda6469146b6039fd3573fc2f0711d3"
      },
      "type": "cosign container image signature"
    },
…
]

Please note that even if these container registries, as OCI-compliant registries, do support cosign, their UI has not been updated at the time of writing this post. Therefore the specific .sig tags that contain the payload of the signatures show up like any other versions tags.

Docker “Official Images” (docker-library) not supported

In addition to the Docker Hub repositories, Docker, Inc. hosts the Docker “Official Images”, which have the particularity to be built on their own infrastructure. For the ELK images, Elastic provides the docker build context files in a public repository as an input to the Docker, Inc. build workflow. 

Therefore, unlike the images published to our own Elastic container registry and copied to the AWS ECR registry and the Docker Hub Elastic repositories, these container images are not signed with the Elastic key. All the ELK images are impacted: Elasticsearch, Kibana, and Logstash. A GitHub issue is opened to request for supporting Sigstore cosign for Docker “Official Images” built from the docker-library repository.

Here is an example of the cosign output command when verifying a Docker “Official Image”:

$ cosign verify --key https://artifacts.elastic.co/cosign.pub elasticsearch:8.8.1
Error: no signatures found for image

That makes sense as this Elasticsearch container image is not the exact same as the one built on the Elastic infrastructure. You can check by comparing the digest value below with the one from the supported registries:

$ crane digest elasticsearch:8.8.1
sha256:e31753d052509353f99be86892bf5c65a070805aadbb295d1b3128163061ea68

We encourage users and customers to pull the Elastic official images from docker.elastic.co or from third-party registries with identical digest values.

Next step: Keyless signing

Updating our release workflow to sign the Elastic Stack images with long-lived signing keys stored is only a first step. In the future, our goal is to leverage the ephemeral keys signing feature (keyless signing) to further improve the security and traceability of this process.

Sign up for Elastic Cloud free trial

Spin up a fully loaded deployment on the cloud provider you choose. As the company behind Elasticsearch, we bring our features and support to your Elastic clusters in the cloud.

Start free trial