Signed, sealed, delivered: ECK certificate management with Vault and cert-manager

Get hands-on with Elasticsearch: Dive into our sample notebooks in the Elasticsearch Labs repo, start a free cloud trial, or try Elastic on your local machine now.

You can run Elastic Cloud on Kubernetes (ECK) on your own enterprise public key infrastructure (PKI) without your certificate authority’s (CA's) private key ever entering the Kubernetes cluster. By default, ECK generates self-signed certificates for Elasticsearch and Kibana. This post hands ECK certificate management for Elasticsearch to HashiCorp Vault, which holds the intermediate CA and signs every certificate, and to cert-manager, which requests and renews them. In the example, HTTP certificates last seven days and renew an hour before they expire. Transport certificates come from the cert-manager container storage interface (CSI) driver, which issues one directly into each Elasticsearch pod as it starts, so new nodes get certificates automatically and their private keys are never stored as Kubernetes Secrets.

ECK's defaults secure every cluster with Transport Layer Security (TLS) and rotate their own certificates, which works well until your security policy says that every certificate must come from your enterprise PKI. At that point, you need central control over issuance and trust, and you still don't want to renew certificates by hand. Vault and cert-manager give you both.

Self-signed certificates that ECK generates by default

The ECK operator generates self-signed certificates for various components of the Elastic Stack. Below is a high-level overview of the different types of self-signed certificates generated by ECK for Elasticsearch and Kibana:

ECK Certificate Management
├── Elasticsearch
│   ├── HTTP
│   │   ├── HTTP CA
│   │   └── HTTP Server Certificate
│   │
│   └── Transport
│       ├── Transport CA
│       └── Node Transport Certificates
│
└── Kibana
    └── HTTP
        ├── HTTP CA
        └── HTTP Server Certificate

Where should your CA private key live?

Using custom certificates with ECK introduces a tradeoff between the risk of exposing a CA private key and automating certificate lifecycle management.

Bring your own CA and private key to ECK

The ECK operator automates certificate issuance and renewal when provided with the CA certificate and private key through a Kubernetes Secret. ECK then manages the certificate lifecycle across the Elastic Stack, including both HTTP and transport certificates for Elasticsearch. However, storing the CA private key within the Kubernetes cluster is considered a security risk and may not align with enterprise security and PKI policies.

Use manually issued certificates with ECK

ECK can consume externally issued certificates provided through Kubernetes Secrets, keeping the CA private key outside the Kubernetes cluster. However, certificate issuance and renewal must be managed separately, as should certificate rotation. This creates an operational bottleneck, particularly for transport certificates, which need to be provisioned dynamically as Elasticsearch pods scale up or down.

Use Vault PKI with cert-manager

Cloud-native tooling removes the tradeoff between bringing your own CA and key and using manually issued certificates. HashiCorp Vault and cert-manager move certificate lifecycle management on Kubernetes out of  the ECK operator, while keeping the enterprise CA private key secure.

• HashiCorp Vault is the external PKI system, securely managing the CA private keys and issuing certificates signed by the enterprise CA. Certificate signing operations stay secure without ever exposing the CA private key as a Kubernetes Secret.

• cert-manager can automate the certificate lifecycle within Kubernetes. It can request certificates from Vault and monitor their validity. It can also automatically renew them before expiration. Once issued, cert-manager can create and update the corresponding Kubernetes Secrets which, in turn, are consumed by Elastic components.

Using Vault PKI with cert-manager gives each component a single responsibility: Vault manages the trust and signing authority, and cert-manager manages the certificate lifecycle. ECK components consume the resulting certificates. 

Vault and cert-manager let organizations keep their existing enterprise PKI controls while reducing the operational overhead associated with certificate issuance, renewal, and rotation.

The table below compares the three approaches:

ECK certificate management architecture

The following diagram illustrates the proposed architecture for the solution:

Prerequisites and tested versions

This blog assumes that the user has a basic understanding of the Vault and cert-manager setup. In our post, we focus only on the configuration required to implement this setup.

Refer to the official Vault and cert-manager documentation for installation instructions related to Vault, cert-manager, and csi-driver.

Test versions

We used the following software versions for the demonstration in this blog. For compatibility considerations or configuration differences with other versions, refer to the respective official documentation.

• Elastic Stack 9.5.3

• cert-manager 1.21.2

• HashiCorp Vault 2.0.3 (Community Edition)

HashiCorp Vault as your enterprise PKI

On the Vault server, the PKI secrets engine needs to be enabled to securely store CA certificates and its private key that’s used to sign leaf certificates.

As a best practice, the Root CA private key remains outside Vault. The Root CA is used to sign the Intermediate CA certificate signing request (CSR), after which the Intermediate CA certificate and private key are stored securely in Vault. Vault then uses this Intermediate CA and key to issue and sign leaf certificates.

In addition to the PKI secrets engine, the Kubernetes authentication method is also enabled so Vault can authenticate requests originating from Kubernetes. A dedicated Vault role is configured with narrowly scoped permissions, ensuring that it issues certificates only for requests from the specified Kubernetes ServiceAccount and namespace.

At a high level, the following tree structure illustrates the Vault components required for this setup:

HashiCorp Vault
│
├── PKI
│   └── PKI Engine
│       ├── CA / Intermediate CA / Intermediate Key
│       └── Sign Role
│           └── Certificate issuance policy
│
└── Authentication
    └── Kubernetes Auth Method
        ├── Kubernetes API Server
        └── Kubernetes Auth Role
            ├── Bound ServiceAccount
            └── Bound Namespace

Configure Vault PKI and Vault Kubernetes auth

1. On a Linux host that has access to the Vault command line interface (CLI), generate a root CA certificate and key:

openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes -keyout eck-root-ca.key -out eck-root-ca.crt -subj "/C=SG/ST=SG/L=SG/O=Elastic Demo/OU=ECK/CN=ECK Root"

NOTE: This is used for demonstration purposes only. In a production implementation, skip the above step and use your organization's existing enterprise CA and key instead.

2. Enable the PKI secrets engine at the eck/esintca path, and configure a maximum certificate lease duration of one year. This PKI engine is used to store the intermediate CA and issue certificates for the Elastic Stack components:

vault secrets enable -path=eck/esintca pki && \
vault secrets tune -max-lease-ttl=8760h eck/esintca

3. Generate an internal Intermediate CA key and CSR using the eck/esintca PKI engine. The generated response includes the CSR and is stored in eck_esint.json for use in the subsequent signing step:

vault write -format=json eck/esintca/intermediate/generate/internal common_name="ES Intermediate" organization="Elastic Demo" ou="ECK" country="SG" province="SG" locality="SG" key_type="rsa" key_bits=2048 > eck_esint.json

4. Extract the Intermediate CA CSR from the Vault response, and save it as eck_esint.csr. This CSR is submitted to the Root CA for signing.

jq -r 
'.data.csr' 
eck_esint.json > eck_esint.csr

5. Create an OpenSSL extensions configuration file defining the Intermediate CA attributes. The configuration marks the certificate as a CA and limits the certificate path length to zero. It also enables the required key usage and certificate identification extensions. This configuration is used when signing the Intermediate CA CSR with the Root CA.

cat > eck-ext.cnf <<'EOF'
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, keyCertSign, cRLSign
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
EOF

6. Sign the Intermediate CA CSR using the Root CA certificate and private key to generate the Intermediate CA certificate. The certificate is valid for 365 days and uses SHA-256, with the CA extensions defined in eck-ext.cnf. The resulting eck_esint.crt is used by Vault to issue and sign leaf certificates.

openssl x509 -req -in eck_esint.csr -CA eck-root-ca.crt -CAkey eck-root-ca.key -CAcreateserial -out eck_esint.crt -days 365 -sha256 -extfile eck-ext.cnf

7. Combine the Intermediate CA certificate with the Root CA certificate to create the complete CA certificate chain. Vault uses this chain when issuing certificates so that clients can trust the chain back to the Root CA.

{ cat eck_esint.crt; printf '\n'; cat eck-root-ca.crt; } > eck_es_cachain.crt

8. Upload the signed Intermediate CA certificate chain to the Vault PKI engine. This completes the Intermediate CA configuration and enables Vault to use it for issuing certificates signed by the Elasticsearch Intermediate CA.

vault write eck/esintca/intermediate/set-signed certificate=@eck_es_cachain.crt

9. Configure the Vault PKI engine with the URLs used to retrieve the issuing CA certificate and Certificate Revocation List (CRL). Replace <replace-with-vault-server-endpoint> with the Vault server endpoint specific to your environment. Clients use these URLs to retrieve the CA certificate and to check certificate revocation status when required.

vault write eck/esintca/config/urls issuing_certificates="https://<replace-with-vault-server-endpoint>:8200/v1/eck/esintca/ca" && \

vault write eck/esintca/config/urls crl_distribution_points="https://<replace-with-vault-server-endpoint>:8200/v1/eck/esintca/crl"

10. From the Linux host that has access to Kubernetes cluster, retrieve the Kubernetes API server endpoint from the current kubectl context. This endpoint is used to configure Vault's Kubernetes authentication and to allow Vault to communicate with the Kubernetes API server for ServiceAccount token validation.

export KUBE_API_SERVER=$(kubectl config view --minify \
  -o jsonpath='{.clusters[0].cluster.server}{"\n"}')

11. Extract the Kubernetes API server's CA certificate from the current kubectl context, and decode it into kubeapi-ca.crt. Vault uses this CA certificate to establish a trusted TLS connection with the Kubernetes API server.

kubectl config view --raw --minify \
  -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' \
  | base64 -d > kubeapi-ca.crt

NOTE: If the same Linux host has access to both the Kubernetes cluster and the Vault server, run the following commands from that host. Otherwise, copy the Kubernetes API server endpoint, audience url, and CA certificate to the host from which the Vault server is accessible, and run the commands there.

12. Enable the Kubernetes authentication method at the eck path. Vault then authenticates requests from Kubernetes workloads using their ServiceAccount tokens.

vault auth enable -path=eck kubernetes

13. Configure Vault with the Kubernetes API server endpoint and its CA certificate. Vault uses the CA certificate to establish and verify a trusted TLS connection to the Kubernetes API server.

vault write auth/eck/config kubernetes_host="$KUBE_API_SERVER" kubernetes_ca_cert=@kubeapi-ca.crt

14. Create a Vault PKI role that defines the domains and certificate parameters allowed for Elasticsearch certificates. The role permits the specified Kubernetes service domains and subdomains, and it accepts the common name (CN) and subject alternative names (SANs) from the CSR. It also limits issued certificates to a maximum time to live (TTL) of 168 hours (seven days).

vault write eck/esintca/roles/elasticsearch allowed_domains="elastic-cluster.svc,elastic-cluster.svc.cluster.local,elasticsearch-es-http" allow_subdomains=true allow_bare_domains=true use_csr_common_name=true use_csr_sans=true max_ttl="168h"

Note: In this example, the Elasticsearch cluster is named elasticsearch and is deployed in the elastic-cluster namespace. Therefore, the Elasticsearch HTTP service is referenced as elasticsearch-es-http and the domain as elastic-cluster.svc. Replace the cluster name, namespace, and corresponding service name with the values applicable to your environment.

15. Create a Vault policy that grants permission to use the elasticsearch PKI signing role. The policy provides only the required update capability on the certificate signing endpoint, following a least-privilege approach.

cat > eck-es-policy.hcl <<'EOF'
path "eck/esintca/sign/elasticsearch" {
  capabilities = ["update"]
}
EOF

16. Create the eck-es-policy policy in Vault using the previously defined policy file. This policy is associated with the Kubernetes authentication role and grants the permissions required to request Elasticsearch certificates.

vault policy write eck-es-policy eck-es-policy.hcl

17. Create the eck-es-role role, and bind it to the cert-manager-vault-access ServiceAccount in the elastic-cluster namespace. The role uses the specified Kubernetes token audience and attaches the eck-es-policy, so the ServiceAccount can authenticate to Vault and request Elasticsearch certificates. The authentication token is configured with a 10-minute TTL.

vault write auth/eck/role/eck-es-role bound_service_account_names=cert-manager-vault-access bound_service_account_namespaces=elastic-cluster audience=
"vault://elastic-cluster/eck-es-issuer" 
policies=eck-es-policy ttl=
10m

Note: In the Kubernetes cluster, ensure that Vault authentication requests originate from the elastic-cluster namespace and use the cert-manager-vault-access ServiceAccount, along with its associated token.

Automate certificate renewal with cert-manager

cert-manager is used to manage the lifecycle of certificates required by the Elastic Stack. It monitors Kubernetes Certificate resources and requests certificates from the configured Issuer, which, in this solution, is backed by Vault.

cert-manager authenticates to Vault using a dedicated Kubernetes ServiceAccount. Vault then validates the identity of the requesting workload and authorizes certificate issuance based on the configured Vault role and policy.

Once Vault issues the certificate, cert-manager creates and maintains the corresponding Kubernetes Secret containing the certificate and private key. cert-manager also monitors certificate validity and automatically manages renewal and rotation before certificates expire.

Issue transport certificates with the cert-manager CSI driver

For Elasticsearch transport certificates, the cert-manager CSI driver provides certificates directly to the Elasticsearch pods. The CSI driver requests certificates and mounts them directly into pods through the Kubernetes CSI interface, so the certificate and private key never need to be persisted as a Kubernetes Secret.

With the CSI driver, cert-manager still manages the certificate lifecycle and obtains certificates from the Vault-backed Issuer. The CSI driver handles the delivery of these certificates to the Elasticsearch pods, where they can be used for secure transport communication between Elasticsearch nodes.

Create the Vault Issuer and HTTP certificate

1. On the Kubernetes cluster, create the elastic-cluster namespace and a dedicated cert-manager-vault-access ServiceAccount within it. This ServiceAccount is used to obtain the Kubernetes token that cert-manager presents to Vault for authentication.

kubectl create ns elastic-cluster

kubectl create sa cert-manager-vault-access -n elastic-cluster

2. Create the required role-based access control (RBAC) permissions in the elastic-cluster namespace to support Vault authentication. The Role and RoleBinding allow the cert-manager ServiceAccount to request a token for the dedicated cert-manager-vault-access ServiceAccount. The ClusterRoleBinding lets Vault use this ServiceAccount to validate Kubernetes ServiceAccount tokens through the Kubernetes API, so Vault can verify the identity of the requesting workload.

kubectl apply -f - <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cert-manager-vault-token-request
  namespace: elastic-cluster
rules:
  - apiGroups: [""]
    resources:
      - serviceaccounts/token
    resourceNames:
      - cert-manager-vault-access
    verbs:
      - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cert-manager-vault-token-request
  namespace: elastic-cluster
subjects:
  - kind: ServiceAccount
    name: cert-manager
    namespace: cert-manager
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cert-manager-vault-token-request
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cert-manager-vault-access-token-review
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: cert-manager-vault-access
    namespace: elastic-cluster
EOF

3. Generate a token for the default ServiceAccount, and decode its JSON Web Tokens (JWT) payload to inspect the token claims. This is used to identify the token audience (aud) when configuring Vault's Kubernetes authentication.

export 
aud=$(kubectl create token default | cut -d. -f2 | base64 -d 
2
>/dev/null | sed -n 
's/.*"aud":\["\([^"]*\)".*/\1/p'
)

4. Create an Issuer object that connects cert-manager to the Vault PKI signing endpoint. Configure the Vault server endpoint and PKI signing path, along with the Vault server CA certificate, so cert-manager can establish a trusted connection to Vault. The Kubernetes authentication configuration specifies the eck-es-role Vault role and the cert-manager-vault-access ServiceAccount. cert-manager uses this identity and audience to authenticate to Vault and to request Elasticsearch certificates. In the YAML below, replace the Vault server endpoint and CABundle, as well as the audience generated in the previous step, and save it in the issuer.yaml file:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: eck-es-issuer
  namespace: elastic-cluster
spec:
  vault:
    server: https://<replace-with-vault-server-endpoint>:8200
    path: eck/esintca/sign/elasticsearch
    caBundle: <ca-cert-of-vault-endpoint-in-base64>
    auth:
      kubernetes:
        role: eck-es-role
        mountPath: /v1/auth/eck
        serviceAccountRef:
          name: cert-manager-vault-access
          audiences:
            - $aud

Apply the manifest file to create the object:

kubectl create -f issuer.yaml

5. Create a cert-manager Certificate resource in the elastic-cluster namespace to request the Elasticsearch HTTP certificate from the eck-es-issuer Issuer. The certificate is configured with the required domain name system (DNS) names for Elasticsearch service access and a seven-day validity period, in addition to renewal 60 minutes before expiry. cert-manager stores the generated certificate and private key in the eck-es-http Kubernetes Secret, using an RSA 2048-bit private key encoded in Public-Key Cryptography Standards #1 (PKCS#1) format.

kubectl apply -f - <<'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: eck-http-certificate
  namespace: elastic-cluster
spec:
  duration: '168h'
  renewBefore: '60m'
  isCA: false
  secretName: eck-es-http
  issuerRef:
    name: eck-es-issuer
    kind: Issuer
  commonName: elasticsearch-es-http.elastic-cluster.svc
  dnsNames:
    - elasticsearch-es-http
    - elasticsearch-es-http.elastic-cluster.svc
    - elasticsearch-es-http.elastic-cluster.svc.cluster.local
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
EOF

6. Verify that cert-manager has successfully processed the Certificate resource and created the corresponding Kubernetes Secret. The Secret should contain the certificate and private key generated for the Elasticsearch HTTP endpoint. The Certificate should show a READY status of True, indicating that the certificate was successfully issued and the Secret was created.

kubectl get certificate eck-http-certificate -n elastic-cluster && \kubectl get secret eck-es-http -n elastic-cluster

Make your custom CA available to ECK

Vault and cert-manager already provide the ca.crt file needed for transport trust between Elasticsearch nodes, mounted through the CSI driver. However, in scenarios where the CA changes frequently or remote cluster setup requires the CA to be trusted across clusters, the custom CA also needs to be made available as a ConfigMap in the elastic-cluster namespace so that the ECK operator itself is aware of it.

There are two options for providing this ConfigMap.

Sync the CA automatically with trust-manager

trust-manager is a Kubernetes operator that keeps trust bundles up to date automatically. It adds a Bundle custom resource that reads the CA from a source, such as a Kubernetes Secret, and publishes it into a ConfigMap, keeping it in sync whenever the source changes.

Install trust-manager with its source secret namespace set to elastic-cluster so that it can read the eck-es-http Secret used in the Bundle below. Refer to its official documentation for more information.

kubectl apply -f - <<'EOF'
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: custom-ca-trust
spec:
  sources:
  - secret:
      name: "eck-es-http"
      key: "ca.crt"
  target:
    configMap:
      key: "ca.crt"
    namespaceSelector:
      matchLabels: 
        kubernetes.io/metadata.name: elastic-cluster
EOF

Create the CA ConfigMap manually

The eck-es-http secret created earlier by cert-manager already contains the CA certificate under its ca.crt key. You can extract it directly and use it to create the ConfigMap.

kubectl get secret eck-es-http -n elastic-cluster -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt && \
kubectl create configmap custom-ca-trust --from-file=ca.crt=ca.crt -n elastic-cluster

Deploy Elasticsearch with enterprise PKI certificates

Deploy the Elasticsearch cluster using the following ECK configuration. The cluster is configured to use the certificates provisioned through the previously configured cert-manager and Vault integration rather than relying on ECK's default self-signed certificate management. The HTTP certificate is provided through the Kubernetes Secret created by cert-manager, while the Elasticsearch transport certificates are delivered through the cert-manager CSI driver.

Because the transport certificates arrive through the CSI driver as mounted files rather than as a Kubernetes Secret, the configuration below points xpack.security.transport.ssl.key and xpack.security.transport.ssl.certificate directly at those file paths:

kubectl apply -f - <<'EOF'
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: elastic-cluster
spec:
  version: 9.5.3
  nodeSets:
    - name: default
      count: 3
      config:
        xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/cert-manager-certs/tls.key
        xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/cert-manager-certs/tls.crt
      podTemplate:
        spec:
          containers:
          - name: elasticsearch
            volumeMounts:
            - name: transport-certs
              mountPath: /usr/share/elasticsearch/config/cert-manager-certs
          volumes:
          - name: transport-certs
            csi:
              driver: csi.cert-manager.io
              readOnly: true
              volumeAttributes:
                csi.cert-manager.io/issuer-name: eck-es-issuer
                csi.cert-manager.io/issuer-kind: Issuer
                csi.cert-manager.io/common-name: "${POD_NAME}.${POD_NAMESPACE}.svc.cluster.local"
                csi.cert-manager.io/dns-names: "${POD_NAME}.${POD_NAMESPACE}.svc.cluster.local"
  transport:
    tls:
      certificateAuthorities:
        configMapName: custom-ca-trust
      selfSignedCertificates:
        disabled: true
  http:
    tls:
      certificate:
        secretName: eck-es-http
EOF

Verify that Elasticsearch is using your enterprise PKI certificates

Verify that Elasticsearch is using the custom certificate issued through the Vault-backed cert-manager configuration. First, retrieve the certificate from the Kubernetes Secret created by cert-manager, and then inspect its subject and issuer, along with its validity period.

The output should show the expected Elasticsearch service name in the certificate's CN/SANs, the Vault-managed Intermediate CA as the issuer, and the certificate's notBefore and notAfter values.

kubectl get secret eck-es-http -n elastic-cluster -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

Verify that Elasticsearch is presenting the certificate issued through the Vault-backed cert-manager configuration. The output should show the expected Elasticsearch service name in the certificate details and the Vault-managed Intermediate CA as the issuer. It should also show the certificate's validity period.

kubectl exec -n elastic-cluster elasticsearch-es-default-0 -- curl -v --cacert /usr/share/elasticsearch/config/http-certs/ca.crt https://elasticsearch-es-http.elastic-cluster.svc:9200 2>&1 | grep -E 'subject:|issuer:|start date:|expire date:|subjectAltName'

Use the following command to verify the health status of the Elasticsearch cluster:

kubectl -n elastic-cluster get elasticsearch

Extend ECK certificate management across the Elastic Stack

The approach we described in this blog post can be summarized in the following diagram:

We focused here on configuring custom certificates for Elasticsearch. The same approach can be extended to other Elastic Stack components, such as Kibana, by using HashiCorp Vault and cert-manager to manage their certificate lifecycle.

Replacing ECK's default self-signed certificates with certificates backed by an enterprise PKI can be implemented using cloud-native tooling, such as Vault and cert-manager. The configuration we’ve demonstrated here provides a practical starting point that can be adapted to different enterprise PKI environments and certificate requirements across the entire ECK stack.

Try the approach in your own environment, starting with a non-production ECK cluster, and explore how it can fit into your organization's existing certificate management and security practices.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.