ECK can be configured to provide a validating webhook that validates Elastic custom resources (Elasticsearch, Kibana, APM Server, Enterprise Search, Beats, Elastic Agent, Elastic Maps Server, and Logstash) before they are created or updated. Validating webhooks provide immediate feedback if a submitted manifest contains invalid or illegal configuration — which can help you catch errors early and save time that would otherwise be spent on troubleshooting.
Validating webhooks are defined using a
ValidatingWebhookConfiguration object that defines the following:
- Type of resource to validate (Elasticsearch, Kibana and so on)
- Type of actions to validate (create, update, delete)
Connection details to the webhook
- Kubernetes service name and namespace
- Request path
- CA certificate for verifying the server
- Failure policy if the webhook is unavailable (block the operation or continue without validation)
Defaults provided by ECKedit
When using the default
operator.yaml manifest, ECK is installed with a
ValidatingWebhookConfiguration configured as follows:
- Validate all known Elastic custom resources (Elasticsearch, Kibana, APM Server, Enterprise Search, Beats, Elastic Agent, Elastic Maps Server, and Logstash) on create and update.
The operator itself is the webhook server — which is exposed through a service named
The operator generates a certificate for the webhook and stores it in a secret named
elastic-systemnamespace. This certificate is automatically rotated by the operator when it is due to expire.
If you installed ECK without the webhook and want to enable it later on, or if you want to customise the configuration such as providing your own certificates, this section describes the options available to you.
You can customise almost all aspects of the webhook setup by changing the operator configuration.
This must be set to
Path to mount the certificate.
Name of the
Name of the secret containing the certificate for the webhook server.
Port to listen for incoming validation requests.
Using your own certificatesedit
This section describes how you can use your own certificates for the webhook instead of letting the operator manage them automatically. There are a few important things to be aware of when going down this route:
- You have to keep track of the expiry dates and manage the certificate rotation yourself. Expired certificates may stop the webhook from working.
- The secret containing the custom certificate must be available when the operator starts.
You must update the
caBundlefields in the
ValidatingWebhookConfigurationyourself. This must be done at the beginning and whenever the certificate is rotated.
Use a certificate signed by your own CAedit
The certificate must have a Subject Alternative Name (SAN) of the form
elastic-webhook-server.elastic-system.svc). A typical OpenSSL command to generate such a certificate would be as follows:
openssl req -x509 -sha256 -nodes -newkey rsa:4096 -days 365 -subj "/CN=elastic-webhook-server" -addext "subjectAltName=DNS:elastic-webhook-server.elastic-system.svc" -keyout tls.key -out tls.crt
Create a secret in the namespace the operator would be deployed to. This secret must contain the certificate under the
tls.crtkey and the private key under the
kubectl create secret -n elastic-system generic elastic-webhook-server-custom-cert --from-file=tls.crt=tls.crt --from-file=tls.key=tls.key
Encode your CA trust chain as base64 and set it as the value of the
caBundlefields in the
ValidatingWebhookConfiguration. Note that there are multiple
caBundlefields in the webhook configuration.
Install the operator with the following options:
webhook-secretto the name of the secret you have just created (
If you are using the Helm chart installation method, you can install the operator by running this command:
helm install elastic-operator elastic/eck-operator -n elastic-system --create-namespace \ --set=webhook.manageCerts=false \ --set=webhook.certsSecret=elastic-webhook-server-custom-cert \ --set=webhook.caBundle=$(base64 -w 0 ca.crt)
Use a certificate from cert-manageredit
This section describes how to use cert-manager to manage the webhook certificate. It assumes that there is a
Create a new certificate
apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: elastic-webhook-server-cert namespace: elastic-system spec: dnsNames: - elastic-webhook-server.elastic-system.svc issuerRef: kind: ClusterIssuer name: self-signing-issuer secretName: elastic-webhook-server-cert subject: organizations: - example
ValidatingWebhookConfigurationto inject the CA bundle:
apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: annotations: cert-manager.io/inject-ca-from: elastic-system/elastic-webhook-server-cert name: elastic-webhook.k8s.elastic.co webhooks: [...]
Install the operator with the following options:
webhook-secretto the name of the certificate secret (
If you are using the Helm chart installation method, you can install the operator by running the following command:
helm install elastic-operator elastic/eck-operator -n elastic-system --create-namespace \ --set=webhook.manageCerts=false \ --set=webhook.certsSecret=elastic-webhook-server-cert \ --set=webhook.certManagerCert=elastic-webhook-server-cert
Disable the webhookedit
To disable the webhook, set the
enable-webhook operator configuration flag to
false and remove the
kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io elastic-webhook.k8s.elastic.co
You might get errors in your Kubernetes API server logs indicating that it cannot reach the operator service (
elastic-webhook-server). This could be because no operator pods are available to handle request or because a network policy or a firewall rule is preventing the control plane from accessing the service. To help with troubleshooting, you can change the
failurePolicy of the webhook configuration to
Fail. This will cause create or update operations to fail if there is an error contacting the webhook. Usually the error message will contain helpful information about the failure that will allow you to diagnose the root cause.
Resource creation taking too long or timing outedit
Webhooks require network connectivity between the Kubernetes API server and the operator. If the creation of an Elasticsearch resource times out with an error message similar to the following, then the Kubernetes API server might be unable to connect to the webhook to validate the manifest.
Error from server (Timeout): error when creating "elasticsearch.yaml": Timeout: request did not complete within requested timeout 30s
If you get this error, try re-running the command with a higher request timeout as follows:
kubectl --request-timeout=1m apply -f elasticsearch.yaml
As the default
failurePolicy of the webhook is
Ignore, this command should succeed after about 30 seconds. This is an indication that the API server cannot contact the webhook server and has foregone validation when creating the resource.
On GKE private clusters, you may have to add a firewall rule allowing access to port 9443 from the API server so that it can contact the webhook. Check the GKE documentation on firewall rules and the Kubernetes issue for more details.
It is possible that a network policy is blocking any incoming requests to the webhook server. Consult your system administrator to determine whether that is the case, and create an appropriate policy to allow communication between the Kubernetes API server and the webhook server. For example, the following network policy simply opens up the webhook port to the world:
kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: allow-webhook-access-from-any namespace: elastic-system spec: podSelector: matchLabels: control-plane: elastic-operator ingress: - from:  ports: - port: 9443
If you want to restrict the webhook access only to the Kubernetes API server, you must know the IP address of the API server, that you can obtain through this command:
kubectl cluster-info | grep master
Assuming that the API server IP address is
10.1.0.1, the following policy restricts webhook access to just the API server.
kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: allow-webhook-access-from-apiserver namespace: elastic-system spec: podSelector: matchLabels: control-plane: elastic-operator ingress: - from: - ipBlock: cidr: 10.1.0.1/32 ports: - port: 9443
Updates failing due to validation errorsedit
If your attempts to update a resource fail with an error message similar to the following, you can force the webhook to ignore it by removing the
kubectl.kubernetes.io/last-applied-configuration annotation from your resource.
admission webhook "elastic-es-validation-v1.k8s.elastic.co" denied the request: Elasticsearch.elasticsearch.k8s.elastic.co "quickstart" is invalid: some-misspelled-field: Invalid value: "some-misspelled-field": some-misspelled-field field found in the kubectl.kubernetes.io/last-applied-configuration annotation is unknown