Elastic Cloud on Kubernetes (ECK) is an easy way to get the Elastic Stack up and running on top of Kubernetes. That’s because ECK automates the deployment, provisioning, management, and setup of Elasticsearch, Kibana, Beats, and more.
As logging and metric data — or time series data — has a predictable lifespan, you can use hot, warm, and cold architecture to easily manage your data over time as it ages and becomes less relevant.
In this guide, we’ll explain how to set up SAML with auth0 as an identity provider (IdP) as well as how to configure your deployment for hot-warm-cold architecture with ECK. We’ll use production features such as:
- Dedicated nodes
- Zone awareness
- Pod and node affinity
- Dedicated storage class
For this example, we’ll deploy ECK to have a centralized way to authenticate into Kibana using auth0.
We’ll deploy the following resources:
- ECK Operator
- Elasticsearch role job
- Ingress controller
- ConfigMap SAML metadata
To set up this environment, we’ll use GKE (Google Kubernetes Engine) with the following configurations:
Three Kubernetes nodes pools
- 1 hot node pool with 6 Kubernetes instances running spread across 3 availability zones
- 1 warm node pool with Kubernetes instances running spread across 3 availability zones
- 1 cold node pool with Kubernetes instances running spread across 3 availability zones
- Hot nodes: c2-standard-4 (4 vCPUs, 16GB memory)
- Warm nodes: e2-standard-2 (2 vCPUs, 8GB memory)
- Cold nodes: e2-standard-2 (2 vCPUs, 8GB memory)
We’ll have nine Elasticsearch instances total (with individual tiers for hot, warm, and cold data, as well as dedicated masters), plus two Kibana instances running on GKE zones
For each GKE node pool, we’ll use a specific Kubernetes label to attach the Elasticsearch instance into the right hardware configuration. The label name is called
type, and the values are
cold. You can change and add the label later by using the command line.
First, let’s dive into the manifest explanation, then how you can get set up.
The Elasticsearch manifest file eck-saml-hot-warm-cold.yml contains the Elasticsearch settings. Let’s examine the relevant parts of the manifest one by one:
We have one node called
hot-zone-b deployed at zone
europe-west1-b. Then, we define two routing awareness attributes:
k8s_node_name: This attribute makes sure that Elasticsearch allocates primary and replica shards to pods running on different Kubernetes nodes and never to pods that are scheduled onto a single Kubernetes node.
- zone: This attribute uses the Kubernetes label
domain.beta.kubernetes.io/zone. If Elasticsearch knows which nodes are on the same zone, it can distribute the primary shard and its replica shards to minimise the risk of losing all shard copies in the event of a failure.
nodeSets: - name: hot-zone-b count: 1 config: node.attr.zone: europe-west1-b cluster.routing.allocation.awareness.attributes: k8s_node_name,zone node.roles: [ data_hot, data_content ] node.store.allow_mmap: false
We explicitly say that this node will have a
node.store.allow_mmap : false will prevent the virtual address space (on Linux distribution) from running into errors or exceptions because this default configuration is too low. You can find more information in our documentation.
This is the SAML configuration. You should follow this guide to configure auth0 as idp and then get the metadata file and the idp.* settings.
attributes.principal: This defines which SAML attribute is going to be mapped to the principal (username) of the authenticated user in Kibana.
idp.entity_id: This is the SAML EntityID of your Identity Provider (which you can get from SAML metadata file).
idp.metadata.path: This is the file path or the HTTPs URL where your Identity Provider metadata is available. In this example, we’ll use a file to demonstrate how to mount a volume.
sp.acs: This is the Assertion Consumer Service URL where Kibana is listening for incoming SAML messages.
sp.entity_id: This is the SAML EntityID of our Service Provider.
sp.logout: This is the
SingleLogoutendpoint where the Service Provider is listening for incoming SAML
xpack: security: authc: realms: saml: saml1: attributes.principal: nameid idp.entity_id: urn:framsouza.eu.auth0.com idp.metadata.path: /usr/share/elasticsearch/config/framsouza_eu_auth0_com-metadata.xml order: 2 sp.acs: https://framsouza.co/api/security/v1/saml sp.entity_id: https://framsouza.co sp.logout: https://framsouza.co/logout
Keep in mind the sp.* configurations must point to the Kibana endpoint, and you must collect the idp.* settings from your Identity Provider.
Also, the SAML metadata is stored in a ConfigMap and mounted as a volume inside Elasticsearch. The idp also may provide the metadata via HTTP. In this case, auth0 provides the metadata file via HTTP, but the purpose is to show to you how to mount ConfigMap (or Secret) as a volume into Elasticsearch pods.
The hot node will have a 50Gi available of disk, which refers to the storage class called
sc-zone-b. (We will define the storage class later.) This is the space available to store Elasticsearch data.
volumeClaimTemplates: - metadata: name: elasticsearch-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 50Gi storageClassName: sc-zone-b
Node and pod affinity
The affinity feature restricts scheduling pods in a group of Kubernetes nodes based on labels. Node affinity is conceptually similar to
nodeSelector that defines which node the pod will be scheduled based on the label.
nodeAffinity greatly extends the types of constraints you can express using enhancement labels.
nodeAffinity is using a required
DuringSchedulingIgnoredDuringExecution affinity type. That means rules must be met for a pod to be scheduled on a node.
In this example, we'll define how to only run the pod on nodes in the zone
nodeSelector will be matched with a label on the Kubernetes node, which in this case is
podTemplate: spec: nodeSelector: type: hot affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: failure-domain.beta.kubernetes.io/zone operator: In values: - "europe-west1-b"
In this example, this node will have 8Gi of memory and is requesting at least 2 CPU cores. It will have 4Gi of heap. New Elasticsearch versions (starting with 7.11) automatically set the heap size based on the node roles and available memory. You can read more about this new approach in our docs.
containers: - name: elasticsearch resources: requests: memory: 8Gi cpu: 2 limits: memory: 8Gi
Creating a ConfigMap
Note: You must create this file before the Elasticsearch manifest.
Get the metadata file from your idp (in this case, auth0). To put the content inside a ConfigMap, you can run the following:
kubectl create configmap saml-metadata --from-file=../framsouza_eu_auth0_com-metadata.xml
Remember to adjust the file location.
Volume SAML metadata
volumeMounts session is where the volume should be mounted inside the pod. In this example, you are mounting a volume called
saml-metadata, and the file is located in
volumes means the volume must be mounted according to a ConfigMap called
volumeMounts: - name: saml-metadata mountPath: /usr/share/elasticsearch/config/framsouza_eu_auth0_com-metadata.xml volumes: - name: saml-metadata configMap: name: saml-metadata
Readiness probe is used to know when a container is ready to start accepting traffic. The timeout is 3 seconds by default, which is acceptable in most cases, but if it’s under a very heavy load, you might need to increase the timeout. In this example, we’ll increase the timeout to 10 seconds and adjust the check time to 12 seconds.
readinessProbe: exec: command: - bash - -c - /mnt/elastic-internal/scripts/readiness-probe-script.sh failureThreshold: 3 initialDelaySeconds: 10 periodSeconds: 12 successThreshold: 1 timeoutSeconds: 12
To install or perform any task at the operational system level before Elasticsearch starts, you should use an initContainer. In this example, we’ll install the GCS repository where we can send snapshots.
initContainers: - command: - sh - -c - | bin/elasticsearch-plugin install --batch repository-gcs name: install-plugins
You can also build your own image and include the GCS plugin. To do so, you can follow this blog. This approach is more production ready, so if you decide to go for it, you don’t need to specify the
These are the sessions from one node. The rest of the node configuration is basically the same, except for the name, zone, and
storageClass name. At the end of the file, you see the following setting (which applies to the whole cluster):
updateStrategy: changeBudget: maxSurge: 1 maxUnavailable: 1
updateStrategy controls the number of simultaneous changes in the Elasticsearch cluster.
maxSurge: 1 means only one new pod is created at a time. After the first new pod is Ready, an old pod is killed and the second new pod is created.
If you don’t specify the strategy you want, the default behaviour is
maxSurge: -1, which means that all the required pods are created immediately. This may cause issues if you don’t have enough resources to deal with it. Learn more in our docs.
maxSurge determines how many new pods to create,
maxUnavailable determines how many old pods to kill. In this case, we can only kill one old pod at a time. This ensures the capacity is anywhere from 3 to 1 pod(s).
How to get set up
First let’s give your Google account administrator privileges on the cluster by setting up the role-based access control (RBAC). Run the following command:
kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud auth list --filter=status:ACTIVE --format="value(account)")
Install the ECK operator. Keep in mind we are installing ECK operator version 1.6.0, but you may want to check for updated versions on our site.
kubectl apply -f https://download.elastic.co/downloads/eck/1.6.0/all-in-one.yaml
This creates the ECK Operator pod and cluster permissions in the namespace elastic-system. You can monitor it by running:
kubectl -n elastic-system logs -f statefulset.apps/elastic-operator
StorageClass you can describe the "classes" of storage you want to use. Different classes might map to quality of service levels, disk types, backup purpose, or any arbitrary policy determined by the administrator.
As we are using hot-warm-cold architecture, we need to specify a
StorageClass for the hot and warm/cold nodes (as they must use different disk types) and associate it with the proper
This is an example of
StorageClass that is attached to the
PersistentVolume in the hot nodes:
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: sc-hot provisioner: kubernetes.io/gce-pd parameters: type: pd-ssd reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true
Some key points:
provisioner: kubernetes.io/gce-pd: This means we are using GCE as a provider.
type: pd-ssd: This provides SSD disks for the volumes attached to this
reclaimPolicy: Delete: This deletes
PersistentVolumeClaimresources if the owning Elasticsearch nodes are scaled down or a deletion.
volumeBindingMode: WaitForFirstConsumer: This prevents the pod from being scheduled, due to affinity settings, on a host where the bound
PersistentVolumeis not available. It will also create it in the zone with the unfulfilled claim.
The warm and cold ones have the following configuration:
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: sc-hot provisioner: kubernetes.io/gce-pd parameters: type: pd-standard reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true
They are basically the same, apart from the disk type, which is pd-standard (a slow disk).
Create SAML ConfigMap
kubectl create configmap saml-metadata --from-file=../framsouza_eu_auth0_com-metadata.xml
Create Elasticsearch and Kibana resource
Once you have the SAML ConfigMap, you can apply the Elasticsearch and Kibana manifest:
kubectl create -f eck-saml-hot-warm-cold.yml && kubectl create -f kibana.yml
It may take a while until all the resources get ready, but after a few minutes, you should see something like this:
kubectl get pods NAME READY STATUS RESTARTS AGE elastic-prd-es-cold-zone-b-0 1/1 Running 0 35m elastic-prd-es-cold-zone-c-0 1/1 Running 0 35m elastic-prd-es-cold-zone-d-0 1/1 Running 0 35m elastic-prd-es-hot-zone-b-0 1/1 Running 0 35m elastic-prd-es-hot-zone-c-0 1/1 Running 0 35m elastic-prd-es-hot-zone-d-0 1/1 Running 0 35m elastic-prd-es-master-zone-b-0 1/1 Running 0 35m elastic-prd-es-master-zone-c-0 1/1 Running 0 35m elastic-prd-es-master-zone-d-0 1/1 Running 0 35m elastic-prd-es-warm-zone-b-0 1/1 Running 0 16m elastic-prd-es-warm-zone-c-0 1/1 Running 0 16m elastic-prd-es-warm-zone-d-0 1/1 Running 0 15m kibana-prd-kb-7467b79f54-btzhq 1/1 Running 0 4m8s
Once you have all the pods running, there's a Job you must run to create the SAML role mapping inside of Elasticsearch and give the right permission to the user who will log in using SAML. You can also create the role and role mapping via Kibana DevTools or curl.
If you try to access Kibana via SAML without running this Job, you will get a permission error. In summary, the job will spin up a container, execute the API calls and kill the pod.
At this point you can test the connection via curl or exposing Kibana service.
Grab the Elastic password:
kubectl get secret elastic-prd-es-elastic-user -o yaml
And the Elasticsearch service name and run the following:
curl -k https://elastic:es-password@es-service-name8:9200/_cluster/health?pretty
You can also temporarily expose the Kibana service and access it with your browser:
kubectl port-forward svc/kibana-prd-kb-http 5601
By using Ingress controller, you can access your Kibana via domain controller. There are some Ingress controllers available, but in this example we are using
ingress-nginx. You can read more about it on GitHub.
To deploy it, you must run the following command:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.46.0/deploy/static/provider/cloud/deploy.yaml
In this example, we are using a valid SSL certificate. To add it as part of our Ingress controller, we must first create a Secret, which contains the key and certificate.
kubectl create secret tls framsouza-cert --key framsouza_co_key.txt --cert framsouza_co.crt
Once you created the Secret, you can deploy the Ingress controller manifest. But first let’s have a look at the Ingress manifest:
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" name: ingress spec: tls: - hosts: - framsouza.co secretName: framsouza-cert rules: - host: framsouza.co http: paths: - path: / backend: serviceName: kibana-prd-kb-http servicePort: 5601
At the annotations level, we are explicitly saying that we are using
nginx as a controller and configuring the communication between the Ingress controller and the service to be established via HTTP (
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"). By the default, Elasticsearch uses HTTPS, and without this annotation you may receive a 503 error.
At the TLS level, we are defining the hostname (In this case
framsouza.co) to check the
framsouza-cert, which is the Secret that contains the SSL certificate.
At the rules level, we are creating a redirection. Every request that arrives at the domain
http://framsouza.co must be redirected to the Kibana service (
kibana-prd-kb-http) at the port 5601.
In some cases, you also want to expose Elasticsearch to be accessed by some application using your own SSL certificate. To do so you can add a new rule like:
- path: /elasticsearch backend: serviceName: elastic-prd-es-http servicePort: 9200
Here, every request to
https://framsouza.co/elasticsearch will be redirected to the Elasticsearch service.
Now, we are ready to apply the Ingress manifest:
kubectl create -f ingress.yml
With that, we can access Kibana using our own domain and our own TLS certificate. This also covers the most important features to run an ECK environment in a production environment. Be sure to check out the repo on GitHub.
Interested in trying it out? Simply follow these easy steps to install Elastic Cloud on Kubernetes. Be sure to connect with other users in the Elastic community or reach out on our Discuss forums.