Tech Topics

How to configure Elastic Cloud on Kubernetes with SAML and hot-warm-cold architecture

Elastic Cloud on Kubernetes (ECK) is an easy way to get the Elastic Stack up and running on top of Kubernetes. That’s because ECK automates the deployment, provisioning, management, and setup of Elasticsearch, Kibana, Beats, and more.

As logging and metric data — or time series data — has a predictable lifespan, you can use hot, warm, and cold architecture to easily manage your data over time as it ages and becomes less relevant.

In this guide, we’ll explain how to set up SAML with auth0 as an identity provider (IdP) as well as how to configure your deployment for hot-warm-cold architecture with ECK. We’ll use production features such as:

• Dedicated nodes
• Zone awareness
• Pod and node affinity
• PodDisruption
• Dedicated storage class

Before you get started, you should be familiar with SAML. To learn more, check out this ECK example

Scenario

For this example, we’ll deploy ECK to have a centralized way to authenticate into Kibana using auth0.

We’ll deploy the following resources:

• ECK Operator
• StorageClass
• Elasticsearch
• Elasticsearch role job
• Kibana
• Ingress controller

Architecture

To set up this environment, we’ll use GKE (Google Kubernetes Engine) with the following configurations:

Three Kubernetes nodes pools

• 1 hot node pool with 6 Kubernetes instances running spread across 3 availability zones
• 1 warm node pool with Kubernetes instances running spread across 3 availability zones
• 1 cold node pool with Kubernetes instances running spread across 3 availability zones

Instances configuration

• Hot nodes: c2-standard-4 (4 vCPUs, 16GB memory)
• Warm nodes: e2-standard-2 (2 vCPUs, 8GB memory)
• Cold nodes: e2-standard-2 (2 vCPUs, 8GB memory)

We’ll have nine Elasticsearch instances total (with individual tiers for hot, warm, and cold data, as well as dedicated masters), plus two Kibana instances running on GKE zones europe-west1-b, europe-west1-c, and europe-west1-d.

For each GKE node pool, we’ll use a specific Kubernetes label to attach the Elasticsearch instance into the right hardware configuration. The label name is called type, and the values are hot, warm, or cold. You can change and add the label later by using the command line.

First, let’s dive into the manifest explanation, then how you can get set up.

Manifest explanation

The Elasticsearch manifest file eck-saml-hot-warm-cold.yml contains the Elasticsearch settings. Let’s examine the relevant parts of the manifest one by one:

nodeSets

We have one node called hot-zone-b deployed at zone europe-west1-b. Then, we define two routing awareness attributes:

• k8s_node_name: This attribute makes sure that Elasticsearch allocates primary and replica shards to pods running on different Kubernetes nodes and never to pods that are scheduled onto a single Kubernetes node.
• zone: This attribute uses the Kubernetes label domain.beta.kubernetes.io/zone. If Elasticsearch knows which nodes are on the same zone, it can distribute the primary shard and its replica shards to minimise the risk of losing all shard copies in the event of a failure.
nodeSets:
- name: hot-zone-b
count: 1
config:
node.attr.zone: europe-west1-b
cluster.routing.allocation.awareness.attributes: k8s_node_name,zone
node.roles: [ data_hot, data_content ]
node.store.allow_mmap: false


We explicitly say that this node will have a data_hot and data_content role

node.store.allow_mmap : false will prevent the virtual address space (on Linux distribution) from running into errors or exceptions because this default configuration is too low. You can find more information in our documentation.

Xpack.security

This is the SAML configuration. You should follow this guide to configure auth0 as idp and then get the metadata file and the idp.* settings.

• attributes.principal: This defines which SAML attribute is going to be mapped to the principal (username) of the authenticated user in Kibana.
• idp.entity_id: This is the SAML EntityID of your Identity Provider (which you can get from SAML metadata file).
• idp.metadata.path: This is the file path or the HTTPs URL where your Identity Provider metadata is available. In this example, we’ll use a file to demonstrate how to mount a volume.
• sp.acs: This is the Assertion Consumer Service URL where Kibana is listening for incoming SAML messages.
• sp.entity_id: This is the SAML EntityID of our Service Provider.
• sp.logout: This is the SingleLogout endpoint where the Service Provider is listening for incoming SAML LogoutResponse and LogoutRequest messages.
      xpack:
security:
authc:
realms:
saml:
saml1:
attributes.principal: nameid
idp.entity_id: urn:framsouza.eu.auth0.com
order: 2
sp.acs: https://framsouza.co/api/security/v1/saml
sp.entity_id: https://framsouza.co
sp.logout: https://framsouza.co/logout


Keep in mind the sp.* configurations must point to the Kibana endpoint, and you must collect the idp.* settings from your Identity Provider.

Also, the SAML metadata is stored in a ConfigMap and mounted as a volume inside Elasticsearch. The idp also may provide the metadata via HTTP. In this case, auth0 provides the metadata file via HTTP, but the purpose is to show to you how to mount ConfigMap (or Secret) as a volume into Elasticsearch pods.

volumeClaimTemplates

The hot node will have a 50Gi available of disk, which refers to the storage class called sc-zone-b. (We will define the storage class later.) This is the space available to store Elasticsearch data.

    volumeClaimTemplates:
name: elasticsearch-data
spec:
accessModes:
resources:
requests:
storage: 50Gi
storageClassName: sc-zone-b


Node and pod affinity

The affinity feature restricts scheduling pods in a group of Kubernetes nodes based on labels. Node affinity is conceptually similar to nodeSelector that defines which node the pod will be scheduled based on the label. nodeAffinity greatly extends the types of constraints you can express using enhancement labels.

nodeAffinity is using a required DuringSchedulingIgnoredDuringExecution affinity type. That means rules must be met for a pod to be scheduled on a node.

In this example, we'll define how to  only run the pod on nodes in the zone europe-west1-b. The nodeSelector will be matched with a label on the Kubernetes node, which in this case is type: hot.

    podTemplate:
spec:
nodeSelector:
type: hot
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: failure-domain.beta.kubernetes.io/zone
operator: In
values:
- "europe-west1-b"


Containers definition

In this example, this node will have 8Gi of memory and is requesting at least 2 CPU cores. It will have 4Gi of heap. New Elasticsearch versions (starting with 7.11) automatically set the heap size based on the node roles and available memory. You can read more about this new approach in our docs.

         containers:
- name: elasticsearch
resources:
requests:
memory: 8Gi
cpu: 2
limits:
memory: 8Gi


Creating a ConfigMap

Note: You must create this file before the Elasticsearch manifest.

Get the metadata file from your idp (in this case, auth0). To put the content inside a ConfigMap, you can run the following:

kubectl create configmap saml-metadata


Remember to adjust the file location.

The volumeMounts session is where the volume should be mounted inside the pod. In this example, you are mounting a volume called saml-metadata, and the file is located in ES_CONFIG directory /usr/share/elasticsearch/config/framsouza_eu_auth0_com-metadata.xml.

Then, volumes means the volume must be mounted according to a ConfigMap called saml-metadata.

          volumeMounts:
volumes:
configMap:


Readiness probe is used to know when a container is ready to start accepting traffic. The timeout is 3 seconds by default, which is acceptable in most cases, but if it’s under a very heavy load, you might need to increase the timeout. In this example, we’ll increase the timeout to 10 seconds and adjust the check time to 12 seconds.

          readinessProbe:
exec:
command:
- bash
- -c
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 12
successThreshold: 1
timeoutSeconds: 12


initContainer

To install or perform any task at the operational system level before Elasticsearch starts, you should use an initContainer. In this example, we’ll install the GCS repository where we can send snapshots.

  initContainers:
- command:
- sh
- -c
- |
bin/elasticsearch-plugin install --batch repository-gcs
name: install-plugins


You can also build your own image and include the GCS plugin. To do so, you can follow this blog. This approach is more production ready, so if you decide to go for it, you don’t need to specify the install-plugins initContainer.

These are the sessions from one node. The rest of the node configuration is basically the same, except for the name, zone, and storageClass name. At the end of the file, you see the following setting (which applies to the whole cluster):

  updateStrategy:
changeBudget:
maxSurge: 1
maxUnavailable: 1


updateStrategy controls the number of simultaneous changes in the Elasticsearch cluster. maxSurge: 1 means only one new pod is created at a time. After the first new pod is Ready, an old pod is killed and the second new pod is created.

If you don’t specify the strategy you want, the default behaviour is maxSurge: -1, which means that all the required pods are created immediately. This may cause issues if you don’t have enough resources to deal with it. Learn more in our docs.

While maxSurge determines how many new pods to create, maxUnavailable determines how many old pods to kill. In this case, we can only kill one old pod at a time. This ensures the capacity is anywhere from 3 to 1 pod(s).

How to get set up

Install ECK

First let’s give your Google account administrator privileges on the cluster by setting up the role-based access control (RBAC). Run the following command:

kubectl create clusterrolebinding cluster-admin-binding
--filter=status:ACTIVE --format="value(account)")


Install the ECK operator. Keep in mind we are installing ECK operator version 1.6.0, but you may want to check for updated versions on our site.

kubectl apply -f


This creates the ECK Operator pod and cluster permissions in the namespace elastic-system. You can monitor it by running:

kubectl -n elastic-system logs -f statefulset.apps/elastic-operator


Configuring StorageClass

With StorageClass you can describe the "classes" of storage you want to use. Different classes might map to quality of service levels, disk types, backup purpose, or any arbitrary policy determined by the administrator.

As we are using hot-warm-cold architecture, we need to specify a StorageClass for the hot and warm/cold nodes (as they must use different disk types) and associate it with the proper PersistentVolume.

This is an example of StorageClass that is attached to the PersistentVolume in the hot nodes:

apiVersion: storage.k8s.io/v1
kind: StorageClass
name: sc-hot
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-ssd
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true


Some key points:

• provisioner: kubernetes.io/gce-pd: This means we are using GCE as a provider.
• type: pd-ssd: This provides SSD disks for the volumes attached to this StorageClass.
• reclaimPolicy: Delete: This deletes PersistentVolumeClaim resources if the owning Elasticsearch nodes are scaled down or a deletion.
• volumeBindingMode: WaitForFirstConsumer: This prevents the pod from being scheduled, due to affinity settings, on a host where the bound PersistentVolume is not available. It will also create it in the zone with the unfulfilled claim.

The warm and cold ones have the following configuration:

apiVersion: storage.k8s.io/v1
kind: StorageClass
name: sc-hot
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true


They are basically the same, apart from the disk type, which is pd-standard (a slow disk).

Create SAML ConfigMap

kubectl create configmap saml-metadata


Create Elasticsearch and Kibana resource

Once you have the SAML ConfigMap, you can apply the Elasticsearch and Kibana manifest:

kubectl create -f eck-saml-hot-warm-cold.yml && kubectl create -f kibana.yml


It may take a while until all the resources get ready, but after a few minutes, you should see something like this:

kubectl get pods
elastic-prd-es-cold-zone-b-0     1/1     Running    0          35m
elastic-prd-es-cold-zone-c-0     1/1     Running    0          35m
elastic-prd-es-cold-zone-d-0     1/1     Running    0          35m
elastic-prd-es-hot-zone-b-0      1/1     Running    0          35m
elastic-prd-es-hot-zone-c-0      1/1     Running    0          35m
elastic-prd-es-hot-zone-d-0      1/1     Running    0          35m
elastic-prd-es-master-zone-b-0   1/1     Running    0          35m
elastic-prd-es-master-zone-c-0   1/1     Running    0          35m
elastic-prd-es-master-zone-d-0   1/1     Running    0          35m
elastic-prd-es-warm-zone-b-0     1/1     Running    0          16m
elastic-prd-es-warm-zone-c-0     1/1     Running    0          16m
elastic-prd-es-warm-zone-d-0     1/1     Running    0          15m
kibana-prd-kb-7467b79f54-btzhq   1/1     Running    0          4m8s


Once you have all the pods running, there's a Job you must run to create the SAML role mapping inside of Elasticsearch and give the right permission to the user who will log in using SAML. You can also create the role and role mapping via Kibana DevTools or curl.

If you try to access Kibana via SAML without running this Job, you will get a permission error. In summary, the job will spin up a container, execute the API calls and kill the pod.

Test connection

At this point you can test the connection via curl or exposing Kibana service.

kubectl get secret elastic-prd-es-elastic-user -o yaml


And the Elasticsearch service name and run the following:

curl -k


You can also temporarily expose the Kibana service and access it with your browser:

kubectl port-forward svc/kibana-prd-kb-http 5601


Ingress controller

By using Ingress controller, you can access your Kibana via domain controller. There are some Ingress controllers available, but in this example we are using ingress-nginx. You can read more about it on GitHub.

To deploy it, you must run the following command:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.46.0/deploy/static/provider/cloud/deploy.yaml


In this example, we are using a valid SSL certificate. To add it as part of our Ingress controller, we must first create a Secret, which contains the key and certificate.

kubectl create secret tls framsouza-cert --key framsouza_co_key.txt --cert framsouza_co.crt


Once you created the Secret, you can deploy the Ingress controller manifest. But first let’s have a look at the Ingress manifest:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
name: ingress
spec:
tls:
- hosts:
- framsouza.co
secretName: framsouza-cert
rules:
- host: framsouza.co
http:
paths:
- path: /
backend:
serviceName: kibana-prd-kb-http
servicePort: 5601


At the annotations level, we are explicitly saying that we are using nginx as a controller and configuring the communication between the Ingress controller and the service to be established via HTTP (nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"). By the default, Elasticsearch uses HTTPS, and without this annotation you may receive a 503 error.

At the TLS level, we are defining the hostname (In this case framsouza.co) to check the secretname called framsouza-cert, which is the Secret that contains the SSL certificate.

At the rules level, we are creating a redirection. Every request that arrives at the domain http://framsouza.co must be redirected to the Kibana service (kibana-prd-kb-http) at the port 5601.

In some cases, you also want to expose Elasticsearch to be accessed by some application using your own SSL certificate. To do so you can add a new rule like:

          - path: /elasticsearch
backend:
serviceName: elastic-prd-es-http
servicePort: 9200


Here, every request to https://framsouza.co/elasticsearch will be redirected to the Elasticsearch service.

Now, we are ready to apply the Ingress manifest:

 kubectl create -f ingress.yml


With that, we can access Kibana using our own domain and our own TLS certificate. This also covers the most important features to run an ECK environment in a production environment. Be sure to check out the repo on GitHub.

Getting started

Interested in trying it out?  Simply follow these easy steps to install Elastic Cloud on Kubernetes. Be sure to connect with other users in the Elastic community or reach out on our Discuss forums.

• We're hiring

Work for a global, distributed team where finding someone like you is just a Zoom meeting away. Flexible work with impact? Development opportunities from the start?