<p>Elastic Cloud on Kubernetes (ECK) is an operator that allows you to automate the deployment of the Elastic Stack — including Elasticsearch, Kibana, and Elastic APM, Elastic SIEM, and more — using Kubernetes. By using this ECK, you can quickly and easily deploy Elasticsearch clusters with Kubernetes, as well as secure and upgrade your Elasticsearch clusters. It is the only official Elasticsearch operator.
</p>
<p>In this article, you will learn how to:
</p>
<ol>
	<li>Create a Kubernetes cluster using Google Kubernetes Engine (GKE)</li>
	<li>Deploy an Elasticsearch cluster using ECK across three zones</li>
	<li>Scale out the Elasticsearch cluster</li>
	<li>Expose the Elasticsearch cluster externally (using Google Cloud’s load balancer)</li>
</ol> <strong>
<h2>Prerequisites</h2></strong>
<p>To run the instructions on this page, you must first install the following:
</p>
<ul>
	<li>Google Cloud SDK</li>
	<li>kubectl</li>
</ul> <strong>
<h3>Create GKE cluster</h3></strong>
<p>Creating a Kubernetes cluster using GKE is very straightforward. Navigate to the Kubernetes Engine page and select <strong>Create Cluster</strong>. To ensure high-availability and prevent data loss, you want to create a cluster with nodes that go across three availability zones in a region, so select <strong>Regional</strong> under <strong>Location Type</strong>. This will create Kubernetes nodes across multiple zones in the region selected.
</p>
<p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt4d8f05514fc9d6eb/5e308b36507d2f74fb78c84a/ha-eck-gke-blog-template.png" data-sys-asset-uid="blt4d8f05514fc9d6eb" alt="Setting up the GKE cluster" style="display: block; margin: auto; width: 600px;">
</p><strong>
<h3>Set up kubectl context</h3></strong>
<p>Now that you have set up the Kubernetes cluster, you need a way to communicate with it. <code>kubectl</code> is the command line tool for Kubernetes which is used to run commands against the Kubernetes cluster. To interact with the Kubernetes cluster you’ve set up, you need to configure the context <code>kubectl</code> should use. You can do this easily using the <code>gcloud</code> command:
</p>
<pre class="prettyprint">gcloud container clusters get-credentials [CLUSTER_NAME]
</pre>
<p>This creates an entry into the <code>kubeconfig</code> file with default credentials, which you can then use to set the context for <code>kubectl</code>. You should see the following:
</p>
<pre class="prettyprint">Fetching cluster endpoint and auth data. 
kubeconfig entry generated for &lt;cluster-name&gt;.
</pre>
<p>You can see the entry that’s been added using:
</p>
<pre class="prettyprint">kubectl config view
</pre>
<p>Now specify the context (cluster name) to use for <code>kubectl</code> with:
</p>
<pre class="prettyprint">kubectl config use-context &lt;cluster name&gt;
</pre>
<p>The name of the cluster should follow the pattern of <code>gke_&lt;project_name&gt;_&lt;region&gt;_&lt;cluster name&gt;</code>. This will be listed under <code>name</code> in <code>kubeconfig</code>.
</p>
<p>Check the context being used by kubectl:
</p>
<pre class="prettyprint">kubectl config current-context
</pre> <strong>
<h2>Installing Elastic Cloud on Kubernetes</h2></strong>
<p>Now the Kubernetes cluster has been set up in Google Cloud, the next step is to install the ECK operator. Follow <a href="https://www.elastic.co/guide/en/cloud-on-k8s/1.0/k8s-quickstart.html#k8s-deploy-eck">these instructions</a> to install ECK.
</p> <strong>
<h3>Create storage class</h3></strong>
<p>Since we chose to deploy to multiple zones, we’ll need to ensure that persistent volumes are created in the correct zone (matching the zone the node is deployed in). To do so you will need to create a new storage class using the following configuration:
</p>
<pre class="prettyprint">kind: StorageClass 
apiVersion: storage.k8s.io/v1 
metadata: 
  name: zone-storage 
provisioner: kubernetes.io/gce-pd 
volumeBindingMode: WaitForFirstConsumer
</pre>
<p>Create it by running:
</p>
<pre class="prettyprint">kubectl create -f storage_class.yml
</pre>
<p>Now switch the default storage class from standard to the one you’ve just created. First switch the default class to false for the standard storage class:
</p>
<pre class="prettyprint">kubectl patch storageclass standard -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
</pre>
<p>Then switch the default class to true for the custom storage class:<strong>&nbsp;</strong>
</p>
<pre class="prettyprint">kubectl patch storageclass zone-storage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
</pre> <strong>
<h3>Deploy Elasticsearch using ECK</h3></strong>
<p>You can now deploy your Elasticsearch cluster by following the steps in the <a href="https://www.elastic.co/guide/en/cloud-on-k8s/1.0/k8s-quickstart.html#k8s-deploy-elasticsearch">ECK documentation</a> using the following <a href="https://www.elastic.co/guide/en/cloud-on-k8s/1.0/k8s-advanced-node-scheduling.html#k8s-availability-zone-awareness">configuration file</a>. You will need to change the <code>node.attr.zone</code> value to match the node zones used in your GKE cluster.
</p>
<pre class="prettyprint">apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: ha
spec:
  version: 7.5.2
  nodeSets:
  - name: zone-b
    count: 1
    config:
      node.attr.zone: europe-west1-b
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: failure-domain.beta.kubernetes.io/zone
                  operator: In
                  values:
                  - europe-west1-b
  - name: zone-c
    count: 1
    config:
      node.attr.zone: europe-west1-c
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: failure-domain.beta.kubernetes.io/zone
                  operator: In
                  values:
                  - europe-west1-c
  - name: zone-d
    count: 1
    config:
      node.attr.zone: europe-west1-d
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: failure-domain.beta.kubernetes.io/zone
                  operator: In
                  values:
                  - europe-west1-d
</pre>
<p>This creates 3 pods in 3 separate zones (in this example: europe-west1-b, europe-west1-c, and europe-west1-d, which are the 3 zones the Kubernetes cluster was created in) with a node set in each zone. To monitor the creation of the Elasticsearch cluster, follow instructions in the <a href="https://www.elastic.co/guide/en/cloud-on-k8s/1.0/k8s-quickstart.html#k8s_monitor_cluster_health_and_creation_progress">ECK monitoring documentation</a>.
</p> <strong>
<h2>Update strategy</h2></strong>
<p>Now the cluster has been created, you might decide you need more capacity and need to increase the number of nodes in your cluster. You can do so by updating the deployment configuration and applying the updates.
</p>
<p>The updated configuration file has a master-eligible only node and two data nodes in each of the three zones.
</p>
<pre class="prettyprint">apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: ha
spec:
  version: 7.5.2
  nodeSets:
  - name: zone-b
    count: 1
    config:
      node.master: true
      node.data: false
      node.ingest: false
      node.attr.zone: europe-west1-b
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: failure-domain.beta.kubernetes.io/zone
                  operator: In
                  values:
                  - europe-west1-b
  - name: zone-b-data
    count: 2
    config:
      node.master: false
      node.data: true
      node.ingest: false
      node.attr.zone: europe-west1-b
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: failure-domain.beta.kubernetes.io/zone
                  operator: In
                  values:
                  - europe-west1-b
  - name: zone-c
    count: 1
    config:
      node.master: true
      node.data: false
      node.ingest: false
      node.attr.zone: europe-west1-c
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: failure-domain.beta.kubernetes.io/zone
                  operator: In
                  values:
                  - europe-west1-c
  - name: zone-c-data
    count: 2
    config:
      node.master: false
      node.data: true
      node.ingest: false
      node.attr.zone: europe-west1-c
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: failure-domain.beta.kubernetes.io/zone
                  operator: In
                  values:
                  - europe-west1-c
  - name: zone-d
    count: 1
    config:
      node.master: true
      node.data: false
      node.ingest: false
      node.attr.zone: europe-west1-d
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: failure-domain.beta.kubernetes.io/zone
                  operator: In
                  values:
                  - europe-west1-d
  - name: zone-d-data
    count: 2
    config:
      node.master: false
      node.data: true
      node.ingest: false
      node.attr.zone: europe-west1-d
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: failure-domain.beta.kubernetes.io/zone
                  operator: In
                  values:
                  - europe-west1-d
</pre>
<p>You can now create additional nodes by running:
</p>
<pre class="prettyprint">kubectl apply -f es.yml
</pre>
<p>To find out more about the update strategy, check out the <a href="https://www.elastic.co/guide/en/cloud-on-k8s/1.0/k8s-update-strategy.html">ECK update documentation</a>.
</p> <strong>
<h2>Exposing the service</h2></strong>
<p>Now the cluster has been created, the next question is how to access Elasticsearch outside of Kubernetes. You can first test locally by port forwarding to localhost:
</p>
<pre class="prettyprint">kubectl port-forward service/ha-es-http 9200
</pre>
<p>The password for the elastic user is stored as a Kubernetes secret and can be retrieved with:
</p>
<pre class="prettyprint">kubectl get secret ha-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode
</pre>
<p>You can now connect to the Elasticsearch cluster on <a href="https://localhost:9200">https://localhost:9200</a>. For more details, check out the <a href="https://www.elastic.co/guide/en/cloud-on-k8s/1.0/k8s-accessing-elastic-services.html#k8s-services">accessing ECK documentation</a>.
</p>
<p>To expose the service externally, you can specify the service to be of <a href="https://www.elastic.co/guide/en/cloud-on-k8s/1.0/k8s-accessing-elastic-services.html#k8s-allow-public-access">LoadBalancer type</a> by including <code>http.service.spec.type</code> in the deployment configuration:
</p>
<pre class="prettyprint">spec: 
  version: 7.5.1 
  http: 
    service: 
      spec: 
        type: LoadBalancer
</pre>
<p>By changing the type to <code>LoadBalancer</code>, the service is assigned an external IP which can be used to access the Elasticsearch cluster externally.
</p>
<pre class="prettyprint">kubectl get svc 
NAME            TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)          AGE 
ha-es-http      LoadBalancer   10.0.44.239   35.241.231.57   9200:31857/TCP   31m
</pre>
<p>You can also create an <code>Ingress</code> for better control of external access.&nbsp;
</p> <strong>
<h2>Conclusions</h2></strong>
<p>You now have a highly available Elasticsearch cluster deployed across 3 zones! If you haven’t run out of energy yet, follow our <a href="https://www.elastic.co/blog/getting-started-with-elastic-cloud-on-kubernetes-data-ingestion">Getting started with Elastic Cloud on Kubernetes: Data ingestion</a> blog to get some data into your cluster. And if you’re feeling really ambitious, why not try <a href="https://www.elastic.co/guide/en/cloud-on-k8s/1.0/k8s-kibana.html">deploying Kibana</a> or <a href="https://www.elastic.co/guide/en/cloud-on-k8s/1.0/k8s-apm-server.html">Elastic APM</a> using ECK next? If you run into any problems, reach out on our <a href="https://discuss.elastic.co/c/eck/79">ECK Discuss</a> forum. Enjoy!&nbsp;
</p>

eck-blog-banner.png

eck-blog-thumb.png

High availability Elasticsearch on Kubernetes with ECK and GKE

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Blog Footer

Lizzie Wann

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Load More

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Elasticsearch Platform

ELK Stack

Elastic Cloud

Observability

Security

Search

By industry

By solution

Customer spotlight

Developers

Connect

Learn

Help

See what's happening at Elastic

Author

Articles by Lizzie Wann

High availability Elasticsearch on Kubernetes with ECK and GKE

Follow us

About us

Join us

Press

Partners

Trust & Security

Investor relations

EXCELLENCE AWARDS