How to deploy Elasticsearch and Kibana on AWS EKS auto mode with ECK

New to Elasticsearch? Join our getting started with Elasticsearch webinar. You can also start a free cloud trial or try Elastic on your machine now.

This article is part of a series where we will learn how to install Elasticsearch using different infrastructures.

This time, we are going to work with the Kubernetes Service (EKS) using Auto Mode. In the other articles, you’ll learn how to use AWS EC2 and AWS Marketplace.

For Elasticsearch, we are going to use Elastic Cloud on Kubernetes (ECK), which is the official Elasticsearch Kubernetes operator that simplifies the Kubernetes deployments of all the Elastic Stack components.

ECK requires significantly more effort than the Marketplace-based Elastic Cloud solutions, but it's more automated than deploying VMs by yourself, because the Kubernetes operator will take care of the system orchestration and node scaling.

What is EKS Auto Mode?

EKS Auto Mode is a deployment model provided by AWS, which simplifies the configuration of the data plane; things like storage, networking, and compute are now managed by EKS. With a one-click installation, Auto Mode will automatically select the optimal resources and scale them for you when needed. You can explore other AWS flavors in our AWS EC2 and AWS Marketplace articles.

How to set up the ECK operator

Now, we are going to install eksctl, which is the official CLI to manage AWS EKS from the terminal, and then install and run the ECK operator on the deployed EKS Kubernetes cluster.

1. Log in to AWS

2. In the bottom left, click on the CloudShell button to access the console, and deploy the EKS cluster from there. Alternatively, you can use AWS CLI

3. Install eksctl to manage the EKS clusters via CloudShell

4. Create an Auto Mode EKS cluster

5. Wait until it is ready. It takes around 10 minutes to create

6. Install the Elastic Cloud on Kubernetes (ECK) operator

7. Create storageClass.

We create a gp3 EBS storageClass and EKS Auto Mode will automatically install the CSI Driver. You can see more disk alternatives here, and ECK storage recommendations here.

8. Let’s create a single-node Elasticsearch instance that utilizes the storageClass we just created. We will request 1GB RAM and 1 cpu.

If you want to check some recipes for different setups, you can visit this link.

9. Let’s also deploy a Kibana single-node cluster. For Kibana, we will add a LoadBalancer, which will give us an external IP we can use to reach Kibana from our device.

Note the annotation:

service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"

It is very important because it tells Auto Mode to provide a public-facing LoadBalancer. If not set, the LoadBalancer will be internal.

10. Check that your pods are running

11. You can also run kubectl get elasticsearch and kubectl get kibana for more specific stats like Elasticsearch version, nodes, and health.

12. Access your services

This will show you the external URL for Kibana under EXTERNAL-IP. It might take a few minutes for the LoadBalancer to provision. Copy the value of EXTERNAL-IP.

13. Get the Elasticsearch password for the elastic user:

14. Access Kibana through your browser:

a. URL: https://<EXTERNAL_IP>:5601

b. Username:elastic

c. Password:Arc133835D8507OIgKqkWBGx (from previous step)

15. When accessing Elastic from your browser you will see this welcome screen.

If you want to change the Elasticsearch cluster specifications like changing or resizing nodes, you can apply the yml manifest again with the new settings:

In this example, we are going to add one more node, increase RAM, CPU, and disk.

The same applies to Kibana:

We can adjust the container CPU/RAM and also the Node.js memory usage (max-old-space-size)

Keep in mind that existing volume claims cannot be downsized. After applying the update, the operator will make the changes with minimal disruption time.

Remember to delete the cluster when you're done testing to avoid unnecessary costs.

Conclusion

In this guide, we deployed Elasticsearch and Kibana on AWS EKS Auto Mode using ECK. Auto Mode handles the cluster resources for you, while ECK manages scaling and orchestration. The result is a setup that takes more effort than using the AWS Marketplace, but much less than running Elasticsearch directly on VMs. It’s a good balance between control and simplicity, and a good base to build on.

Next steps

If you want to learn more about Kubernetes and Auto Mode, check out these articles:

• Elastic Cloud on Kubernetes | Elastic Docs
• Amazon Elastic Kubernetes Service (EKS)
• Automate cluster infrastructure with EKS Auto Mode - Amazon EKS