Monitor Kubernetes with Beats Autodiscover Feature and Elasticsearch

Editor's Note (August 3, 2021): This post uses deprecated features. Please reference the map custom regions with reverse geocoding documentation for current instructions.

This post is part of the Elastic{ON} 2018 blog series where we recap specific demos and related deep-dive sessions from the conference. From machine learning forecasting to APM to security analytics with Mr. Robot — check out the full list below.

For any monitoring solution, it is critical to be able to dynamically adjust the configuration when something happens in your Kubernetes cluster. This past year Elastic embraced a new way of building applications — in particular we’ve done a lot when it comes to autodiscover for various services running in a common platform. Watch Monica Sarbu demo how Metricbeat and Filebeat can help dynamically monitor your Kubernetes ecosystem.

Because you don't want to change your Beats configuration each time a new container is deployed, the autodiscover feature allows you to watch for Kubernetes/Docker changes plus the application inside the container and will dynamically adapt settings for them as they occur. This is especially useful when running your infrastructure on containers.

Beginning with the release of Beats 6.2, for Kubernetes there are manifest files for easier deployment of Metricbeat and Filebeat. For example, in the Metricbeat manifest file, you can configure the autodiscover feature to listen to Kubernetes events — or other containers, such as Docker if that’s what you use.

To make monitoring easier, we are working to add a new app for Kibana called Infra-UI that can quickly provide the overview of running services as well as your overall infrastructure. Again, for this demo we show how this works for Kubernetes, yet it will also monitor containers in Docker. In Kibana, there will be a Kubernetes tab where you will see all of the different pods — and for each pod, you can see an overview dashboard. The dashboard will show metrics such as CPU usage, memory usage, network traffic, and more. Beats makes it easy to collect metrics and logs from your pods with Metricbeat and Filebeat.

In our demo, we show a deployment for Nginx for two pods running on different nodes, then we scale the deployment to five Nginx pods. You’ll see data in real time on the dashboard. Each time a new container for Nginx starts, then it enables the modules to deploy the Nginx service inside the new pod. Additionally, you can see the collected logs from all of the instances as well as the server and access logs, and in the Metricbeat overview, you can see aggregated metrics that are coming from Nginx instances, such as active connections, request rate, growth rate, and more.

At Elastic, we try to see beyond how logging and metrics are done currently. We look at the present state of technology and try to figure out how these things will be done in the very near future. During What’s Brewing in Beats, Monica — along with Tudor Golubenco — talk about what’s coming up for the platform. Also, watch Kubernetes, Docker, and Containers at Elastic: Monitoring, Logging, and More for more futuristic inspirations or check out a real-world use case with Monitoring Anything and Everything with Beats at eBay.

See what else we announced during the conference in these recaps: