Beats 5.2.0 released
Today we are pleased to announce that Beats 5.2.0 was released. This is the latest stable version and it comes with a few goodies like uptime monitoring, network connections tracking, and the Prometheus exporters integration, enough good reasons to upgrade.
Latest stable release:
Heartbeat, for uptime monitoring
Heartbeat (Beta) is the newest addition to the official Elastic Beats. It periodically checks the status of your services to determine whether they are available and measures the round-trip-time. Like all the other Beats, Heartbeat is lightweight and has no dependencies, so you can install it on multiple locations in your infrastructure and monitor reachability in a distributed fashion.
The idea for Heartbeat came from the Elastic Cloud team, who is using it already for monitoring thousands of Elasticsearch and Kibana clusters. A second source of inspiration was one of the first community Beats, Pingbeat, written by Joshua Rich.
Heartbeat can also be useful for scenarios other than uptime monitoring, such as security use cases, when you need to verify that no one from the outside can access the services on your private enterprise server.
With Heartbeat you can monitor a list of hosts via:
- ICMP (IPv4 and Ipv6) Echo Requests to check whether a service is available. Note that ICMP requires root access.
- TCP to check whether you can connect via TCP to the service. You can optionally verify the endpoint by sending and/or receiving a custom payload
- HTTP to check whether you can connect via HTTP to the service. You can optionally verify that the service returns the expected response, such as specific status code, response header, or content.
Here is an example configuration for using the HTTP monitor to check an Elasticsearch endpoint:
# Configure monitors heartbeat.monitors: - type: http # List or urls to query urls: ["http://localhost:9200"] # Configure task schedule schedule: '@every 10s' # Total test connection and data exchange timeout #timeout: 16s
Heartbeat already supports automatic configuration reloading, so you can dynamically add or remove monitored targets without restarting.
By default, it monitors one IP address for a hostname, but it has support for pinging all resolvable IPs for a hostname. This is useful if you are using a DNS-load balancer and want to ping every IP address for the specified hostname.
Heartbeat is released as Beta in 5.2.0, and we do not recommend to use it in production during the Beta phase.
Track network connections with Metricbeat
Starting with the 5.2 release, Metricbeat exports the network connections between your applications on Linux systems, so you can see the traffic exchanged between services. The system.socket metricset was added. For each TCP socket, it reports the process that opened the socket, the local and remote IPs involved in the communication, and the direction (incoming, outgoing or listening). It can also perform a reverse lookup on the remote IP.
Metricbeat gets the network connections by polling the Linux kernel to get the sockets, so a short polling interval is recommended to catch short lived connections.
Because data gets more valuable when you can visualize it, here is a sample of using Graph to visualize the network connections:
Collect metrics from Prometheus exporters
Starting with the 5.2.0 release, Metricbeat comes with a Prometheus module that collects metrics from the Prometheus exporters or any application that offers a Prometheus endpoint and indexes them to Elasticsearch.
With this module, you can use the Elastic Stack to monitor apps instrumented with the Prometheus libraries, or monitor services for which a Prometheus exporter exists but for which we don’t have a Metricbeat module yet.
To fetch metrics periodically from a Prometheus exporter, you just need to configure collector as part of metricsets and add the host from where to pull the metrics. The metric will be exported under the field configured in namespace. With this, Metricbeat adds support for dynamic metrics where the metrics and their types are not known in advance. While the Prometheus module can be used to get data from a variety of systems, we recommend using the native Metricbeat modules when they are available, because they structure the data in a way that matches the Elastic stack better.
For example to collect metrics from jmx exporter and Varnish exporter, you can use the following configuration:
- module: prometheus metricsets: ["collector"] enabled: true period: 10s hosts: ["localhost:5555"] namespace: jmx - module: prometheus metricsets: ["collector"] enabled: true period: 10s hosts: ["localhost:9131"] namespace: varnish
The Prometheus module in Metricbeat is released as experimental in 5.2.0, and we reserve the right to do breaking compatibility changes.
Feedback
If you want to make use of the new features added in Beats 5.2.0, please download the latest stable version, install it, and let us know what you think on Twitter (@elastic) or in our forum.