It’s finally here! After about 11 months of work, five Alpha versions, one Beta, and one RC, we have a new major version of your favorite data shippers. And it’s not only Beats that are releasing today. In fact, starting with this release, all the projects from the Elastic stack are releasing at the same time and with the same version numbers down to the patch level. That’s why we’ve got the nice jump from 1.3.1 to 5.0.0 in version numbers for Beats.
If you’ve been following the alphas and betas, you know already what’s in, so we won’t stay in your way if you’re just looking for the Download Now link. If not, then this blog post highlights the biggest changes coming with 5.0.
If you are upgrading from 1.x, please read the Breaking Changes docs and the Upgrading guide.
Metricbeat - You know, for metrics
Metricbeat replaces Topbeat as the primary tool for collecting metrics in the Elastic stack. Like Topbeat, Metricbeat collects “top” like statistics about host and per process resources (CPU, memory, disk, network). Unlike Topbeat, Metricbeat also collects metrics from systems such as Apache, HAProxy, MongoDB, MySQL, Nginx, PostgreSQL, Redis, or Zookeeper, with more to come in the near future.
The system module in Metricbeat offers all the functionality of Topbeat, and it also extends this functionality with new features like per container metrics, disk IO metrics, and network IO metrics. The configurations of the two Beats are not compatible, but migrating from one to another is fairly simple.
And the good news doesn’t stop here: Metricbeat takes advantage of the new number types and storage implementation of Elasticsearch 5.0, which make numbers faster to index, faster to search, more disk-efficient and more memory-efficient.
Filebeat - under the hood improvements
We’ve refactored the way Filebeat stores the state of of harvested files (which files are opened, inode numbers for each, how far we’ve read in each one, etc.). We’ve also cleaned up the settings around when the files should be closed, ignored or forgotten from the state. We expect these changes to improve the robustness of Filebeat against corner cases (for example inode reuse or very quick file rotation), so we recommend that you upgrade.
In addition, Filebeat can now natively decode JSON objects from log lines. This is useful for structured logging, where the logging library writes the metadata directly formatted as JSON.
Packetbeat now reports statistics like packet count and byte count about IP and TCP flows, regardless of the upper layer protocols. This opens Packetbeat to a new set of use cases, giving insights into how the traffic is flowing through the network and offers a limited level of visibility into encrypted traffic.
In addition, Packetbeat got support for a few more application protocols, including AMQP, Cassandra, EDNS, DNSSEC, and NFS.
Filter with the power of processors
Beats “processors” are simple rules that manipulate the events just before they are shipped from the Beat. While similar in nature with the Ingest Node processors or Logstash filters, don’t expect the same power. The Beats processors are... well, lightweight, and focus on filtering the data to avoid sending too much of it over the network when it will be discarded anyway by Logstash.
That said, processors give you great flexibility in choosing the data that should be shipped. Here is a quick example:
processors: - drop_event: when: equals: http.code: 200 - drop_fields: fields: [“http.phrase”]
We listened to your feedback and we’ve added native Kafka output support in Beats, at the same time greatly improving the existing Redis output. This means that if you are passing all messages through a Kafka queue anyway, you no longer need an extra Logstash instance to convert between Beats and Kafka.
The Beats Kafka output is compatible with Kafka versions 0.8, 0.9, and 0.10 and supports encryption, authentication, as well as dynamic topic and partition selection.
Beats 5.0 uses the same YAML-based configuration format, but improved with many small features that make the deployment of Beats easier and more pleasant. The new features include better namespacing, reference variables, better regular expression support, and the ability to overwrite any setting from the command line with the
You can read more about the new Beats configuration language in the docs.
Tell me more
Do you want to read more on what’s new in 5.0? Have a look at the release notes and the blog posts for the pre-releases:
- Beats 5.0.0-alpha1
- Beats 5.0.0-alpha2
- Beats 5.0.0-alpha3
- Beats 5.0.0-alpha4
- Beats 5.0.0-alpha5
- Beats 5.0.0-beta1
- Beats 5.0.0-rc1
Thank you and community credits
And now, it’s party time!
First of all, we would like to thank everyone that has tried the 5.0 pre-releases and posted the issues they found. Each one of you will receive a small thank you gift from Elastic.
We’d also like to give special credit to the following community members for contributing significant code to the 5.0 release:
- @kofemann for the NFS support in Packetbeat
- @mrkschan for the Nginx module in Metricbeat
- @mkstork for the EDNS and DNSSEC support in Packetbeat
- Al Lefebvre for the HAProxy module in Metricbeat
- Arnout Engelen for the IPv6 support in Packetbeat’s socket to process mapping code
- Chris Black for the Redis module in Metricbeat
- Cyrille Verrier for packaging improvements
- Radovan Ondas for the Apache module in Metricbeat
- Thomas Paulmyer for the AMQP support in Packetbeat