At almost precisely a year after the Packetbeat team has joined Elastic, we’re excited to reveal the first alpha release of the Filebeat, Packetbeat, Topbeat, and Winlogbeat next major versions.
One version to rule them all
You might be wondering why we’re jumping from version 1.2 directly to 5.0. To make our software suddenly more stable and to one-up our competition, of course. In seriousness, all the projects in the Elastic stack are doing releases in sync and will use the same version numbers from now on. As Kibana is currently at 4.5, we’re all going with 5.0 as the next major.
This is to avoid the support matrix from hell, for example now you need to know that Beats 1.2 were tested against Elasticsearch 2.3, Logstash 2.3 and Kibana 4.5. Starting with 5.0, you’ll know that if Beats and Elasticsearch have the same version number, they were released at the same time and we have tested them together. It simplifies communication all around.
New Features
Beats 5.0-alpha1 comes packed with new features and you can expect more of them to land in during the alpha and beta phases. Here are some of the highlights from Alpha 1:
Custom fields and generic filtering
You now have more freedom over how the documents created by the Beats look like. On one hand, you can now add custom fields and tags per Beat and module. On the other hand, you can use the newly introduced generic filtering to remove the fields that you don’t want. These features are implemented at the libbeat level, meaning that all community Beats automatically benefit from them as soon as they upgrade.
JSON support in Filebeat
Filebeat can now natively decode JSON objects from log lines. This is useful for structured logging, where the logging library writes the metadata directly formatted as JSON. This can also be used as a convenient way of collecting logs from Docker hosts, because Docker uses JSON to wrap the log lines from the application.
Integration with Ingest Node
The new Ingest Node functionality, released with Elasticsearch 5.0.0-alpha1, is big news because it gives users processing capabilities similar with Logstash directly in Elasticsearch! This makes it really easy to get started with the Elastic stack. For simple logging usecases, for example, you only need Filebeat and Elasticsearch.
All Beats can work with the Ingest Node, simply set the pipeline parameter in the Elasticsearch output configuration.
Packetbeat IP/TCP flows
So far Packetbeat was focused on the application layer protocols, giving you visibility into the business transactions as seen in the network. Packetbeat now also reports statistics like packet count and byte count about IP and TCP flows, regardless of the upper layer protocols. This opens Packetbeat to a new set of use cases, giving insights into how the traffic is flowing through the network.
Kafka output
We listened to your feedback and we’ve added Kafka output support in Beats, at the same time removing the deprecation mark for the Redis output. This means that if you are passing all messages through a Kafka queue anyway, you won’t need a Logstash instance to convert between Beats and Kafka.
Winlogbeat improvements
Winlogbeat now extracts all the fields from Windows event log records including the EventData and UserData fields and includes them in the documents it indexes. In addition, now it is possible to select events by event ID, level, and provider. Winlogbeat efficiently implements this event selection by using a query with Windows APIs so that only the requested events are returned.