Brewing in Beats: JSON support in Filebeat

Welcome to Brewing in Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.

JSON support in Filebeat

Since merging this PR last week, Filebeat can natively decode JSON objects from log lines. This is useful for structured logging, where the logging library writes the metadata directly formatted as JSON. Of course, this was already possible with Logstash, but  this enables people to take the direct Filebeat -> Elasticsearch path when they have their logs in JSON already.

Another interesting use case for the JSON decoding is that it can be used to ship the logs from a Docker host. When writing the logs to files, Docker wraps the log lines of the application in JSON to add some meta-data. Because Filebeat decodes the JSON before applying line filtering and multiline rules, it is able to unwrap the JSON and then apply these rules, so these features combine well in the Docker and other similar use cases.

Apachebeat and Redisbeat merged into Metricbeat

Radovan Ondas, the creator of Apachebeat, and Chris Black, the creator of Redisbeat, have contributed modules for Apache and Redis to Metricbeat. It’s a great sign for Metricbeat that the Beats community devs are embracing it.

Metricbeat progress

Speaking of Metricbeat, Nicolas continued to shape it over the last few weeks, adding a developer guide, a common way of handling timeouts, adding host metadata, restructuring Topbeat so its functionality can be shared in Metricbeat, adding MySQL authentication, and many others.

New 5.0 Elasticsearch templates

After making sure that Elasticsearch 5.0 is able to upgrade automatically the mapping templates used by the Beats, Adrien opened a PR to upgrade our templates with the the options accepted by 5.0. These are the templates that we will ship with Beats 5.0-alpha1.

Winlogbeat - select events by level, event_id, and provider

Winlogbeat is now able to filter the events by these key fields. It does this efficiently by adjusting its Windows API query to only return the events needed. The details are in the PR.

Packetbeat - Split real_ip_header to only have one value

This fixes an issue, where Packetbeat’s Geoip resolving didn’t work if the X-Forwarder-For (or similar) header had multiple IP addresses inside. With this fix, Packetbeat takes the first IP address when there are multiple defined in the header.  The  fix will be available in 1.2.