Brewing in Beats: JSON support in Filebeat
Welcome to Brewing in Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.
JSON support in Filebeat
Since merging this PR last week, Filebeat can natively decode JSON objects from log lines. This is useful for structured logging, where the logging library writes the metadata directly formatted as JSON. Of course, this was already possible with Logstash, but this enables people to take the direct Filebeat -> Elasticsearch path when they have their logs in JSON already.
Another interesting use case for the JSON decoding is that it can be used to ship the logs from a Docker host. When writing the logs to files, Docker wraps the log lines of the application in JSON to add some meta-data. Because Filebeat decodes the JSON before applying line filtering and multiline rules, it is able to unwrap the JSON and then apply these rules, so these features combine well in the Docker and other similar use cases.
Apachebeat and Redisbeat merged into Metricbeat
Radovan Ondas, the creator of Apachebeat, and Chris Black, the creator of Redisbeat, have contributed modules for Apache and Redis to Metricbeat. It’s a great sign for Metricbeat that the Beats community devs are embracing it.
Speaking of Metricbeat, Nicolas continued to shape it over the last few weeks, adding a developer guide, a common way of handling timeouts, adding host metadata, restructuring Topbeat so its functionality can be shared in Metricbeat, adding MySQL authentication, and many others.
New 5.0 Elasticsearch templates
After making sure that Elasticsearch 5.0 is able to upgrade automatically the mapping templates used by the Beats, Adrien opened a PR to upgrade our templates with the the options accepted by 5.0. These are the templates that we will ship with Beats 5.0-alpha1.