Brewing in Beats: Kafka output

Welcome to Weekly Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.

Early last week we released Beats 1.1.0 together with the first version of Winlogbeat. You can read more details about it on the blog post.

Other notable things going on:

Add support for Redis and Kafka outputs

Currently you can configure Elasticsearch, Logstash and Redis as outputs. Redis became deprecated after we added support for Logstash as you can simply use the redis output plugin to dump your data to Redis. What happened was that a lot of people deployed it like this:

Beats -> Logstash -> Redis -> Logstash -> Elasticsearch

We received many requests from the community to add support for more outputs, besides Elasticsearch and Logstash to avoid having an additional Logstash instance to translate the data to the desired queuing system. After internal discussions, we decided to follow the community feedback, so we will un-deprecate the Redis output and add support for the Kafka output. So the deployment scenario becomes:

Beats -> Redis/Kafka -> Logstash -> Elasticsearch

More details can be found in the GitHub meta issue.

Merge together the scripts for generating docs and template

Each Beat had two scripts to generate the Elasticsearch template and the documentation based on the fields.yml. The fields.yml file contains details about all exported fields and it is located in each Beat repository.

The scripts were sharing a lot of common code among all the Beats, so we decided to merge them together into two generic scripts that are able to generate the template and the documentation no matter what the Beat is.

The scripts are now part of libbeat and they can also be used by the community Beats.