Brewing in Beats: Kafka module in Metricbeat

Welcome to Brewing in Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.

New community Beat: Consulbeat

This new Beat collects service health metrics from Consul and pushes them to the Elastic stack.

Metricbeat Kafka module (in progress)

Not merged yet, but Nicolas has made good progress on adding support for monitoring Kafka with Metricbeat. The work is partially inspired by Kafkabeat, so hat tip to Dale.

Fix: Remove duplicate fields from the index patterns

We had an issue where several fields, including @timestamp, were showing up multiple times in Kibana. This happened because we build up the Elasticsearch mapping template and the Kibana index pattern from several fragments, so some duplicates sneaked up on us. We’ve added code to automatically check if the same field is accidentally defined in multiple places, so this shouldn’t happen again.

Filebeat: tail_files is applied only once at start

A common problem with using the tail_files option was that people used it to skip old files but, because it was applied on any new file, it could easily drop the first few lines from each file. We’ve now changed the behaviour to only apply the tail_files option at the Filebeat start.

Remove intermediary files during generation

We’ve got a lot of auto-generated files in the repository, to the point that it is hard for occasional contributors, and sometimes even ourselves, to figure out where things need to be changed. So we’re looking to reduce the number of auto-generated files that we commit in git, and this PR takes a first step and removes the intermediary fields.yml files used for generating the templates, docs, and index patterns.

Packetbeat code cleanup

Steffen continued his work on applying coding standards to the older parts of our code base, with a massive PR against Packetbeat, mostly renaming things and reducing the exported fields.

Godocs for go-ucfg

Go-ucfg is our fairly advanced (we worry about it becoming self aware soon) library for handling configuration files. For the moment it just got better docs.