Brewing in Beats: Configurable Kafka partitioning strategies

Welcome to Brewing in Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.

Kafka output: configurable partitioning strategies

The Kafka output in libbeat now implements several strategies for selecting the Kafka partition: hash, round robin, random. The hash partitioner works on a key configured via the string formatter (i.e. can be a field or a combination of fields).

Long running Filebeat tests

To detect potential issues with long running filebeat instances, a test set up was created. Metricbeat is used to monitor Filebeat and report the open file handlers from the operating system to Elasticsearch.

Log total internal metrics values on shutdown

The Beats now dump their internal metrics on shutdown at INFO level. This completes the effort of reducing the log verbosity on INFO/WARN while still providing the needed information for troubleshooting.

Docs on Metricbeat container support

Besides improving the documentation on cgroups, this includes a guide on how to run Metricbeat in a container to monitor the other containers and the services they run.

Packetbeat: IPv6 support for matching processes

When running on the same host as the monitored application, Packetbeat can correlate the network traffic with the process that created the traffic based on data in the proc file system. This PR, coming from a community contribution by @raboof, makes this work also when IPv6 addresses are involved.

Packetbeat: display IP addresses when listing the devices

Via another community contribution, this time by @noricube, the -devices flag from Packetbeat now also lists the IP addresses. This makes it easier to figure out which device is which on Windows.

Sample dashboards import/export improvements

We’ve got several small fixes and improvements for the scripts that we use to save, package, and load the sample dashboards.

Fix panic on empty host array

This was affecting the libbeat Logstash output, in the master version only.