Brewing in Beats: Add support for new Kafka versions

Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

Update sarama (Kafka) library

We have updated the client library that we use in the Kafka output. This adds compatibility with newer versions of Kafka (between 0.8.2.0 and 0.11.0.0) and a lz4 compression option. The updated client library is in our 6.0 branch and will be released with 6.0.0-beta2.

Metricbeat: Total CPU usage metrics

This PR introduces two new metrics in cpu metricset of the system module: system.cpu.total.pct and system.cpu.total.norm.pct. While these metrics could be computed based on other existing metrics, they are so commonly needed that it makes sense to export them directly. They will be available in 6.0.0-beta2.

Filebeat: Add registry for prospectors

With this PR, the prospectors in Filebeat use the same module pattern as we have in Metricbeat and other places, making it easy to create custom Beats that use Filebeat as a framework and add new prospector types.

New community Beat: Cloudfrontbeat

Created by @jarl-tornroos Cloudfrontbeat reads log events from Amazon Web Services Cloudfront and publishes them to Elasticsearch for further analysis.

Other changes:

Repository: elastic/beats

Affecting all Beats

Changes in master:

  • add_kubernetes_metadata processor: add support for "/var/log/containers/" log path #4981
  • Event ACK callbacks only report the events private field #4978
  • Add option to enable/disable LS output slow start #4972

Changes in 6.0:

  • Add support for init contianers in add_kubernetes_metadata #4890
  • Change deprecated _default_ mapping to doc in index templates #4864
  • Event ACK callbacks only report the events private field #4978
  • Add option to enable/disable LS output slow start #4972
  • Pipeline cleanups #4776
  • Add more info for each option under setup.template.settings #4792
  • Adding support to exclude labels from kubernetes pod metadata #4757
  • Allow template pattern to be overwritten #4769
Metricbeat

Changes in master:

  • Add ceph cluster status to metricbeat #4990

Changes in 6.0:

  • Adds a "type" field to the filesystem beat #4717
Packetbeat

Changes in master:

  • Adapt packetbeat tcp protocol generator to new publisher pipeline #4956
  • Delay device opening #4939

Changes in 6.0:

  • Delay device opening #4939
  • Adapt packetbeat tcp protocol generator to new publisher pipeline #4956
  • Fix flow timestamp update #4955
Dashboards

Changes in master:

  • Fixes and renamings for Packetbeat dashboards #5008
  • Metricbeat modules: rename dashboards and visualizations #4979 #5015
  • Filebeat modules: Rename dashboards and visualization #4952
  • Change index name in the dashboards and index-pattern #4949
  • Add 5.x rabbitmq dashboards #4912

Changes in 6.0:

  • Change index name in the dashboards and index-pattern #4949
  • Fixes and renamings for Packetbeat dashboards #5008
  • Metricbeat modules: rename dashboards and visualizations #4979 #5015
  • Filebeat modules: Rename dashboards and visualization #4952
  • Add 5.x rabbitmq dashboards #4912
Infrastructure

Changes in master:

  • closes #4837 Travis errors on chmod command #4936