Brewing in Beats: Monitor Kafka logs with Filebeat

Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

Filebeat: Kafka module

Based on an older blog post, this pull request adds a new module in Filebeat for monitoring the Kafka logs. The module collects and parses the Kafka logs, extracts the most interesting information to store in Elasticsearch. The Kafka module was tested with logs from versions 2.11. It comes with a sample dashboard:

filebeat-kafka-logs-overview.png

This feature is currently scheduled to be released with 6.1.

Custom template pattern

In Beats we allow the user to specify a custom index naming scheme (e.g. mybeat-%{[beat.version]}-%{+yyyy.MM.dd}). It is important, however, that the mapping template is still applied when the index name is changed. This PR gives more flexibility over specifying the index pattern (e.g. mybeat-%{[beat.version]}-*) in the template, by supporting the same syntax. With this change, the Beat also refuses to start if the indexed is changed but the template is not, to catch common configuration errors. This change will be included in 6.0.

Repository: elastic/beats

Affecting all Beats

Changes in master:

  • Add support for init contianers in add_kubernetes_metadata #4890
  • Register kubernetes field_format matcher and remove logger in Encode API #4888

Changes in 6.0:

  • Register kubernetes field_format matcher and remove logger in Encode API #4888
  • Remove setup deprecation message #4889
  • Fix go plugins not loaded when beat starts #4799
Metricbeat

Changes in master:

  • Remove mar.local queries from dashboards #4953
  • Add 5.x rabbitmq dashboards #4912
  • Cleanup iostats field naming #4900

Changes in 6.0:

  • Remove mar.local queries from dashboards #4953
  • Fix missing MongoDB config from the modules.d folder #4870
  • Fix MySQL CRUD expression #4857
Packetbeat

Changes in master:

  • Fix flow timestamp update #4955
Filebeat

Changes in master:

  • Fix PostgreSQL dashboards #4905
Infrastructure

Changes in master:

  • Update latest testing environment to 6.0.0-beta1 #4911
Documentation

Changes in master:

  • Let docs build fail if not index.asciidoc found #4945
  • [Docs] Add missing review changes #4941
  • [Docs] Add udp prospector to list of types #4940
  • [Docs] Add module config reload docs #4930
  • Update and reorganize system module config docs #4909
  • Add filesystem.ignore_types to ignore filesystem types #4823
  • Beats testing guide #4866
  • Update getting started to include info about credentials and new command syntax #4841
  • Add docs for new subcommands #4832

Changes in 5.5:

  • Bump docs version for 5.5.2 #4932
  • Close changelog for 5.5.2 #4908

Changes in 5.6:

  • Add filesystem.ignore_types to ignore filesystem types #4823

Changes in 6.0:

  • Backport to 6.0: Config reload docs #4942
  • Add filesystem.ignore_types to ignore filesystem types #4823
  • Update and reorganize system module config docs #4909
  • Backport to 6.0: Add docs for new subcommands (#4832) and update getting started (#4841) #4910
  • Backport to 6.0: #4760 and #4829 #4897
  • Remove settings from docs #4753
New community Beats

Changes in master:

  • Add kafkabeat to community beats doc #4935