Brewing in Beats: Monitor Kafka logs with Filebeat
Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.
Filebeat: Kafka module
Based on an older blog post, this pull request adds a new module in Filebeat for monitoring the Kafka logs. The module collects and parses the Kafka logs, extracts the most interesting information to store in Elasticsearch. The Kafka module was tested with logs from versions 2.11. It comes with a sample dashboard:
This feature is currently scheduled to be released with 6.1.
Custom template pattern
In Beats we allow the user to specify a custom index naming scheme (e.g. mybeat-%{[beat.version]}-%{+yyyy.MM.dd}
). It is important, however, that the mapping template is still applied when the index name is changed. This PR gives more flexibility over specifying the index pattern (e.g. mybeat-%{[beat.version]}-*
) in the template, by supporting the same syntax. With this change, the Beat also refuses to start if the indexed is changed but the template is not, to catch common configuration errors. This change will be included in 6.0.
Repository: elastic/beats
Affecting all Beats
Changes in master:
- Add support for init contianers in
add_kubernetes_metadata
#4890 - Register kubernetes field_format matcher and remove logger in Encode API #4888
Changes in 6.0:
- Register kubernetes field_format matcher and remove logger in Encode API #4888
- Remove setup deprecation message #4889
- Fix go plugins not loaded when beat starts #4799
Metricbeat
Changes in master:
- Remove mar.local queries from dashboards #4953
- Add 5.x rabbitmq dashboards #4912
- Cleanup iostats field naming #4900
Changes in 6.0:
- Remove mar.local queries from dashboards #4953
- Fix missing MongoDB config from the
modules.d
folder #4870 - Fix MySQL CRUD expression #4857
Packetbeat
Changes in master:
- Fix flow timestamp update #4955
Filebeat
Changes in master:
- Fix PostgreSQL dashboards #4905
Infrastructure
Changes in master:
- Update latest testing environment to 6.0.0-beta1 #4911
Documentation
Changes in master:
- Let docs build fail if not index.asciidoc found #4945
- [Docs] Add missing review changes #4941
- [Docs] Add udp prospector to list of types #4940
- [Docs] Add module config reload docs #4930
- Update and reorganize system module config docs #4909
- Add filesystem.ignore_types to ignore filesystem types #4823
- Beats testing guide #4866
- Update getting started to include info about credentials and new command syntax #4841
- Add docs for new subcommands #4832
Changes in 5.5:
Changes in 5.6:
- Add filesystem.ignore_types to ignore filesystem types #4823
Changes in 6.0:
- Backport to 6.0: Config reload docs #4942
- Add filesystem.ignore_types to ignore filesystem types #4823
- Update and reorganize system module config docs #4909
- Backport to 6.0: Add docs for new subcommands (#4832) and update getting started (#4841) #4910
- Backport to 6.0: #4760 and #4829 #4897
- Remove settings from docs #4753
New community Beats
Changes in master:
- Add kafkabeat to community beats doc #4935