Brewing in Beats: Auditbeat, for Linux auditing
Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.
Auditbeat - The new Beat on the block
If you have been following along, you know that Andrew has been working for a while on harnessing the auditing framework of the Linux kernel, creating among other things a Metricbeat module that could be used as an alternative for the old auditd daemon. We have now decided to double down on this approach and to promote this module to be its own Beat: Auditbeat.
Making it a Beat, it brings more freedom in extensibility and configurability, as well as giving more exposure.
The first version of Auditbeat will be released as Beta in 6.0.
Filebeat modules: Machine Learning jobs
This adds support for loading Xpack Machine Learning job configurations from the Filebeat modules. An example ML configuration is added to the Nginx Filebeat module. This sample applies ML anomaly detection on the response codes rates.
If a job configuration with the same ID exists, it is not overwritten, because deleting jobs could potentially delete user data. The user should manually delete the jobs in the UI if they want to upgrade.
We currently plan to include this in 5.6 and 6.0.
Libbeat publisher refactoring
We have merged the first part of the libbeat publisher/outputs refactoring. This is a large change without visible effects, but it was required for implementing features like spooling to disk, or dynamically reloadable outputs. It simplifies the handling of batches by buffering them in a single place, which should make the Beats easier to configure and tune for performance.
This refactoring introduces a limitation that we didn’t have before: only one output can be active at a given time. Prior to this, it was possible to have multiple outputs enabled, but only if they had different types (e.g. one ES and LS output worked, but two ES outputs didn’t).
Add more Beat commands
This adds three more Beat commands:
$ filebeat completion bash
Outputs a Bash auto-complete file. This command is hidden, meaning that it doesn’t show in the output of filebeat help.
$ filebeat export template
Exports the generated Elasticsearch mapping template file.
$ filebeat export config
Exports the evaluated configuration in YAML format. Useful for troubleshooting configuration issues.
Other changes:
Repository: elastic/beats
Affecting all Beats
Changes in 5.4:
- Fix typo in docs #4542
Changes in master:
- Reduce verbosity of some make commands #4552
- Do not execute start / stop runners if no runners #4549
- Add umask to generator travis to fix system tests #4531
- Improve `add_docker_metadata` examples for their common cases #4526
- Enhance config.Namespace #4339
Filebeat
Changes in master:
- Enable Cgo for Filebeat & Heartbeat #4546
Metricbeat
Changes in master:
- Add normalized CPU values and number of cores #4553
- Alphabetize system metricsets #4551
- Scale system.cpu.*.pct metrics by the number of cores #4544
Changes in 5.5:
- Scale system.cpu.*.pct metrics by the number of cores #4544
Heartbeat
Changes in master:
- Add Fields and Tag to monitor config #4141
Documentation
Changes in 5.4:
- Close changelog for 5.4.2 #4527
Changes in master:
- Fix path.conf in docs to path.config #4547
- Fixed vsphere module documentation so example config works #4541
- Remove manual beats input plugin installation steps #4530
Changes in 5.5:
- Close changelog for 5.5.0 #4528
- Document complex object support in env vars #4200,
- Add docs for Docker images #4312,
- Provide more detail on log levels #4405,
- Add simple examples that illustrate multiline settings #4408,
- Minor: add link to flows doc #4505
Infrastructure
Changes in master: