Brewing in Beats: Auditbeat, for Linux auditing

Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

Auditbeat - The new Beat on the block

If you have been following along, you know that Andrew has been working for a while on harnessing the auditing framework of the Linux kernel, creating among other things a Metricbeat module that could be used as an alternative for the old auditd daemon. We have now decided to double down on this approach and to promote this module to be its own Beat: Auditbeat.

Making it a Beat, it brings more freedom in extensibility and configurability, as well as giving more exposure. 

The first version of Auditbeat will be released as Beta in 6.0.

Filebeat modules: Machine Learning jobs

This adds support for loading Xpack Machine Learning job configurations from the Filebeat modules. An example ML configuration is added to the Nginx Filebeat module. This sample applies ML anomaly detection on the response codes rates.

If a job configuration with the same ID exists, it is not overwritten, because deleting jobs could potentially delete user data. The user should manually delete the jobs in the UI if they want to upgrade.

We currently plan to include this in 5.6 and 6.0.

Libbeat publisher refactoring

We have merged the first part of the libbeat publisher/outputs refactoring. This is a large change without visible effects, but it was required for implementing features like spooling to disk, or dynamically reloadable outputs. It simplifies the handling of batches by buffering them in a single place, which should make the Beats easier to configure and tune for performance.

This refactoring introduces a limitation that we didn’t have before: only one output can be active at a given time. Prior to this, it was possible to have multiple outputs enabled, but only if they had different types (e.g. one ES and LS output worked, but two ES outputs didn’t).

Add more Beat commands

This adds three more Beat commands:

$ filebeat completion bash

Outputs a Bash auto-complete file. This command is hidden, meaning that it doesn’t show in the output of filebeat help.

$ filebeat export template

Exports the generated Elasticsearch mapping template file.

$ filebeat export config

Exports the evaluated configuration in YAML format. Useful for troubleshooting configuration issues.

Other changes:

Repository: elastic/beats

Affecting all Beats

Changes in 5.4:

Changes in master:

  • Reduce verbosity of some make commands #4552
  • Do not execute start / stop runners if no runners #4549
  • Add umask to generator travis to fix system tests #4531
  • Improve `add_docker_metadata` examples for their common cases #4526
  • Enhance config.Namespace #4339

Changes in master:

  • Enable Cgo for Filebeat & Heartbeat #4546

Changes in master:

  • Add normalized CPU values and number of cores #4553
  • Alphabetize system metricsets #4551
  • Scale system.cpu.*.pct metrics by the number of cores #4544

Changes in 5.5:

  • Scale system.cpu.*.pct metrics by the number of cores #4544

Changes in master:

  • Add Fields and Tag to monitor config #4141

Changes in 5.4:

  • Close changelog for 5.4.2 #4527

Changes in master:

  • Fix path.conf in docs to path.config #4547
  • Fixed vsphere module documentation so example config works #4541
  • Remove manual beats input plugin installation steps #4530

Changes in 5.5:

  • Close changelog for 5.5.0 #4528
  • Document complex object support in env vars #4200,
  • Add docs for Docker images #4312,
  • Provide more detail on log levels #4405,
  • Add simple examples that illustrate multiline settings #4408,
  • Minor: add link to flows doc #4505

Changes in master:

  • Fix generation of object with enabled: false #4536
  • Allow enabled to be used in fields.yml #4532
  • We're hiring

    Work for a global, distributed team where finding someone like you is just a Zoom meeting away. Flexible work with impact? Development opportunities from the start?