Brewing in Beats: File integrity monitoring with Auditbeat

Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

Auditbeat: File integrity monitoring

With this PR, Auditbeat gets support for monitoring file and directory changes. When a file is changed, it will calculate the MD5, SHA1, and SHA256 hashes for the file. This functionality uses inotify (linux), fsevents (macos), and ReadDirectoryChangesW (windows) to watch for changes. The hashes can be compared against known malicious files like it is done in this blog post about detecting WannaCry.

The configuration looks like this:

- module: audit
  metricsets: [file]
    - /usr/bin
    - /usr/sbin

This functionality will be included in 6.0.0-beta1.

Metricbeat: Aerospike module

Thanks to a community contribution by @alexshadow007, Metricbeat now has a module for monitoring the Aerospike database. Aerospike is a distributed NoSQL database.

This module will be released with 6.0.0-beta1.

JSON logging

It is now possible for the Beats to write their log files in the JSON format. This is following our own advice because we generally recommend writing the logs in JSON if you can control the format. Our JSON logs currently have only a few standard keys (e.g. timestamp, level, message), but we plan to extend it in the future. This functionality will be included in 6.0.0-beta1, for now off by default.

Load dashboards using the Kibana API

In the 5.x versions and older, Beats are importing the Kibana dashboards directly in Elasticsearch, in the Kibana index (.kibana). Every time the Beats are importing the dashboards, they are overwriting the .kibana index, which is not ideal, especially because Kibana uses the index to store its settings, along with the saved objects.

With the removal of types in Elasticsearch, Kibana team is planning to change the format of the .kibana index in 6.0.0-beta1. As the change breaks the way Beats are importing the Kibana dashboards into Elasticsearch, and it requires the creation of new dashboards, we thought it’s a good time to make the dashboard loading more robust.

Starting with this PR, we store two versions of the dashboards, one for 6.0 and above, and one for 5.x and below. At load time, the Beat checks the Elasticsearch/Kibana version and depending on that either uses a Kibana API to load the dashboards or insert in the .kibana index as before.

This functionality will be present in 6.0.0-beta1.

Other changes

Repository: elastic/beats

Affecting all Beats

Changes in master:

  • Update to latest k8s version #4580
  • Do not override pre-existing flag values when setting default #4561
  • Move processors into new pipeline #4554

Changes in master:

  • Add "action" to auditbeat fields.yml #4575
  • Add missing arg to logp.Warn #4572

Changes in 5.x:

  • Filebeat modules: Machine learning jobs #4506

Changes in master:

  • Add experimental flag to udp prospector type #4573
  • scan older files to be harvested based on mod time #4374

Changes in master:

  • Update screenshot for the system module #4589
  • Aerospike module #4560

Changes in master:

  • Update dashboards for Packetbeat, Heartbeat and Winlogbeat #4594
  • Convert heartbeat to new publisher pipeline #4591

Changes in master:

  • Reduce verbosity of make #4583
  • Fix make tests command #4578
  • Add support for analyzers and multifields #4574
  • Rename .full.yml to .reference.yml #4563

Changes in master:

  • Added stub docs for Auditbeat #4565

Changes in 5.4:

  • Bump docs version to 5.4.3 #4571

Changes in master:

  • Fix the path of the Kibana dashboards 6.x #4602

Changes in master:

  • Rename heartbeat packages #4601
  • Remove import_dashboards from packaging #4586

Repository: elastic/go-libaudit

Changes in master:

  • Add SetFailure to AuditClient #8