Brewing in Beats: File integrity monitoring with Auditbeat
Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.
Auditbeat: File integrity monitoring
With this PR, Auditbeat gets support for monitoring file and directory changes. When a file is changed, it will calculate the MD5, SHA1, and SHA256 hashes for the file. This functionality uses inotify (linux), fsevents (macos), and ReadDirectoryChangesW (windows) to watch for changes. The hashes can be compared against known malicious files like it is done in this blog post about detecting WannaCry.
The configuration looks like this:
auditbeat.modules:
- module: audit
metricsets: [file]
file.paths:
binaries:
- /usr/bin
- /usr/sbin
This functionality will be included in 6.0.0-beta1.
Metricbeat: Aerospike module
Thanks to a community contribution by @alexshadow007, Metricbeat now has a module for monitoring the Aerospike database. Aerospike is a distributed NoSQL database.
This module will be released with 6.0.0-beta1.
JSON logging
It is now possible for the Beats to write their log files in the JSON format. This is following our own advice because we generally recommend writing the logs in JSON if you can control the format. Our JSON logs currently have only a few standard keys (e.g. timestamp, level, message), but we plan to extend it in the future. This functionality will be included in 6.0.0-beta1, for now off by default.
Load dashboards using the Kibana API
In the 5.x versions and older, Beats are importing the Kibana dashboards directly in Elasticsearch, in the Kibana index (.kibana). Every time the Beats are importing the dashboards, they are overwriting the .kibana index, which is not ideal, especially because Kibana uses the index to store its settings, along with the saved objects.
With the removal of types in Elasticsearch, Kibana team is planning to change the format of the .kibana index in 6.0.0-beta1. As the change breaks the way Beats are importing the Kibana dashboards into Elasticsearch, and it requires the creation of new dashboards, we thought it’s a good time to make the dashboard loading more robust.
Starting with this PR, we store two versions of the dashboards, one for 6.0 and above, and one for 5.x and below. At load time, the Beat checks the Elasticsearch/Kibana version and depending on that either uses a Kibana API to load the dashboards or insert in the .kibana index as before.
This functionality will be present in 6.0.0-beta1.
Other changes
Repository: elastic/beats
Affecting all Beats
Changes in master:
- Update to latest k8s version #4580
- Do not override pre-existing flag values when setting default #4561
- Move processors into new pipeline #4554
Auditbeat
Changes in master:
Filebeat
Changes in 5.x:
- Filebeat modules: Machine learning jobs #4506
Changes in master:
- Add experimental flag to udp prospector type #4573
- scan older files to be harvested based on mod time #4374
Metricbeat
Changes in master:
Heartbeat
Changes in master:
- Update dashboards for Packetbeat, Heartbeat and Winlogbeat #4594
- Convert heartbeat to new publisher pipeline #4591
Infrastructure
Changes in master:
- Reduce verbosity of make #4583
- Fix make tests command #4578
- Add support for analyzers and multifields #4574
- Rename .full.yml to .reference.yml #4563
Documentation
Changes in master:
- Added stub docs for Auditbeat #4565
Changes in 5.4:
- Bump docs version to 5.4.3 #4571
Dashboards
Changes in master:
- Fix the path of the Kibana dashboards 6.x #4602
Packaging
Changes in master:
Repository: elastic/go-libaudit
Changes in master:
- Add SetFailure to AuditClient #8