Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.
UPDATE: We have decided to promote the auditd module in Metricbeat to be its own Beat: Auditbeat. Please read the blog post for more details.
Auditd alternative in Metricbeat, simpler to use
Auditd is the userspace component to the Linux Auditing System. It can audit every syscall in the Linux Kernel, and write audit records to log files. For example, it writes a log line every time a user accesses a file, starts a process, or connects to a socket. It’s part of the Linux kernel starting with 2.6.14. The power of auditd is that by inspecting the audit log files, you can track security-relevant events, detect misuse or unauthorized activities.
The auditd log format is a bit challenging to parse, but the auditd Filebeat module (available starting with Filebeat 5.4) parses the individual log lines for you.
Another issue with Auditd is that can generate a lot of data, so you need to be specific on what you look for in order to make use of it. This is partly because the Linux Audit Framework sends multiple messages for a single auditable event. For example, a rename
syscall causes the kernel to sent eight separate messages. Each message describes a different aspect of the activity that is occurring (the syscall itself, file paths, current working directory, process title). To be more valuable, these events need to be correlated together.
So, we decided to create our own version of auditd
that is easier to use and better integrated with the Elastic stack. This pull request creates the audit module in Metricbeat. It establishes a subscription to the kernel to receive the events as they occur. Messages for one event can be interleaved with messages from another event. The audit module buffers the messages in order to combine related messages into a single event even if they arrive interleaved or out of order.
The Linux kernel only supports a single subscriber to the audit events so the audit Metricbeat module cannot be used simultaneously with a service like auditd
. Auditd
should be disabled if the audit module is being used.
The audit module is based on go-libaudit library, developed by us. It comes with two sample applications: audit and auparse. Audit registers to receive audit events from the kernel and outputs the data it receives to stdout. Auparse parses the log files from the Linux auditd process or the output of the audit example command and it combines related log messages that are a part of the same event.
The audit module is not complete yet. Currently, you must manually install audit rules using auditctl
. This will be part of the next 6.0 release.
Filebeat: Add option to force end a multiline event
This PR, submitted by @TheoAndersen, adds a new option to the multiline configuration, called flush_pattern
, which receives a regular expression to match against the input lines. When there is a match, the current multiline event will be ended. This is useful for using multiline to capture events with “start” and “end” lines. For example:
multiline: pattern: 'start' negate: true match: after flush_pattern: 'end'
Elasticsearch mapping settings via the Beats configuration files
As we’re now generating the mapping template at runtime in Beats, we can also allow for simple tuning via the Beats configuration file. With the pull request, it’s possible to write:
setup.template.settings: index.number_of_shards: 1 index.number_of_replicas: 1
This will be part of the 6.0 release.
Other changes
Repository: elastic/beats
Affecting all Beats
Changes in master:
- Fixing configuration documentation for kubernetes processor #4313
- Set path.home in system tests #4301
- Remove OS specific error message check from mockbeat #4267
- Update CHANGELOG with missing tasks #4255
Filebeat
Changes in 5.x:
- Deprecate document_type in filebeat 5.5 #4225
Changes in 5.4:
- Fix mysql slowlog parser #4183
Changes in master:
- Fix race condition for harvester Start / Stop in registry #4314
- Fix flaky tests in filebeat #4296
- Rename input_type to type in config and input_type to prospector.type in event #4294
- Extract harvester registry #4292
- Wait for the test index to be deleted #4276
- Move prospector log to its own package #4273
- Use shorter filename in Filebeat test for Windows #4271
- First step to move all code which is Log prospector specific to the Log prospector #4232
- Remove document_type from Filebeat #4204
- Move json_error to error.message and error.type #4167
Metricbeat
Changes in 5.x:
- Fix MongoDB dbstats fields mapping #4025
Changes in master:
- Fix process summary test case #4308
- Follow up on comments from #4231 #4305
- Fix for flaky docker tests #4298
- Perfmon metricset add real data example #4287
- Fix 2x and 5x by skipping Kibana #4286
- Add filesystem name to test error message #4272
- Fix Metricbeat module debug statement #4264
- Add process summary metrics #4231
- Add data.json files for vsphere module. WIP #4209
- Move elasticsearch node_stats metricset under node.stats namespace #4142
- Bug: Fix massive values for cpu metricset for docker module #3682
Packetbeat
Changes in 5.x:
- Clean geoip.paths before using the path #4306
Winlogbeat
Changes in 5.x:
- Fix Winlogbeat test by checking full hostname #3942
Changes in 5.4:
- Fallback on LevelRaw If the Level is not in the RenderingInfo section of the event #4257
Infrastructure
Changes in 5.x:
- Use .go-version to specify the Go version for all CI builds #4303
Changes in master:
- Use .go-version to specify the Go version for all CI builds #4303
- Run make fmt inside python env #4293
- Stop environment #4283
- Use docker-compose down to remove networks #4279
- Add commit id to build environment #4275
- Add 5x build for cross version testing #4259
Processors
Changes in master:
- Add format option for add_locale processor #4106
Documentation
Changes in master:
- Fix developer guide _meta/beat.yml instead of etc/beat.yml #4300
- Small fixes for the HTTP module docs #4297
- Fix multiline docs issues #4288
- Fix metricbeat audit docs #4278
- Fix repositories in docs #4266
- Fail build if one of the docs builds fail #4262
- Fix broken doc build #4260
- Add more info about flows to the doc #4197
Changes in 5.4:
Dashboards
Changes in master:
- Adding units to the disk io derivative to scale properly #4261
Repository: elastic/gosigar
Changes in master:
- Use MemAvailable when it's available on Linux 3.14+ kernel #71