Brewing in Beats: Auditd alternative in Metricbeat

By
Monica Sarbu

Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

UPDATE: We have decided to promote the auditd module in Metricbeat to be its own Beat: Auditbeat. Please read the blog post for more details.

Auditd alternative in Metricbeat, simpler to use

Auditd is the userspace component to the Linux Auditing System. It can audit every syscall in the Linux Kernel, and write audit records to log files. For example, it writes a log line every time a user accesses a file, starts a process, or connects to a socket. It’s part of the Linux kernel starting with 2.6.14.  The power of auditd is that by inspecting the audit log files, you can track security-relevant events, detect misuse or unauthorized activities.

The auditd log format is a bit challenging to parse, but the auditd Filebeat module (available starting with Filebeat 5.4) parses the individual log lines for you.

Another issue with Auditd is that can generate a lot of data, so you need to be specific on what you look for in order to make use of it. This is partly because the Linux Audit Framework sends multiple messages for a single auditable event. For example, a rename syscall causes the kernel to sent eight separate messages. Each message describes a different aspect of the activity that is occurring (the syscall itself, file paths, current working directory, process title). To be more valuable, these events need to be correlated together.

So, we decided to create our own version of auditd that is easier to use and better integrated with the Elastic stack. This pull request creates the audit module in Metricbeat. It establishes a subscription to the kernel to receive the events as they occur. Messages for one event can be interleaved with messages from another event. The audit module buffers the messages in order to combine related messages into a single event even if they arrive interleaved or out of order.

The Linux kernel only supports a single subscriber to the audit events so the audit Metricbeat module cannot be used simultaneously with a service like auditd. Auditd should be disabled if the audit module is being used.  

The audit module is based on go-libaudit library, developed by us. It comes with two sample applications: audit and auparse. Audit registers to receive audit events from the kernel and outputs the data it receives to stdout. Auparse parses the log files from the Linux auditd process or the output of the audit example command and it combines related log messages that are a part of the same event.

The audit module is not complete yet. Currently, you must manually install audit rules using auditctl. This will be part of the next 6.0 release.

Filebeat: Add option to force end a multiline event

This PR, submitted by @TheoAndersen, adds a new option to the multiline configuration, called flush_pattern, which receives a regular expression to match against the input lines. When there is a match, the current multiline event will be ended. This is useful for using multiline to capture events with “start” and “end” lines. For example:

multiline:
    pattern: 'start'
    negate: true
    match: after
    flush_pattern: 'end'

Elasticsearch mapping settings via the Beats configuration files

As we’re now generating the mapping template at runtime in Beats, we can also allow for simple tuning via the Beats configuration file. With the pull request, it’s possible to write:

setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1

This will be part of the 6.0 release.

Other changes

Repository: elastic/beats

Affecting all Beats

Changes in master:

  • Fixing configuration documentation for kubernetes processor #4313
  • Set path.home in system tests #4301
  • Remove OS specific error message check from mockbeat #4267
  • Update CHANGELOG with missing tasks #4255
Filebeat

Changes in 5.x:

  • Deprecate document_type in filebeat 5.5 #4225

Changes in 5.4:

  • Fix mysql slowlog parser #4183

Changes in master:

  • Fix race condition for harvester Start / Stop in registry #4314
  • Fix flaky tests in filebeat #4296
  • Rename input_type to type in config and input_type to prospector.type in event #4294
  • Extract harvester registry #4292
  • Wait for the test index to be deleted #4276
  • Move prospector log to its own package #4273
  • Use shorter filename in Filebeat test for Windows #4271
  • First step to move all code which is Log prospector specific to the Log prospector #4232
  • Remove document_type from Filebeat #4204
  • Move json_error to error.message and error.type #4167
Metricbeat

Changes in 5.x:

  • Fix MongoDB dbstats fields mapping #4025

Changes in master:

  • Fix process summary test case #4308
  • Follow up on comments from #4231 #4305
  • Fix for flaky docker tests #4298
  • Perfmon metricset add real data example #4287
  • Fix 2x and 5x by skipping Kibana #4286
  • Add filesystem name to test error message #4272
  • Fix Metricbeat module debug statement #4264
  • Add process summary metrics #4231
  • Add data.json files for vsphere module. WIP #4209
  • Move elasticsearch node_stats metricset under node.stats namespace #4142
  • Bug: Fix massive values for cpu metricset for docker module #3682
Packetbeat

Changes in 5.x:

  • Clean geoip.paths before using the path #4306
Winlogbeat

Changes in 5.x:

  • Fix Winlogbeat test by checking full hostname #3942

Changes in 5.4:

  • Fallback on LevelRaw If the Level is not in the RenderingInfo section of the event #4257
Infrastructure

Changes in 5.x:

  • Use .go-version to specify the Go version for all CI builds #4303

Changes in master:

  • Use .go-version to specify the Go version for all CI builds #4303
  • Run make fmt inside python env #4293
  • Stop environment #4283
  • Use docker-compose down to remove networks #4279
  • Add commit id to build environment #4275
  • Add 5x build for cross version testing #4259
Processors

Changes in master:

  • Add format option for add_locale processor #4106
Documentation

Changes in master:

  • Fix developer guide _meta/beat.yml instead of etc/beat.yml #4300
  • Small fixes for the HTTP module docs #4297
  • Fix multiline docs issues #4288
  • Fix metricbeat audit docs #4278
  • Fix repositories in docs #4266
  • Fail build if one of the docs builds fail #4262
  • Fix broken doc build #4260
  • Add more info about flows to the doc #4197

Changes in 5.4:

  • Close changelog 5.4.1 #4310
  • Backport: #4147 and #4135 to 5.4 #4176
Dashboards

Changes in master:

  • Adding units to the disk io derivative to scale properly #4261

Repository: elastic/gosigar

Changes in master:

  • Use MemAvailable when it's available on Linux 3.14+ kernel #71