Beats 6.0.0-alpha2 released
It's time for the second Alpha version of the upcoming 6.0 release of the Elastic stack. The Beats 6.0.0-alpha2 release comes with new features, some rethinking of old features, and bug fixes. This blog post covers the highlights.
Quick links:
Auditd alternative in Metricbeat, simpler to use
Auditd is the userspace component to the Linux Auditing System. It can audit every syscall in the Linux Kernel, and write audit records to log files. For example, it writes a log line every time a user accesses a file, starts a process, or connects to a socket. It’s part of the Linux kernel starting with 2.6.14. The power of auditd is that by inspecting the audit log files, you can track security-relevant events, detect misuse or unauthorized activities.
The auditd log format is a bit challenging to parse, but the auditd Filebeat module (available starting with Filebeat 5.4) parses the individual log lines for you.
Another issue with Auditd is that can generate a lot of data, so you need to be specific on what you look for in order to make use of it. This is partly because the Linux Audit Framework sends multiple messages for a single auditable event. For example, a rename syscall causes the kernel to sent eight separate messages. Each message describes a different aspect of the activity that is occurring (the syscall itself, file paths, current working directory, process title). To be more valuable, these events need to be correlated together.
So, we decided to create our own version of auditd that is easier to use and better integrated with the Elastic stack. For this, we created the audit module in Metricbeat. It establishes a subscription to the kernel to receive the events as they occur. Messages for one event can be interleaved with messages from another event. The audit module buffers the messages in order to combine related messages into a single event even if they arrive interleaved or out of order.
The Linux kernel only supports a single subscriber to the audit events so the audit Metricbeat module cannot be used simultaneously with a service like auditd. Auditd should be disabled if the audit module is being used.
The audit module is based on go-libaudit library, developed by us. It comes with two sample applications: audit and auparse. The audit application registers to receive audit events from the kernel and outputs the data it receives to stdout. The auparse application parses the log files from the Linux auditd process or the output of the audit application and it combines related log messages that are a part of the same event.
To make use of the audit module, you need to install manually the audit rules to the Kernel. The audit rules are where you configure the activities that are audited. These rules are configured as either syscalls or files that should be monitored. For example you can track all connect syscalls or file system writes to /etc/passwd. As a next step, we are planning to have the audit module load automatically the audit rules, to make it easier for the user.
The audit module is a beta feature, and is subject to change.
New processor: add_docker_metadata
Inspired by the add_kubernetes_metadata processor, which we added with Alpha1, the add_docker_metadata processor queries the Docker API to enhance the events with the Docker name, image, and labels. To lookup the metadata, the processor currently needs the container ID to be present in one of the event fields. This is the case, for example, with the cgroups events created by the system module in Metricbeat.
processors:
- add_docker_metadata:
match_fields: ["system.process.cgroup.id"]
host: "unix:///var/run/docker.sock"
TLS configuration for the connection with the Docker server is also possible.
Optimized Metricbeat default configuration
This release introduces a new default metricbeat.yml, one that is better suited for the majority of metrics use cases. The new configuration takes advantage of the top N way of filtering processes, which we introduced in Alpha1, and it also uses the new way of generating the Elasticsearch mapping template to set the number of shards to 1 and to enable the best_compression codec. Finally, the new configuration sets better defaults for the filesystem data collection.
We expect this new default configuration to consume dramatically less Elasticsearch storage per day-host, while delivering almost the same value.
Metricbeat new modules
Just like Alpha1, this release comes with a set of new Metricbeat modules and improvements. Notable are the Vsphere and RabbitMQ modules, both contributed by @amandahla, the MacOS support for the diskio metricset, and several improvements to the Kubernetes module.
Feedback
If you want to make use of the new features added in Beats 6.0.0, please download the latest preview release, install it, and let us know what you think on Twitter (@elastic) or in our forum.