August 8, 2017 Releases

Beats 6.0.0-beta1 released

By Monica SarbuTudor Golubenco

Today the Elastic stack 6.0 is leaving the Alpha phase behind and boldly enters the Beta. The Beta1 brings tons of improvements to your favorite Beat, be it Filebeat, Heartbeat, Metricbeat, Packetbeat, or Winlogbeat. Not only that, but we also have a surprise: the first public release of Auditbeat.

Before going through the highlights, let’s get those handy links out of the way:

Say Heya to Auditbeat

If you read the Alpha2 release blog post, you know that we have been working on harnessing the auditing framework of the Linux kernel. The audit framework unlocks tons of valuable data, including detection of short lived processes, network connections, unauthorized file accesses, and so on. Check out the Security @ Slack talk from Elastic{ON} for an inspiring presentation on how the audit data and the Elastic stack are used against attackers.

In Beta1, we’re doubling down on this approach by adding a new Beat to our collection: Auditbeat. Auditbeat works as an alternative to the auditd daemon, but one that is designed to integrate perfectly with the Elastic stack. If you have tried to write parsing rules for the auditd logs, you know it’s not exactly straightforward. Auditbeat not only does that for you, but it also automatically groups related events together in one document, so that everything is ready to visualize in Kibana. Because it’s built on top of libbeat, our mature framework for building Beats, it can send the events via secure and back-pressure sensitive connections to Logstash, Elasticsearch, and Kafka.

auditbeat-file-integrity-dashboard.png

In addition to collecting audit logs from the Linux kernel, Auditbeat also has a file integrity module. It watches files and directories for changes, and when a file changes it computes the MD5, SHA1, and SHA256 hashes and publishes them to Elasticsearch. The hashes can be compared against known malicious files, like it is done in this blog post about detecting WannaCry. This functionality is available on Windows, macOS, and Linux.

New commands and configuration layout

We rethought the way the Metricbeat and Filebeat modules are enabled and configured. Instead of one huge configuration file, we now have a modules.d directory with individual configuration files for each module. The Beats also get commands to list, enable or disable modules. For example:

$ metricbeat modules list
$ metricbeat modules enable redis
$ metricbeat modules disable redis

And these are not the only useful new commands. There are also commands to export the configuration, the Elasticsearch mapping template, to do a test fetch for a Metricbeat module, and even to test the connectivity with Logstash or Elasticsearch. Here is an example run for the test output command:

$ filebeat test output
elasticsearch: https://61fc3e1983ebd7b13e9d6098e4651345.us-east-1.aws.found.io:9243...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 54.235.139.222, 54.221.244.80, 54.243.122.128, 54.204.28.14, 54.221.223.25, 54.243.118.44, 54.243.142.98, 54.235.122.205
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.2
    dial up... OK
  talk to server... OK
  version: 5.5.0

Add Docker metadata to the Docker logs

The add_docker_metadata processor, which we introduce in alpha2, got a little smarter. A little smarter, and a lot more useful. You might know that when using the default logging driver in Docker, the log files are placed under a path like var/lib/docker/containers/<container_id>/*.log. The add_docker_metadata processor can now extract the container ID from the file path and use it to query the Docker API for the metadata.

This makes it really easy to collect and annotate the Docker logs.

Internal pipeline refactoring

6.0.0-beta1 also comes with a refreshed internal pipeline architecture. While this change is mostly internal, meant to simplify and improve the Beats overall architecture, it does have some visible effects.

The new pipeline is asynchronous by default, meaning, for example, that while Filebeat is waiting for a network acknowledgement from Logstash/Elasticsearch, it continues to read and process lines from disk. We expect this to bring an increase in the maximum throughput.

The Filebeat internal spooler is removed, as its functionality is covered by the new pipeline. This means that tuning Filebeat for performance is easier (only one internal queue size to play with). Be aware that this also implies some breaking changes in the configuration file.

Another effect of the pipeline refactoring is that you can no longer enable two outputs at the same time. In previous versions of the Beats, it was possible to enable simultaneously multiple outputs of different types (e.g. one Logstash and on Elasticsearch), but not multiple outputs of the same type. In a back-pressure sensitive shipper, like Filebeat is, having multiple outputs generally means that the slowest output decides the rhythm. This is not ideal and often took our users by surprise. We therefore removed the possibility, which also simplified our internal architecture, and instead we recommend using Logstash for duplicating the events to multiple sinks.

Become a Pioneer

A big “Thank You” to everyone that has tried the alpha releases and posted issues or provided feedback. We’d like to also remind you that if you post a valid, non-duplicate bug report during the alpha/beta period against any of the Elastic stack projects, you are entitled to a special gift package. You can find more details about the Pioneer Program in the Pioneer Program announcement post.