Brewing in Beats: Conditional Filtering

Welcome to Brewing in Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.

Last week we have released Beats 1.2.3 containing a few important bug fixes. Besides that, we have worked on a set of improvements that we plan to release with 5.0.0-alpha3:

Conditionals for generic filtering

It is now possible to use conditionals when filtering out fields and events in libbeat. This makes it really easy to remove unneeded data even before sending it over the network. Here is an example filter that drops the HTTP header fields if the code is 200 and the status is OK:

- drop_fields:
    fields: ["http.request_headers", "http.response_headers"]
    equals:
      http.code: 200
      status: OK     

Metricbeat gets an Nginx module

KS Chan, the person behind Nginxbeat, has contributed an Nginx module to Metricbeat.

Metricbeat progress

As anticipated last week, the work has started to normalize the metric names so that users get an uniform experience regardless of the system they monitor. For example, here is the PR for Redis and here is the one for Apache.

Other Metricbeat improvements include:

Configuration refactoring

The work on reorganizing the options in our configuration files is almost finished. We now distribute two versions of the configuration files: the default one which contains only the most common options and a “full” one, that contains all non-deprecated options with longer comments. The full versions can be almost used as a reference and it is easy to copy and paste between them.

Another configuration related improvement is that all duration options now accept the same specifiers (e.g 10s, 1ms). This change was done in a backwards compatible way.

Static builds

We now produce statically linked binaries for Filebeat and Winlogbeat (which don't use CGo) by using simple cross compilation, without Cgo enabled. The advantage of the statically linked binaries is that they tend to be more portable. The Beats that use CGo continue to be dynamically linked against libc and not much else.

Package names

We have updated our artifact naming to be consistent with the other Elastic stack projects.

Filebeat to Logstash performance

The work on figuring out why the Filebeat to Logstash communication is slower than expected is now resumed. The current intention is to rewrite the beats input plugin in Logstash to use Netty. It’s really early, but there are some very promising results already.