Brewing in Beats: Add data path support and enhance Redis output

Welcome to Brewing in Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.

Highlights of the week:

Enhance Redis output support

Last week we were working on improving the Redis output by adding support for failover and loadbalancing if multiple Redis hosts are configured. In addition it adds support for TLS, SOCKS5 and backoff strategy in case Redis is unresponsive.

Data path and default directory layouts

We are improving our default directory layouts (for the tgz/zip case, as well as for DEB/RPM) to be closer to the directory layout used by Elasticsearch. With this occasion, we introduced also the notion of a “data path”, which we use to store the Filebeat/Winlogbeat registry file and we will use in the future for other things, for example for storing a persistent UUID. This new directory layout also simplifies  our packaging, as we require less platform specific changes of the configuration files.

Enhance environment variables replacement

Starting with this pull request you are able to escape the environment variables in the configuration file to avoid replacing the variable. In addition, unterminated environment variable expressions in configuration file will throw an error at startup.

Fix file rotation registrar in Filebeat

When multiple files were rotated at the same time, the first rotation was overwriting the states for the other rotated files. The problem is now fixed by not writing the state to disk every time state changes and only when a scan is done. The downside of this solution is that the registrar is also updated, if no changes did happen.


Merged this week:


Ongoing:

  • Add condition to generic filtering. Follow the status.
  • Load Elasticsearch template on connect. Follow the status.
  • Define local filters for each Metricbeat module in addition to global filters defined per Beat. The advantage is that the local filters are much faster than the global ones as they are applied while creating the event. Follow the status.
  • Add support for properly closing the publisher in libbeat. Follow the status.
  • Create a docbook for Metricbeat. Follow the status.