Brewing in Beats: New sample dashboards for Docker and Redis

Welcome to Brewing in Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.

New sample Kibana dashboards

We added a sample Kibana dashboard for flows data in Packetbeat, and also sample Kibana dashboards for the Redis and Docker modules in Metricbeat. 

Available on Homebrew

Metricbeat is part of Homebrew now, and we are working to make Filebeat and Packetbeat part of Homebrew as well.

Metricbeat: Hide username/password

If the username and the password were part of the DSN in the hosts configuration option of MongoDB module, they were exported as part of metricset.host field. To fix the issue, we  added two separate configuration options for username and password to the MongoDB module in Metricbeat.

A similar change was done for the PostgreSQL module in Metricbeat.

Define exported fields per module

We define for each Beat the type and the format of each field that is exported under fields.yml. This information is used to not only generate the Elasticsearch index template, but also to generate the index pattern, that lets Kibana know how to format a certain field.

With the growing number of supported protocols in Packetbeat, it grows also the number of fields that Packetbeat exports, and also the length of the fields.yml file. To make it easier for the user to understand and add new fields into fields.yml, we decided to split the fields per protocol under protos/{PROTO}/_meta/fields.yml, and use make collect to group all the fields.yml from all the modules into a single fields.yml per Beat.

Metricbeat has a similar approach, and splits the fields per module under {MODULE}/_meta/fields.yml.

In addition, we also changed that the new fields, generated by a processor, to be defined under libbeat/processors/{PROCESSOR}/_meta/fields.yml.

Libbeat: Fix console output on Windows

Enabling console output was failing on Windows systems, due to a check that stdout is actually available. We fixed the issue by disabling the check on Windows systems.

Filebeat: State handling change for ignore_older

If a file is falling under ignore_older during startup, offset is now set to end of file instead of 0. With the previous logic the whole file was sent in case a line was added and it was inconsistent with files which were harvested previously.