Brewing in Beats: Manage dashboards per Beat

Welcome to Brewing in Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.

Last week we released the long-expected Beats 5.0.0-alpha1 (blog post) and the patch release Beats 1.2.1 (blog post).

The highlights of the week are:

Require braces in the environment variable expansion

Version 1.2.0 introduced the possibility of using environment variables in the configuration file by replacing strings like $NAME or ${NAME} with the value of the NAME environment variable. Together with this great feature, we also introduced a bug that replaces the environment variable when it’s not wanted if the dollar sign shows up in passwords or in regular expressions. This PR restricts to replace the environment variable only when the ${NAME} form is encountered.

Change exported fields for Packetbeat flows

The PR re-organizes the exported fields for flows. The ip, ipv6, port, statistics and the associated location are grouped per source and destination. In the case of flows, the source is considered the one that sends the first SYN packet.

Manage Kibana dashboards per Beat

In the current version, the sample Kibana dashboards are available in a single archive for all the Beats together with a bash and powershell script to load them all in Kibana. In most of the cases, you don’t need all the Kibana dashboards and only the ones for a single Beat.

Last week we adjusted the bash and the powershell scripts to be able to import and export the Kibana dashboards together with visualizations, searches and index patterns to Kibana only for a single Beat. They will be available in elastic/beats/dev-tools with this PR.

Now it’s easier to contribute with Kibana dashboards to the beats project. You can just create your own dashboard in Kibana for any Beat and export all the Beat dashboards together with visualizations, searches and index patterns by using the python script export_dashboards.py 

Customize Discovery page for Packetbeat 

Starting with 5.0.0-alpha1, Packetbeat exports two different types of data: transactions of various protocols and flows. Both are available in the packetbeat-* index pattern, but with different type values depending if it’s a HTTP transaction or a flow. The PR creates two new searches to customize the transactions view and the flows view with the most important fields. They are already available in beats-dashboards-5.0.0-alpha1 and you can start using them by following the steps.

Loading Elasticsearch template is now on by default

With this PR, when the Beat starts the Elasticsearch template is loaded automatically by default, if it was not previously loaded. You can force to overwrite the template by enabling the overwrite option. This change is already available in 5.0.0-alpha1.

If you want to learn more about this, please check the 5.0.0-alpha1 docs.