Brewing in Beat: New Beats dashboards management

New community beat: Springbeat

In addition to Logstashbeat, Valentin Crettaz created Springbeat to collect different data from Spring Boot applications running with the actuator module. For now, it supports only health and metrics data. Please give it a try and let us know what do you think.

Packetbeat: Add Cassandra support

With this PR, Medcl adds support for Cassandra in Packetbeat, so you can now monitor the network traffic exchanged between your applications and Cassandra, and visualize it with Kibana. The PR comes with a sample Kibana dashboard for Cassandra traffic that you can use as a starting point for your dashboards.


Easily import/export the Beats dashboards

We were providing two scripts for importing the Kibana dashboards for a single Beat in Elasticsearch, one in bash and one in powershell in order to have support for Unix and Windows systems. Maintaining two scripts became complex with the time, and we decided to re-write the script in Golang that can be compiled for all Go supported platforms.  The script is called import_dashboards, and it will be part of the Beat package. With this change, we are trying to make it easier for the community to share their own custom Kibana dashboards. If you created some awesome Kibana dashboards, just create a zip archive that has a certain structure, and share it with us. Starting with the next release, we will provide the sample Kibana dashboards for all the Elastic Beats in a separate common package, instead of including the dashboards in each Beat package. To import the Kibana dashboards and the index pattern for a single Beat, together with the dependencies, visualizations, searches, you just need to run:

./scripts/import_dashboards -beat Metricbeat

This will download the right version of the Metricbeat dashboards, and import them for you to your local Elasticsearch node. You can also specify a different Elasticsearch URL in `-es URL`, or an username/password (-user USER -pass PASSWORD) to connect to Elasticsearch. By default, this uses the default index pattern, metricbeat-* in this case, but you can specify a different index pattern in the -i INDEX. Please check the documentation for more details.

You can also use the import_dashboards script to import any custom Kibana dashboards of an Elastic Beat or a community Beat:

./scripts/import_dashboards -url

Where the -url represents the zip archive with the Kibana dashboards of a Beat or multiple Beats.

Metricbeat: Monitor Filebeat with Metricbeat

Add Filebeat MetricSet  in Metricbeat to monitor Filebeat. It exports statistics like the number of renamed, open or truncated files per harvester or per prospector.

Completed Postgresql module in Metricbeat

Added metricsets for per-database statistics and for the background writer, finalizing the new Postgresql module in Metricbeat.

Switch to Go 1.7

We have upgraded the Golang version to 1.7, so all the Beats version 5.0 and 1.3 will run with Go 1.7. This is needed for supporting OS X Sierra and should also bring some performance improvements. Upgrading required us to update also the OS X cross compiler.

Packaging Makefile refactoring

This PR moves part of the Makefile from the packer to the main Beats Makefile. The result is that the community Beats no longer have to provide a custom Makefile for packaging, simplifying the Beat generator and making it easier for the community authors to package their Beats.

Vendor libbeat in community Beats

The Beat generator now copies the Beats repo code in the vendor folder of the generated Beat. This encourages the practice of dependency vendoring (which we do for the official Beats) and makes it easier to cross-compile and package the Beats.

Metricbeat: Introduce experimental flag for cgroups

Mark experimental the cgroups support in the system module of Metricbeat. In this regard a new configuration option was added to enable/disable gathering data from cgroups for each process. By default it’s set to false to not send any cgroups information.

# EXPERIMENTAL: cgroups can be enabled for the process metricset.
#cgroups: false

Document the configuration file format

We now have a guide that walks the user through the various features and conventions used by our configuration files.

Document how to throttle the output bandwidth

We’ve also added a FAQ entry that explains how to use Linux tool to cap the bandwidth used by the Beats to Logstash/Elasticsearch communication. Because the Beats outputs are handling well the network backpressure, an OS level solution like this one is effective.