30 May 2016

Brewing in Beats: Getting Metricbeat ready for release

By Tudor Golubenco

Welcome to Brewing in Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.

New community Beat: CloudTrailBeat

An interesting new Beat was created by Aidan Rowe. CloudTrailBeat relies on a combination of SNS, SQS and S3 to create a processing 'pipeline' to process new log events quickly and efficiently. The Beat polls the SQS queue for notification of when a new CloudTrail log file is available for download in S3. Each log file is then downloaded, processed and sent to the configured receiver (logstash, elasticsearch, etc).

Metricbeat: export CPU usage fields as percentages

With this change, the system module in Metricbeat exports all CPU usage fields (user, system, idle, etc.) as percentages from the total CPU time. This makes it easier to make sense of the values. The old “ticks” units are also available, but disabled by default.

Metricbeat: export network IO stats

This new metricset provides network IO metrics collected from the operating system. An event is generated for each network interface. This metricset is available on Darwin (aka OSX), Windows, Linux, and FreeBSD.

Filebeat new registry file format

The previous format for the registry file was a dictionary with the file paths as keys. This had the potential to lead to overwrite and conflicts on file rotation. As the file path is also stored inside the state object this information was duplicated. Now a pure array is stored in the registry which makes the format more flexible and brings it close to the format used by Logstash.

The migration from the old format happens automatically on the first run.

Lumberjack code extracted in a library

We now have a go-lumber library that implements both the client and the server side of the Lumberjack protocol, which we use between Beats and Logstash. The unit tests for logstash output now use lumberjack server implementation from go-lumber, simplifying test code quite a bit. Custom incomplete lumberjack server has been removed from output plugin. The above referenced PR also does the first steps towards replacing the default JSON encoding with custom code that does JSON encoding without reflection and with fewer allocations.

Generator for Metricbeat modules/metricsets

Adding a MetricSet requires creating several files with a predefined file structure for automation. To simplify the task of creating a MetricSet and make sure the correct structure is used, this task is now automated with a script.

Normalizing Metricbeat fields

We’ve continued to normalize the fields exported by Metricbeat to use a common format. This week the MySQL module and the system module field names were refactored.

Configuration files improvements

Continuing the theme from the last weeks, the configuration files were improved by adding Kafka and Redis outputs to the “full” versions. Our system tests were updated to use a similar structure for configuration as the files that we ship. Also, the docs were updated to use the same style in configuration samples.

Packetbeat NFS fixes and refactoring

Community contributor kofemann has done a great job improving the NFS module in packetbeat with fixes and refactorings to make the code cleaner.