25 July 2016

Brewing in Beats: How Filebeat works

By Tudor Golubenco

Welcome to Brewing in Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.

In progress: add cgroup information to gosigar

As part of a larger effort to make container monitoring easy with the Beats, gosigar (the library we use in Metricbeat) is getting support for parsing and understanding the data from /proc/cgroups. This gives us not only the CPU and memory usage of each container, but also the configured limits for each. This is a feature requested, for example, by our Cloud team to properly report on the cpu usage as a percentage of the configured limit.

Metricbeat: Conversion helpers refactoring

The effort started last week to make most of the parsing and conversion work in Metricbeat declarative was completed this week. The bulk of the work involved in writing and maintaining a Metricbeat module lies in renaming and converting the metrics, so we hope that doing this declaratively will be an important productivity win. The Metricbeat developer guide was also updated.

How Filebeat works documentation

Getting confused about what a Prospector or what a Harvester is? Don’t know the difference between close_inactive and clean_inactive? We now have a single docs page explaining the high level Filebeat concepts and docs sections to explain all the relevant configuration options.

Path variables available in the configuration file

It is now possible to refer to path variables (e.g. path.home, path.data, path.config) in the configuration file. The benefit is that it is a bit more clear to write filebeat.registry_file: ${path.data}/registry than just the equivalent filebeat.registry_file: registry.

Format string support in libbeat

In several planned features (the lookup feature, dynamic pipeline selection, dynamic index pattern) it is useful to be able to format strings based on event fields. This PR adds supports in libbeat for using a format similar to the one used by Logstash. E.g. `pipeline: %{[fields.type]}`.

Metricbeat: add PGID to process data

Coming from a community contributor, this PR adds the process group id information to the per process docs.

Quickly switch Packetbeat protocols on/off in config

We have standardized on using enabled: false as a way to disable most things in the configuration file. For example, if you want to disable a Packetbeat protocol without commenting out 10 lines of config, add enabled: false. Want to disable an output? Add enabled: false.

Filebeat cleanups

The effort (meta ticket) of cleaning up the Filebeat code continued last week with this PR doing lots of small improvements.

Potential Filebeat file descriptor leak fixed

A potential FD leak was discovered and fixed. The file descriptors would have gotten garbage collected eventually, but it’s better to close them as soon as they are not needed anymore.

Kibana Dashboard for the Metricbeat network stats

This PR adds a new sample dashboard that can be used with the networking stats collected by the Metricbeat system module.