Brewing in Beats: Configurable index patterns
Welcome to Brewing in Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.
New community beat: Burrowbeat
Burrowbeat is a new Beat that can be used to monitor consumer lags in Kafka queues. It is based on the Burrow project.
Configurable index patterns
The Beats now offer a lot more flexibility over the index names. They used to enforce daily indices of the form beatname-%{+yyyy.MM.dd}
, which is still the default. But it’s now possible to do weekly or hourly indices using a syntax similar to Logstash, or something like beatname
to work with the new rollover APIs from Elasticsearch, or even beatname-%{type}-%{+yyyy.MM.dd}
to automatically split the data into separate indices based on any field from the events.
Use scaled floats for percentages
After adding support for half floats a few weeks back, Adrien did it again and added scaled floats to Elasticsearch 🎉. Scaled floats are stored as longs behind the scenes, which makes them benefit of the compression scheme used for integers in Lucene. This makes them a great fit for the way we store percentages in Beats: a number between 0 and 1 that gets formatted as a percentage by Kibana.
This PR switches our percentages to use scaled_float. We had quite a few of those, so we can expect a significant improvement in the storage footprint.
Automatically generate Kibana index patterns
The sample Beats dashboards contain the Kibana index patterns, which allow us to define custom formatting for some of our fields (think of percentages). Providing the index pattern also saves the user a step while getting started. We used to create these index patterns by exporting them from Kibana, in a mostly manual process. To improve on this, Monica created a script to generate the index patterns from our fields.yml
files, which are the primary source data for everything that the Beats export.
Lookup functionality
The lookup feature allows attaching arbitrary metadata to events by calling external scripts. The scripts are called with parameters that can be taken from the original event, and the results cached for that parameters. This can be used, for example, to attach extra metadata to every file read by Filebeat by calling the Kubernetes APIs. The same could be used to add custom metadata to every process monitored by Metricbeat. Since the results are cached, the external script is called only once per file/process.
This functionality is now merged into an experimental branch called x-exec-lookup
, which we plan to merge into master after we add several security checks around calling external scripts.
Filebeat: shorter close_inactive default
Theclose_inactive
option sets after what time interval Filebeat closes the open files that don’t receive updates. The old default used to be 1h, which could mean that we keep a lot of open files in the case of quickly rotating files. This PR changes the default to 5m. If a file receives updates after it was closed, it is picked up again by Filebeat, so the lower default doesn’t mean any risk of data loss.