Brewing in Beats: Filebeat filtering inputs

Welcome to Weekly Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.

Last week we released Beats 1.0.1.  Read the blog post, try it out and send us your feedback on discuss.

Jstatbeat

We are happy to see another Beat created by the community: Jstatbeat created by Shin Tanimoto.  It used for monitoring the JVM garbage collector monitoring by reading jstat results and indexing them in Elasticsearch. There is a blog post about the new Beat here (in Japanese).

Fixing ignore outgoing transactions in Packetbeat

An issue regarding ignore_outgoing configuration option was reported last week. The problem appeared as the configuration option was working only when the topology map was created. The fix allows usage of ignore_outgoing configuration option without the need to have the toplogy map enabled.

Additionally, setting the “direction” to outgoing is fixed in the same pull request.  

Reduce bulk size

The bulk size value is changed from 10k to 200 with the pull request. The high value of 10k caused memory issues when Logstash was not available or slow to process data. This is because the 10k gets multiplied with the worker queue size (1000).

Exclude files in Filebeat

The pull request adds support for the “exclude_files” configuration option to the Filebeat prospector. You can specify a list of regular expressions to match the files that will be ignored. This is useful if you want to include all the files in a directory (/vat/log/apps), but exclude those that have been rotated and zipped (.gz).

filebeat:
  prospectors:
  - paths: [ "/var/log/apps/*"]
    exclude_files: "\\.gz$"

Non-string fields values in Filebeat

This community contribution adds support for non-string values of the custom Logstash fields in the  configuration file.

Each field can now be a scalar value, an array, a dictionary, or any nested combination of these.

Add basic instrumentation to the Publisher

This pull request extends the -httpptof interface to export a few basic metrics from the libbeat publishers over expvar. These are very useful for us to troubleshoot performance issues and to detect when there are drops in Packetbeat and Topbeat.

Working on adding multiline support in Filebeat

Multiline support in Filebeat is an important feature request by our community. We started working on it and you can follow its status in this pull request.

Winlogbeat in the next release

The first version of Winlogbeat is almost ready and it will be included in the next minor release. It has the ability to read event logs from Windows Event Log API and Event Logging API. You can follow the Winlogbeat status under the GitHub meta issue.