Brewing in Beats: Filebeat filtering inputs
Welcome to Weekly Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.
We are happy to see another Beat created by the community: Jstatbeat created by Shin Tanimoto. It used for monitoring the JVM garbage collector monitoring by reading jstat results and indexing them in Elasticsearch. There is a blog post about the new Beat here (in Japanese).
Fixing ignore outgoing transactions in Packetbeat
An issue regarding ignore_outgoing configuration option was reported last week. The problem appeared as the configuration option was working only when the topology map was created. The fix allows usage of ignore_outgoing configuration option without the need to have the toplogy map enabled.
Additionally, setting the “direction” to outgoing is fixed in the same pull request.
Reduce bulk size
The bulk size value is changed from 10k to 200 with the pull request. The high value of 10k caused memory issues when Logstash was not available or slow to process data. This is because the 10k gets multiplied with the worker queue size (1000).
Exclude files in Filebeat
The pull request adds support for the “exclude_files” configuration option to the Filebeat prospector. You can specify a list of regular expressions to match the files that will be ignored. This is useful if you want to include all the files in a directory (/vat/log/apps), but exclude those that have been rotated and zipped (.gz).
filebeat: prospectors: - paths: [ "/var/log/apps/*"] exclude_files: "\\.gz$"
Non-string fields values in Filebeat
This community contribution adds support for non-string values of the custom Logstash fields in the configuration file.
Each field can now be a scalar value, an array, a dictionary, or any nested combination of these.
Add basic instrumentation to the Publisher
This pull request extends the -httpptof interface to export a few basic metrics from the libbeat publishers over expvar. These are very useful for us to troubleshoot performance issues and to detect when there are drops in Packetbeat and Topbeat.
Working on adding multiline support in Filebeat
Winlogbeat in the next release
The first version of Winlogbeat is almost ready and it will be included in the next minor release. It has the ability to read event logs from Windows Event Log API and Event Logging API. You can follow the Winlogbeat status under the GitHub meta issue.