19 Juli 2016

Logstash Lines: Persistent Queue progress, more monitoring API fixes

Von Suyog Rao

Welcome back to The Logstash Lines! In these weekly posts, we'll share the latest happenings in the world of Logstash and its ecosystem.

Persistent Queues

Nice progress again this week. On Friday, we were able to plug the various components (producer, disk-backed store, consumers), write test data to disk, successfully read it on the other side and acknowledge it. We now have a meta issue to track progress, and have a design issue in Logstash GH repo.

2.4 Readiness

Updated all the plugins to use a relaxed version constraint. This allows plugins written for LS 5.0 to be used in the upcoming 2.4 release, so we don't have to maintain 2 separate branches.

Monitoring API

Work in progress to add support for specifying a comma-separated list as parameters to these APIs. Just like Elasticsearch. For example, http://localhost:9600/_node/stats/jvm,process will filter and show only jvm and process information. Other small fixes to make sure these APIs are implemented as per the specs, and are consistent with ES APIs.

Beats Input Rewrite

More manual testing this week surfaced a couple of issues. Tal found and fixed a memory leak when using the compression feature. He also changed the assembly process which packages Java, JRuby and its dependencies (jar libs) to use Gradle instead of jar_dependencies. We released a beta version of this plugin, which can be installed via

bin/logstash-plugins install --version 3.1.0.beta1 logstash-input-beats

Kafka Plugin

Added feature to allow regex patterns in topics, so you can subscribe to multiple ones. Thanks to Anup Chatterjee (@anupchat) for this contribution. Added extra Kafka based metadata (size, partition, etc) for Events

Conference Speaking: Aaron Mildenstein has been on a speaking marathon this week! He presented a 3 hour crash course on the Elastic stack at the OpenWest conference. His other talks were on writing custom LS plugins, managing and performance tuning ES + LS! I'm sure his swag bag was full as well :)

Bug Fixes:

  • Fixed an issue where String setting when created with a range of possible values, was not properly checked. This allowed a setting to have a value outside of the array.
  • Fix human URI parameter for hot_threads API to properly format response suitable for humans.
  • Fixed Netflow codec to work with logstash 2.3.4