12 July 2016

Logstash Lines: Persistent Queues, 2.3.4 Release, Pipeline Error Handling

By Suyog Rao

Welcome back to The Logstash Lines! In these weekly posts, we'll share the latest happenings in the world of Logstash and its ecosystem.

Released 2.3.4, which addressed an important security issues in Elasticsearch Output.

Persistence Queues

Good progress on Persistence Queues feature this week. Implementation is in full swing now. We added the first end-to-end tests that integrate all the building blocks classes — low level PageIO, in-memory checkpointing etc with the Queue interface. We've also introduced a Queue client class that handles batch of Events as its taken out of the queue by the consumer, manages its acks.

Improved pipeline error handling (5.0)

Added a configurable flag (pipleine.continue_on_error ) in Logstash to continue processing when there is an error in the filter stage. Sometimes there is a badly written Ruby filter, or a particular data or bug in plugin which triggers an error. For some users the right thing to do here is to crash Logstash so an operator is forced to diagnose and fix the issue. For others, they generally just want it to log an error and keep on running. One long term solution is to have a dead letter queue so bad events can be moved from the main pipeline. In the meanwhile, having this continue option will help some users.

Performance Tuning Guide (2.x)

We are putting together a performance tuning guide for Logstash 2.x. Until 5.0 GA we will not have any monitoring APIs, so debugging performance issues in 2.x can be hard. Especially since we changed the pipeline work unit (-w) to include both filters and outputs (in 2.2) we've had confusions about how to tune Logstash. In addition, to help with this, we are crafting a blog post to explain in detail the pipeline evolution from pre 2.0 to 2.3 timeframe.

Prepping for a 2.4 Release

LS 2.4 will be released with ES 2.4 and the rest of the stack. We did some ground work on 2.4 so that plugins developed in 5.x can be installed on top of 2.4. Dependency management in LS plugins is hard. And this involved some back and forth discussion before settling on a solution. Now all plugins will be mass updated to reflect the new version constraints.


  • JRuby announced a verbal EOL warning for JRuby 1.7, which we use in LS. We rebooted our efforts to test the latest stable version — jruby9k. The move to 9K is expected to happen in 5.x timeframe.
  • Added support for hostname:port syntax to SafeURI class.
  • Kafka Input: Fix shutdown sequence bug with a rogue shutdown event, and bump version (2.0.9).
  • Elasticsearch Filter: Fixed a bug where search was being executed on all indices even if index option was specified.
  • JDBC Input: Working on a PR to make sure charset conversion is happening at the input level. Event class in Logstash relies on all data being UTF-8 encoded and this input was passing through non-UTF8 data which caused issues downstream in the pipeline.