02 August 2016

Logstash Lines: Beats Input Performance and other fixes

By Suyog Rao

Welcome back to The Logstash Lines! In these weekly posts, we'll share the latest happenings in the world of Logstash and its ecosystem.

Persistent Queues
In preparation for persistence feature landing in 5.x, we added some supporting infra to the 5.0 pipeline. We now have a separate read (consumer) client and write (producer) client that deals with queue operations. We also encapsulated all the event batch management to a separate class.

Chasing a performance issue with Redis: A couple of us were involved in helping tune a user's throughput issue with Redis input for LS 2.3.4. User was migrating from 1.4.x and had experienced a slowdown. We'll convert all the good info in this issue to a blog.

Beats Input Performance: We are still investigating beats input's performance. The current rewrite in Java has given us a 50% increase in throughput performance, but our preliminary tests had shown ~100% increase! Anyhow, the good news is that the current beats input + filebeat combo is much faster than the LSF + lumberjack input combo, which was our original goal for this project. We are shipping this change in 5.0.0-alpha5 and will backport it to 2.4.


  • All our plugin tests were broken for a brief period because they didn't have a defined path.data directory. We fixed this by pointing to a temp directory which is only used in tests.
  • Throttle Filter: The plugin is now thread-safe, supports asynchronous input and properly tracks past events. Thanks to (@frpaex).
  • File Input: SinceDB will now be located in a new default path: ${path.data}/plugins/inputs/file/. This is a breaking change for 5.0.
  • Elasticsearch Output: We now have separate templates for 2.x and 5.x versions. Based on what ES version is running, we choose the right template to upload.
  • S3 Output: This output's code base was a dinosaur. We are refactoring it to make it more maintainable, use AWS 2.0 APIs, and improve performance.