04 April 2017

Logstash Lines: Foundation to pipeline viz, HTTP proxy fixes

By Suyog Rao

Welcome back to The Logstash Lines! In these weekly posts, we'll share the latest happenings in the world of Logstash and its ecosystem.

Fixes in 5.4

  • Fixed a thread-safety issue affecting a few codecs when used in the context of inputs. Typically, codecs are stateless, but a few ones like line codec and multiline keep state of the events. We changed the clone behavior in inputs to also clone the codecs so each input owns a separate codec instance.
  • Using a http proxy with Logstash has been buggy in the last few releases. We merged a fix where the proxy information is passed to the subsystems that handles RubyGems as well as jar dependencies from maven.
  • Windows: Changed the startup batch script to search for the right Java executable in PATH environment variable
  • A user reported a potential data deletion issue when using the offline plugin pack command. We patched this command to a) exit with warning if file already exists b) added an explicit flag to remove existing file(s). We will be publishing a bug-fix release for 5.3 soon.
  • Merged a file-based store (implemented based on RecordIO) infrastructure that will be used in the upcoming DLQ feature. This library will be used to store “dead” events in 32kb blocks in segments, optimized to search events using a timestamp.
  • Http Input: Improve error logging to include full exception details.

Logstash Intermediate Representation (6.0.0)

This week we merged the backend code which adds support to convert the existing Logstash config into a graph based representation. This new config representation in JSON can then be used to draw the pipeline visually, along with its metrics in the Kibana. The project is also called Logstash Intermediate Representation or LIR internally and can form the foundation for many new features (in addition to the visualizer). For example, we can now start thinking about a UI config editor, rewriting the pipeline execution in Java, new config language, etc. Stay tuned for updates in this area. If you are interested, you can also check our Elastic{ON} 17 video for live demos in this area!