24 May 2016 Engineering

Logstash Lines: New Settings File, Release Packages Improvements

By Suyog Rao

Welcome back to The Logstash Lines! In these weekly posts, we'll share the latest happenings in the world of Logstash and its ecosystem.

Following projects are in progress:

Monitoring: New System Stats

Following up on monitoring APIs that got added in alpha1, we've enhanced /_node/stats to include detailed, process level information like file descriptors, cpu and memory stats. All this can be found in _node/stats/process. As part of exposing these stats, we've refactored this module for maintenance — removed boilerplate code, simplified classes to mirror API structure, etc. Plan is to merge this into alpha3 (#4923).

Release Packages

This is a breaking change to make the directory structure in LS release packages (RPM, DEB) identical to ES. As part of this work, we are also doing a much needed upgrade to our service scripts to be able to use systemd and upstart.

Exploring Use Of Log4j

To make improvements to logging framework used by Logstash, we are exploring a Log4j. This will allow us to do component based logging (think per-plugin), log rotation, dynamic log level setting etc. Tricky bit is to come up with a solution that works well with existing structured logging APIs used in JRuby and pure Java (#4548).

Improve Beats Input

Collaborating with the Beats team to improve performance of LS beats input. A POC of rewriting this input using an async Netty based approach has showed promising results.

Logstash Settings File

Say hello to logstash.yml. This week we merged a feature to master and 5.0 which allows users to configure Logstash bootstrap settings using a yml file instead of doing it via CLI options. Please be aware that this is a breaking change in that most long form CLI options have been changed to mirror the yml dot notation. Using a settings file will un-clutter the already crowded CLI options and allow us to introduce more configurations for future features. Most importantly, we can now ship experimental features using feature flags. (#4401).


  • Java Event#toMap should return copy (#5926).
  • Added plugin manager tests to acceptance test branch.
  • Fix JRuby bigdecimal handling in Time.at function.
  • Released v3.1.0 for Netflow Codec with many improvements (thanks to Jorrit Folmer):
    • Added IPFIX support
    • Include scope fields when creating option templates
    • Fix various exceptions after receiving invalid Netflow v5 and v9 packets