Logstash Lines: Monitoring API, Logging Enhancements, S3 Output changes

Welcome back to The Logstash Lines! In these weekly posts, we'll share the latest happenings in the world of Logstash and its ecosystem.

Monitoring API

  • Added stats for reporting on dynamic config reloads. We now measure the timestamp when most recent config reload successes and failures happened.
  • Added support to reset pipeline specific stats when a reload happens. Stats for JVM, GC, global event count and others are preserved across pipeline restarts.

Log4j

  • By default, Logstash's internal logs are now emitted to both stdout and a rolling file called logstash.log in a dir configured using the path.logs settings. By default path.logs points to $LS_HOME/logs.
  • Modifications to settings API to use module name and log severity to make it consistent with ES. Also added an endpoint to list all available logger module name.
PUT _node/settings
{"logger.logstash.output.elasticsearch" : "error"}
GET _node/logging?pretty
{
  ....
  "loggers" : {
    "logstash.registry" : "WARN",
    "logstash.instrument.periodicpoller.os" : "WARN",
    "logstash.instrument.collector" : "WARN",
    "logstash.runner" : "WARN",
    "logstash.inputs.stdin" : "WARN",
    "logstash.outputs.stdout" : "WARN"
  ...  
  }
}

Java Event

  • Renamed package namespace from com.logstash to org.logstash.
  • As part of fix-it-friday, we updated many community maintained plugins to use the new Event get/set API. We still have about 50 left, but we're in good shape to knock them off before RC1.

Persistent Queues

This project is chugging along nicely. Last week we got all the core unit tests to pass with the in-memory queue infrastructure. Integration tests are yet to be added which will use the real file-backed queue. Plan is to merge this feature branch to master in 2 weeks, following a code review.

S3 Output

PH is working on adding tons of improvements to this output to make it easier to maintain/test. We're also adding popular feature requests such as — using event based data to determine the target S3 bucket/location. Like bucket => "logstash-output-bucket/%{type}". Also, updating to AWS ruby client v2 should fix many bugs and knock off some enhancements like multi-threaded uploader. Suffice to say, this plugin needed some love.