Logstash Lines: Fixes for 5.x version, more metrics APIs

Welcome back to The Logstash Lines! In these weekly posts, we'll share the latest happenings in the world of Logstash and its ecosystem.

Fixes in 5.2.0

  • Improved user facing error when logstash.yml is invalid.
  • Generate license/notice file during artifact build. This file now includes notices and copyright info from all bundled dependencies.
  • Elasticsearch Output: Fixed a regression where the password option required URL escaping to be done by users. We also added new integration tests (using nginx proxy to ES) to validate authentication/security features for this plugin.
  • S3 Output: Cleanup temporary files used to upload data to S3.
  • Simplified the API response for persistent queue stats to only include “events”. This stat tells the user how many events are in the queue, waiting to be processed. A more detailed per-page stats is being worked on which exposes acked/unacked messages.

In-progress items

Metrics: events.in stats were previously recorded when the events entered the filter stage. This was ok until persistent queues were introduced, which made the existing calculation method inaccurate (due to buffering). This PR modifies the code to calculate "in" metrics before inserting into queue.

Persistent Queues: Implementing a recover process for the queue which helps in recovering data that was written to the data file, but not yet checkpoint'd. Consider a situation when the input has written a bunch of data to the queue, but Logstash crashes before checkpointing. Now, with this PR, on startup, we scan if there is any data in the queue but ahead of the checkpoint location and re-process them.

New Plugin: Google Pub/Sub Input

Eric Johnson from Google contributed a Logstash input for pulling events from Google Pub/Sub service.