08 August 2016

Logstash Lines: Grok timeout option, SSL for TCP output and more

Von Suyog Rao

Welcome back to The Logstash Lines! In these weekly posts, we'll share the latest happenings in the world of Logstash and its ecosystem.

Migrating Logstash logging to Log4j (In-progress for 5.0)
LS has used a home-grown library called Cabin to do all its structured logging. As we write more Java components in core and in plugins, we've had to consolidate our logging framework across the board to Log4j. Also, log4j brings in nice features like log rotation, per-component based logging... With this feature, we will also be able to control setting log levels dynamically via an API. Tal recently demoed a working version of this, and we're putting finishing touches.

Document generation
Logstash's plugin documentation gets generated from the asciidoc embedded in the code itself. Recently we've been running into many issues with plugin generation tool, which was written a while ago. PH has been refactoring this tool to make it more robust while adding much-needed enhancements like versioned docs, generating directly from Github repo, easy html preview, dealing with dependencies etc.

Plugin updates:

  • Grok Filter: Added support to cancel long-running execution. Many times users write runaway regular expressions which lead to a stalled Logstash. You can configure timeout_millis to cancel current execution and continue processing the event downstream.
  • Elasticsearch Output: Updating LS's index template to match ES's 5.x changes. We are blocked on whether we should use .raw or .keyword as the default sub-field for the multi-field option for strings. There are some discussions ongoing in this ES issue.
  • JDBC Input: You can now use any timestamp column to 'sync' data from a table. tracking_column_type is a new config which tells the plugin whether you track current position using a numeric column or timestamped column. Users can then use this saved state in the SQL expression (Thanks to @growse).
  • Ganglia Input: Added ganglia metric name and value to property object, a popular feature request.
  • TCP Output: SSL/TLS support has been added to this output, thanks to @michaelweiser.

Core bugs:

  • Fixed bug where plugin manager provided confusing information to a user for upgrades. This was originally put in to protect users from accidentally upgrading to a plugin major version, but in the end, it just confused users.