Logstash Lines: Ingest to Logstash converter, upgrade to JRuby 9k

Welcome back to The Logstash Lines! In these weekly posts, we'll share the latest happenings in the world of Logstash and its ecosystem.

Last week we shipped 6.0.0-alpha2 -- the headline feature was the ability to run multiple pipelines in a single instance.

Ingest to Logstash conversion tool (6.0.0-beta1)

We now have a CLI tool in Logstash that takes an ingest pipeline in JSON and produces a corresponding Logstash config DSL. You can then run this config natively in Logstash, make changes to it etc. There are a few processors that are available in ingest, but don't have a LS counterpart -- these will be skipped during conversion. Here are some reasons why users would want to migrate from ingest to LS:

  • Take advantage of the richer transformation capabilities in Logstash. Ingest processors today is a subset of what Logstash filters provide.
  • Use multiple outputs. Ingest was designed to only support ES as an output. For example, users may want to archive their incoming data to S3 as well as indexing it in ES.
  • Use the persistent queue feature to handle spikes when ingesting data (from beats and other sources).

Example

$LS_HOME/bin/ingest-convert.sh —input file:///tmp/ingest/apache.json —output file:///tmp/ingest/apache.conf

Upgrade to JRuby 9K

Master branch now uses JRuby 9k as the underlying Ruby engine — all our core + plugin tests pass! This change also brings in Ruby 2.x support which means we can now update a bunch of our gem dependencies to newer versions. The previous JRuby version we were using (1.7.x) was only compatible with an older Ruby engine. 1.7.x was also EOL'd last year. JRuby 9k is the latest stable release, so we'll get new features and bug fixes. Big shoutout to PH and Guy for leading this effort

Changes in 5.x

  • We added stats for Dead Letter Queue (DLQ) feature. The API now reports DLQ's size on disk when the feature is enabled (#7287).
  • Fixed an issue where log levels could not be changed dynamically via an API. This was a regression from 5.1 (#7277).
  • Fixed a concurrency issue in the persistent queue. This issue was discovered when running 2 consumer threads and one producer in a specific sequence (#7382).

Changes in 5.5

  • Improved error logging in modules. Additionally, reduced the number of logging statements shown when running modules with default logging configuration (#7347).
  • Improved the general performance of the pipeline by 6% by locking more efficiently in the critical code path (#7300).
  • In preparation for running our integration tests on our internal Jenkins platform, we've dockerized the Elasticsearch and Filebeat services.