30 January 2018

Logstash Lines: Update for January 30, 2018

By Andrew Cholakian

Logstash 5.6.7 and 6.1.3 Released

They're hot off the press and ready for downloading. The notable new things here are:

6.1.3 fixes a bug where single shot pipelines, like the ES input or stdin, might not fully process all incoming data.

Both of these releases include support for scheduled queries. The plugin docs aren't yet updated on the site, but you can view them in the git repo for the moment.

There are more items in the release notes for 6.1.3 and the release notes for 5.6.7, but I'll omit them here for brevity

Filebeat pipelining may cause extra memory usage

Filebeat now defaults to enabling the pipelining option. This can cause Logstash to buffer more data than normal. There has been one case in the field of this causing an issue, but that involved a very high ratio of Filebeats to Logstashes. We're currently working on improving efficiency on the beats input side.

A short term workaround is to set pipelining in the Filebeat Logstash output to a lower number than the new default of 5.

Pipeline Viewer gets timeseries charts in detail drawer

In 6.2.0 we're targeting the release of a detail drawer for the Pipeline Viewer. Users could click on a vertex in the pipeline and view the vertex's details in the drawer. This included scalar metrics about that vertex (e.g. a grok filter vertex).

In 6.3.0 we are targeting an enhancement of these detail drawer metrics to be timeseries charts. Users can now get a historical picture of how a particular plugin was performing in a pipeline over time.

detail-graph-ss.png

Javafication and Core Cleanups Continue!

We've been able to do a ton of work moving various bits of Ruby code in Logstash core to Java, as well as removing some redundant classes. By moving to Java we've: 1.) Improved maintainability 2.) Improved performance. 

Notable Plugin Fixes/Improvements

Fix for thread leak in Grok filter.

EU Decimal comma support for mutate.