12 January 2018

Logstash Lines: Update for January 12, 2018

By Andrew Cholakian

In this edition of Logstash Lines we have some great news and new features!

JDBC Static Filter

We have an awesome new filter, the JDBC static filter . This filter enriches a events with a fast in memory SQL database. That database is seeded with data from a remote database via a JDBC connection. This is useful if there's a relatively static set of data on a remote DB you'd like to enrich local data with. The local DB can be reloaded periodically via a scheduler.

Due to some technical issues we haven't updated the docs yet, but you can find them here for now.

The Secret Store Moves one Step Closer

The secret store is a big change to LS, one that we're targeting for 6.2.0. We've almost got it ready to merge into master sans some docs and a couple more tests. It looks like it'll be close to the feature freeze date in 4 days, but should make it. The secret store will let users encrypt credentials on disk thus helping customers who have a requirement for no plaintext passwords on disk. 

Looking Forward on Core

We're also making significant progress in our effort to bring a framework for distributed processing to Logstash. We don't have much we want to discuss now since a lot of it is ideas stage, but we've got exciting stuff cooking that we can't wait to show you once we've gotten a bit further along. 

RabbitMQ Output now Parallel

We received a note that the RabbitMQ output was quite slow. We then discovered that it was completely serial, no matter how many workers you had. This has now been patched . Update your RabbitMQ output to the latest version to get this performance boost.

Beats Input NPE Fix

The beats input could crash with an NPE under load before a fix we released in 5.0.6. We recommend users to update to the latest version, 5.0.7 whenever possible. 

Other LS Core Changes:

Perf improvements to java exec.

We now use the multiple pipelines feature by default in RPM/DEB packages via this PR.

Plugin Releases in the past two weeks:

logstash-output-http v5.1.2

Add check to avoid hanging pipeline if an empty event array is passed in. #80

logstash-input-s3 v3.2.0

Add support for auto-detecting gzip files with .gzip extension, in addition to existing support for *.gz

Improve performance of gzip decoding by 10x by using Java's Zlib

logstash-input-beats v5.0.6

Re-order Netty pipeline to avoid NullPointerExceptions in KeepAliveHandler when Logstash is under load

Improve exception logging

Upgrade to Netty 4.1.18 with tcnative 2.0.7

logstash-output-rabbitmq v5.1.0

Use shared concurrency / multiple channels for performance

logstash-filter-csv v3.0.8

Added support for tagging empty rows which users can reference to conditionally drop events

Notable Pull Requests in the past two weeks:

logstash-input-beats

Re-order pipeline to avoid NPE in KeepAliveHandler #283

logstash-input-s3

use Java gzip libraries for buffered decoding of gzip files #127

logstash-filter-csv

Added feature to skip empty rows preventing broken event warnings #60  *contribution by Abdul Hussain (@AbdulHaseebHussain)*

logstash-filter-elasticsearch

add ability to copy meta/docinfo fields and aggregations into Logstash event #83  *contribution by Fabien Baligand (@fbaligand)*

support copying nested fields #84 *contribution by Sam Woods (@samwoods1) + improvement by our Ry*

logstash-filter-kubernetes_metadata

Authorization header fix #5 *contribution by @antalov*

logstash-filter-mutate

Capitalize Feature Addition #106 

logstash-output-email

Add mustache templating for email body #51  *contribution by Julien Bordellier (@jstoja)*

Add bcc parameter #55 *contribution by Julien Bordellier (@jstoja)*

logstash-output-rabbitmq

Use shared concurrency for performance / simplicity #71