Logstash Lines: Update for January 12, 2018
In this edition of Logstash Lines we have some great news and new features!
JDBC Static Filter
We have an awesome new filter, the JDBC static filter . This filter enriches a events with a fast in memory SQL database. That database is seeded with data from a remote database via a JDBC connection. This is useful if there's a relatively static set of data on a remote DB you'd like to enrich local data with. The local DB can be reloaded periodically via a scheduler.
Due to some technical issues we haven't updated the docs yet, but you can find them here for now.
The Secret Store Moves one Step Closer
The secret store is a big change to LS, one that we're targeting for 6.2.0. We've almost got it ready to merge into master sans some docs and a couple more tests. It looks like it'll be close to the feature freeze date in 4 days, but should make it. The secret store will let users encrypt credentials on disk thus helping customers who have a requirement for no plaintext passwords on disk.
Looking Forward on Core
We're also making significant progress in our effort to bring a framework for distributed processing to Logstash. We don't have much we want to discuss now since a lot of it is ideas stage, but we've got exciting stuff cooking that we can't wait to show you once we've gotten a bit further along.
RabbitMQ Output now Parallel
We received a note that the RabbitMQ output was quite slow. We then discovered that it was completely serial, no matter how many workers you had. This has now been patched . Update your RabbitMQ output to the latest version to get this performance boost.
Beats Input NPE Fix
The beats input could crash with an NPE under load before a fix we released in 5.0.6. We recommend users to update to the latest version, 5.0.7 whenever possible.
Other LS Core Changes:
Perf improvements to java exec.
We now use the multiple pipelines feature by default in RPM/DEB packages via this PR.
Plugin Releases in the past two weeks:
logstash-output-http v5.1.2
Add check to avoid hanging pipeline if an empty event array is passed in. #80
logstash-input-s3 v3.2.0
Add support for auto-detecting gzip files with .gzip extension, in addition to existing support for *.gz
Improve performance of gzip decoding by 10x by using Java's Zlib
logstash-input-beats v5.0.6
Re-order Netty pipeline to avoid NullPointerExceptions in KeepAliveHandler when Logstash is under load
Improve exception logging
Upgrade to Netty 4.1.18 with tcnative 2.0.7
logstash-output-rabbitmq v5.1.0
Use shared concurrency / multiple channels for performance
logstash-filter-csv v3.0.8
Added support for tagging empty rows which users can reference to conditionally drop events
Notable Pull Requests in the past two weeks:
logstash-input-beats
Re-order pipeline to avoid NPE in KeepAliveHandler #283
logstash-input-s3
use Java gzip libraries for buffered decoding of gzip files #127
logstash-filter-csv
Added feature to skip empty rows preventing broken event warnings #60 *contribution by Abdul Hussain (@AbdulHaseebHussain)*
logstash-filter-elasticsearch
add ability to copy meta/docinfo fields and aggregations into Logstash event #83 *contribution by Fabien Baligand (@fbaligand)*
support copying nested fields #84 *contribution by Sam Woods (@samwoods1) + improvement by our Ry*
logstash-filter-kubernetes_metadata
Authorization header fix #5 *contribution by @antalov*
logstash-filter-mutate
Capitalize Feature Addition #106
logstash-output-email
Add mustache templating for email body #51 *contribution by Julien Bordellier (@jstoja)*
Add bcc parameter #55 *contribution by Julien Bordellier (@jstoja)*
logstash-output-rabbitmq
Use shared concurrency for performance / simplicity #71