In this edition of Logstash Lines we have some great news and new features!
JDBC Static Filter
We have an awesome new filter, the JDBC static filter . This filter enriches a events with a fast in memory SQL database. That database is seeded with data from a remote database via a JDBC connection. This is useful if there's a relatively static set of data on a remote DB you'd like to enrich local data with. The local DB can be reloaded periodically via a scheduler.
Due to some technical issues we haven't updated the docs yet, but you can find them here for now.
The Secret Store Moves one Step Closer
The secret store is a big change to LS, one that we're targeting for 6.2.0. We've almost got it ready to merge into master sans some docs and a couple more tests. It looks like it'll be close to the feature freeze date in 4 days, but should make it. The secret store will let users encrypt credentials on disk thus helping customers who have a requirement for no plaintext passwords on disk.
Looking Forward on Core
We're also making significant progress in our effort to bring a framework for distributed processing to Logstash. We don't have much we want to discuss now since a lot of it is ideas stage, but we've got exciting stuff cooking that we can't wait to show you once we've gotten a bit further along.
RabbitMQ Output now Parallel
We received a note that the RabbitMQ output was quite slow. We then discovered that it was completely serial, no matter how many workers you had. This has now been patched . Update your RabbitMQ output to the latest version to get this performance boost.
Beats Input NPE Fix
The beats input could crash with an NPE under load before a fix we released in 5.0.6. We recommend users to update to the latest version, 5.0.7 whenever possible.
Other LS Core Changes:
Perf improvements to java exec.
We now use the multiple pipelines feature by default in RPM/DEB packages via this PR.
Plugin Releases in the past two weeks:
Add check to avoid hanging pipeline if an empty event array is passed in. #80
Add support for auto-detecting gzip files with .gzip extension, in addition to existing support for *.gz
Improve performance of gzip decoding by 10x by using Java's Zlib
Re-order Netty pipeline to avoid NullPointerExceptions in KeepAliveHandler when Logstash is under load
Improve exception logging
Upgrade to Netty 4.1.18 with tcnative 2.0.7
Use shared concurrency / multiple channels for performance
Added support for tagging empty rows which users can reference to conditionally drop events
Notable Pull Requests in the past two weeks:
Re-order pipeline to avoid NPE in KeepAliveHandler #283
use Java gzip libraries for buffered decoding of gzip files #127
Added feature to skip empty rows preventing broken event warnings #60 *contribution by Abdul Hussain (@AbdulHaseebHussain)*
add ability to copy meta/docinfo fields and aggregations into Logstash event #83 *contribution by Fabien Baligand (@fbaligand)*
support copying nested fields #84 *contribution by Sam Woods (@samwoods1) + improvement by our Ry*
Authorization header fix #5 *contribution by @antalov*
Capitalize Feature Addition #106
Add mustache templating for email body #51 *contribution by Julien Bordellier (@jstoja)*
Add bcc parameter #55 *contribution by Julien Bordellier (@jstoja)*
Use shared concurrency for performance / simplicity #71