New Plugin API Compatibility
While 2.4 does not have many headlining features, it's an important release that brings in key internal API changes. These changes make 2.4 compatible with plugins developed for the 5.0 API. In other words, Logstash 2.4 will be able to install most plugins that use the 5.0 API. As maintainers, this API backport eases efforts which mean we don't have to maintain 2 separate code branches for each plugin. We have a lot of plugins.
The Logstash 2.4 release enables new plugin features that are not available in the remaining 2.x series. We've recently updated a bunch of plugins to add new functionality. A popular enhancement request has been to bring in new features in Kafka's 0.9.x and 0.10.x releases like security. In 2.4, we don’t package Kafka input/output plugin versions that support newer Kafka versions (see the compatibility matrix). This is because most config options for these plugins have been changed which makes it not compatible, and we don't intend to break configurations in point releases. If you still want to get all the goodness of the latest Kafka versions in 2.4, fear not. You can simply install the plugin on top of 2.4, like so:
bin/logstash-plugin install —version 5.0.4 logstash-input-kafka
This is just an example. Like Kafka, there are many plugins which have been updated. They'll be packaged in our upcoming 5.0 release, but if you can't wait, you can always install them on top of 2.4. Please review the changelog for individual plugins before upgrading to them.
Beats Input v3.0: Locked And Loaded
Beats input has been reimplemented using Netty, an asynchronous IO framework for Java. This rewrite for performance brings it in line with Logstash Forwarder + LS combination. In some test cases, it is faster. Go on, migrate all your LSF instances to the Filebeat goodness. I've heard Filebeat 5.0 is packed with features, so you don't want to miss out on that. As part of the Beats refactor we now only supports private keys in the PKCS8 format, you can use OpenSSL to convert them, newer version of OpenSSL will already create the key in the right format. Oh and a note for our subscription customers — LSF will reach EOL on Nov 7, 2016.
Back in the day, Logstash used to emit its logs in JSON format. Because of endless encoding issues and crashes in the logger, we had to roll it back. We've worked through most of these issues, so we're bringing structured logging back. You can tell LS to switch to JSON logging by using the
--log-in-json CLI flag. Be aware that logging in JSON is verbose and may fill up disk even faster than before!
Kibana has it. Beats has it. And now Logstash has it too! We're talking about the plugin generator tool that makes it easier to bootstrap new plugins for Logstash. Previously we've recommended developers to clone/fork the examples repositories, but now you can simply do:
bin/logstash-plugin generate --type input --name xkcd --path ~/ws/elastic/plugins
This subcommand bootstraps a new plugin logstash-input-xkcd with the right directory structure and all the required files (templates) for you to start developing this plugin right away. So, go on, create that xkcd input to stream those fine comic strips to Kibana!