Product release

Logstash 2.0.0 released

作者
Suyog Rao

We are happy to announce that Logstash 2.0.0 has been released today! This release introduces breaking changes in functionality for some configurations, so please read our changelog for details, or the new breaking changes documentation.

You can read our previous posts for changes in other 2.0.0 pre-releases:

Here's a recap of headlining changes in 2.0:

Compatibility with Elasticsearch 2.0

Elasticsearch 2.0 is generally available as of today which contains many new features and enhancements. Logstash 2.0 is compatible out of the box with this release. Previous releases of Logstash defaulted to using the Java node client to communicate with Elasticsearch; 2.0 defaults to the HTTP client. This provides a seamless experience for users to get their data, enrich it, and store and analyze with Elasticsearch.

HTTP has functional parity with the other protocols (node and transport), is only marginally slower when connected to a single client, yet far easier to administer and work with. When using the HTTP protocol, one may upgrade Elasticsearch versions without having to upgrade Logstash in lock-step. For more detailed information (including performance numbers) on the switch to HTTP as default, please check the beta1 blog.

We'll continue to support the other (node, transport) protocols, but the plugin to use them requires manual installation:

bin/plugin install --version 2.0.0 logstash-output-elasticsearch_java

Compatibility Matrix

The below table lists Logstash's compatibility with Elasticsearch versions.

enter image description here

Compatibility with Shield 2.0

This release is compatible with Elastic Shield 2.0 release. For HTTP protocol, no additional plugins are required. Please follow the documentation described here. For the transport protocol, a separate plugin has to be installed for integrating with Shield 2.0 and above:

bin/plugin install --version 2.0.0 logstash-output-elasticsearch_java_shield

Performance improvements

This release also includes performance improvements in many areas so you can process your data faster with Logstash. Below are a few mentions:

  • UserAgent and GeoIP Filter: In both these filters, we've optimized lookups by adding a LRU cache. This takes advantage of the fact that IPs and user agents commonly seen in web requests often appear in a clustered sequence. In the case of the user agent filter, we saw a boost of ~3.7x on our sample dataset. For GeoIP, the performance boost compared to old version is 1.69x.
  • JSON processing: We updated to a new version of JrJackson, which is the JSON serialization/deserialization library used in Logstash. In addition to fixes, this provides good performance boosts when dealing with JSON.
  • Better Filter Worker Defaults: Before this release, the filter_worker setting had a default value of 1, which meant having a single worker to process events entering the filter stage. Now, out of the box, the default value of the filter_workers setting will be set to half of the CPU cores of the machine. Increasing the number of workers provides parallelism in filter execution, which is crucial when doing heavier processing like complex grok patterns or the useragent filter.

Filebeat Support

In case you missed it, we recently released a beta version of Filebeat -- the next-generation Logstash Forwarder. Filebeat is an agent to ship file-based logs to Logstash for further processing. 2.0.0 works out of the box with Filebeat version 1.0.0-beta4 using the logstash-input-beats plugin.

Shutdown Handling

In previous Logstash releases, when a shutdown was initiated, an Exception mechanism was used to signal the plugins to start shutting down. This process was problematic because plugins frequently use third-party code. When Logstash did not know how to handle the exceptions, unpredictable behavior often resulted. We fixed this by adding API calls (for example, stop) for each plugin to communicate a shutdown event and let the plugin gracefully stop itself. This meant updating over 200+ plugins in the Logstash ecosystem to adhere to the new APIs! Although shutting down Logstash is not completely fixed yet -- stalled outputs can still delay the shutdown -- we have all the breaking API changes in 2.0 and can start iterating on fixes in point releases.

Plugin developers: If you have developed plugins for Logstash 1.5, please consult the breaking changes document for a list of new APIs related to shutdown. Also, the example input repo provides sample code for using this new shutdown mechanism.

Documentation

The updated documentation for 2.0 and all plugins are available here. Please consult this reference for any configuration changes.

Updating to 2.0

Before updating to 2.0, please consult the updating guide in our reference docs.

Feedback

Many thanks to our users and contributors for making 2.0 a successful release. We appreciate all the testing of pre-releases and numerous patches contributed to this release. Please follow our Roadmap to stay tuned on future enhancements, releases, etc. So, go on, give 2.0 a spin today and let us know what you think!