Logstash 2.3.0 and 2.2.3 Released
Important note: It's highly recommended to move directly to the Logstash 2.3.1 and 2.2.4 releases as they include various important bug fixes.
Logstash 2.3.0 has been released today, packed with extremely useful new features, an awesome performance boost and a few bug fixes. You can read the detailed release notes here, or jump directly to our downloads page if you can't wait to test drive this release! Additionally, we've released an update to the 2.2.x series, with a bunch of important bugs packaged in 2.2.3. You can read the changelog for 2.2.3 here.
Dynamic Config Reload
Today, any changes made to Logstash configuration files requires the entire process to be restarted, which is not ideal when you manage a multi-instance deployment. Logstash is often run as a centralized service to process data from multiple sources, departments and users. In such deployments it is fairly common for operators to onboard new data sources, and update Grok patterns to extract fields from new, unstructured data. Therefore, each iteration means you'd have to make config changes locally, push them to all Logstash instances and restart every instance to apply those changes. Starting with 2.3, you can set Logstash to detect and reload configuration changes automatically! This feature also reduces the feedback loop as you develop and debug new configurations.
To enable automatic config reloading, start Logstash with the --auto-reload
(or-r
) command-line option specified. For example:
bin/logstash -f apache.config --auto-reload
By default, Logstash checks for configuration changes every 3 seconds. To change this interval, use the --reload-interval <seconds>
option, where seconds specifies how often Logstash checks the config files for changes.
Alternatively, if you would like to manually force a reload and pipeline restart you can send a SIGHUP
(signal hangup) to the process.
Environment Variables In Configuration
Using environment variables is a common pattern to parameterize server instances, and many times it is convenient to use these variables to inject values into the Logstash configuration. Maybe you'd like to use the EC2 instance ID to tag the input, or use environment variables from your Dockerized Logstash instance to populate the hosts settings for Elasticsearch Output, and so on.. The possibilities are endless here.
Prior to this release, users had to use m4 (or any templating system) to achieve this before launching Logstash. Thanks to an amazing contribution from our community member Fabien Baligand, you can now reference environment variables directly from Logstash configuration. Simply use the syntax ${ENV_VAR:default}
and you're done. For example
input { tcp { port => "${TCP_PORT:54321}" } }
Java Event: Performance Gains Across The Board
A new chapter in Logstash implementation begins in 2.3. Some of you may know that Logstash is written in JRuby, and runs on JVM. In this release, we've completely rewritten the Event representation — a core component which encapsulates the data flow — in pure Java. So what does this mean for users? In our performance testing we've seen consistent throughput increases across multiple configurations. In some cases, we've seen up to 75% increase in events processed through Logstash. This change also provides the foundation for future persistence work where events will be persisted to disk while being processed by the pipeline. The Java implementation will make serialization to disk faster. Using the excellent interoperability between Java and JRuby, we were able to make this change 100% backward compatible — all the existing plugins and configuration will work seamlessly (#4191).
The graph below illustrates the performance throughput between the 2.3.0 and 2.2.0 release:
Note: All benchmarks were performed using the Logstash Performance tool, default settings, and configurations available here. Machine specifications can be found here.
Deprecations
We'd like to note that bin/plugin command — which is used to manage plugins — has been deprecated in favor of bin/logstash-plugin command. bin/plugin will be removed in the next major version mainly to prevent PATH being polluted when other components of the Elastic Stack are installed on the same instance. Also, this new command will be enhanced in future versions to work with cross-component plugin packs
Bug Fixes
Additionally, there have been dozens of bug fixes and minor enhancements spanning plugins and core. We've listed few of the important ones here:
- Beats Input: Added support to verify client certificates against the CA (on the server side). This was missing in both Beats input and its predecessor Lumberjack input (#8).
- RabbitMQ Input: A previous change which added metadata handling regressed performance. In this release, we disabled additional metadata by default, and also improved performance by using an internal queue and bulk ACKs.
- DNS Filter: Improved performance by adding caches to both successful and failed DNS requests. Added support for retrying with the :max_retries setting.
- Redis Input Batching: Changed the default behaviour of Redis input to read batches of events instead of one. As expected, this gives a good boost in performance (#36)
- Elasticsearch Output: Fixed a memory leak when sniffing feature was used to detect other hosts in the Elasticsearch cluster (#392).
- Fixed an issue where specifying JMX parameters in LS_JAVA_OPTS caused Logstash to not restart properly (#4319).
Feedback
We are super excited for this release of Logstash and look forward to your feedback. You can reach us at our forum, open issues in your GitHub repo or twitter(@elastic). Happy 'stashing!