Tech Topics

Announcing Logstash 1.5.0 RC1

By
Suyog Rao

We are announcing the first candidate for the release of Logstash 1.5.0. You can download Logstash 1.5.0 RC1 and review the associated changelog. As we mentioned in the release of Logstash 1.5 Beta1, the main areas of focus for 1.5.0 were plugin management, performance improvements, and Apache Kafka integration. Thanks to all our users who reported their experiences with the beta1 release. We made a number of fixes and improvements since then; we would like to highlight few of them here.

[UPDATE] Known Issues

If you are using the “node" or “transport" protocol with the Elasticsearch output, follow these steps to set up your output with Logstash 1.5.0 RC1 (#2780): 

<code>% bin/plugin uninstall logstash-output-elasticsearch
...
% bin/plugin install logstash-output-elasticsearch
...

CouchDB Input

We added a brand new plugin to stream events from the _changes API of CouchDB. With this input, any future changes will automatically be streamed as well, making it easy to synchronize your CouchDB data to any target destination. Check out the reference documentation for more information! We will be packaging CouchDB input in the Logstash release package, so it'll be ready to use as soon as you upgrade!

Retries in Elasticsearch Output

We made the Elasticsearch output more resilient to transient errors in Elasticsearch. Previously, partial failures from the bulk indexing functionality were not handled properly. With this fix, we added the ability to capture failed requests from Elasticsearch and retry them. Error codes like 429 (too many requests) will now be retried by default for 3 times. The number of retries and the interval between consecutive retries can be configured. (#1631)

Heartbeat Input Plugin

We created a new input plugin for generating heartbeat messages at periodic intervals. Use this to monitor Logstash — you can measure the latency of the pipeline using these heartbeat events, and also check for availability. For more information and configuration, check the reference docs.

Improved S3 Input and Output

We made a number of important fixes to the S3 input and output plugin. Among them:

  • Fixed a critical problem in the S3 Output plugin when using the size_file option. This could cause data loss and data corruption of old logs.
  • Added IAM roles support so you can securely read and write events from S3 without providing your AWS credentials (#1575).
  • Added support for using temporary credentials obtained from AWS STS.
  • Improved the reference documentation and added new configurations.

Windows Support

We have made great improvements for running Logstash and plugin related infrastructure on Windows, which was degraded since the 1.4.2 release. Users can now install, upgrade, and remove individual plugins on Windows at any time. We also resolved issues related to initial setup, upgrade and file input plugin:

  • Fixed issue for file locking which was causing log files to not rollover. This was also causing log messages to be dropped (#1557).
  • Better support for storing last read file positions using SinceDB (#1902).

Other fixes

  • Multiline Filter: Handled cases where we unintentionally deduplicated lines, such as repeated lines in xml messages (#3).
  • Lumberjack Input: Resolved issue where unrelated events were getting merged into a single event while using this input with with the multiline codec (#1). Added support to validate config file encoding when it contains non-ascii characters (#1468).
  • Mutate Filter: Fixed issue where convert option was creating an extra field in the event (#2268).
  • For plugin developers: We deprecated the milestone configuration in the plugins. Milestone config existed to indicate multiples level of maturity for a plugin inside one specific version of Logstash. Since the plugins were separated to self-contained RubyGems with version information, we have decided to remove the milestone option.

Documentation Updates

We continue to enhance our reference documentation. Since beta1, we added a comprehensive guide for writing and publishing plugins. Check this tutorial to add an input plugin; you can also find resources for developing other plugins in the same location.

Update from beta1

Since releasing beta1 we made internal changes to the way we install and load plugin gems using the Bundler framework. Unfortunately with this change, we cannot upgrade plugins installed during beta1 to the RC1 version. It is strongly recommended to go with a clean install of LS 1.5.0 RC1. Moving forward, you should be able to upgrade the RC1 package to GA release without any issues. It is worth emphasizing that any Logstash configuration created in beta1 and RC1 can be moved safely to the final 1.5.0 release.

Check it out!

We are closing on the final few issues before we can release Logstash 1.5.0. As always, we would love your feedback on this release. Please download it and let us know what you think. You can open issues on Github and provide feedback on twitter (@elasticsearch).

Beyond 1.5

We already started planning beyond 1.5, and to make our efforts easier to follow, we published the Logstash public roadmap with the themes of the next major release. We would love for you to help us in the release planning process by taking the Logstash User Survey!