Product release

Logstash 2.1.0 released

著者
Suyog Rao

We are happy to announce that Logstash 2.1.0 has been released today! This release is packed with enhancements, some of which we highlight below. Jump to the downloads page for the binaries, where you can also find all the changes that made this release.

Installing Plugins Offline

Logstash has a thriving plugin ecosystem where our community is constantly adding the ability to connect to different inputs, outputs, and processing data. The latest count of plugins you ask? 210 plugins. Previously, with the 1.5.0 release, we made it really easy to install and update available plugins by hosting them on public RubyGems.org. Users who deployed Logstash in an environment which did not have internet connectivity - air-gapped datacenter or firewalled access - wanted to interact with plugins. In this release, we provide multiple solutions to address this situation. Details below:

1. Mirrored RubyGems server

As mentioned before, Logstash uses the public RubyGems.org as a repository for its plugins. If you are in a situation where you don't have access to this server, or maybe you are developing plugins for internal use, you can mirror RubyGems.org and use this as the installation source. Several open source projects enable you to run your own plugin server, for example - Geminabox, Artifactory, etc. Once you have this setup, you can simply point the Gemfile shipped with Logstash deployment to point to this source. Detailed steps here.

2. Offline Plugin Packages

We've added enhancements to the plugin script to prepare plugin packages (with dependencies) so they can be installed in an offline box. The solution requires a staging machine running Logstash that has access to a public or private Rubygems server. This staging machine downloads and packages the files used for offline installation. Once you have this package, you can transfer this to your air-gapped machine or host it on a shared network drive. This becomes the installation source for plugins. With this you can simply do

bin/plugin install --local logstash-input-jdbc

3. Uber Package

In addition to the above solution, we've decided to periodically publish - every point release - a full-resolved Logstash binary that includes all known plugins and dependencies. You can download this uber package here.

Shutdown Improvements

Building on shutdown enhancements which were delivered in Logstash 2.0.0, we've added the ability to start Logstash with a CLI option (--allow-unsafe-shutdown) that allows it to predictably shutdown when the operator initiates it. This is particularly useful in situations where the processing pipeline is blocked, trying to communicate to external sources. Users were frustrated with not being able to shutdown Logstash in these situation. Please be aware that shutting down Logstash could still lead to loss of any in-flight messages.

To help with the shutdown situation, logic has been added to detect a potential stall while processing events. The stalled detection uses both the count of inflight events in internal queues and an analysis on busy worker threads. A staleness in both count and running threads triggers the warning mechanism and forces an unsafe shutdown.

A report is published when this happens:

^CSIGINT received. Shutting down the pipeline. {:level=>:warn}
Received shutdown signal, but pipeline is still waiting for in-flight events
to be processed. Sending another ^C will force quit Logstash, but this may cause
data loss. {:level=>:warn}
 {:level=>:warn, "INFLIGHT_EVENT_COUNT"=>{"input_to_filter"=>20, "total"=>20}, "STALLING_THREADS"=>{["LogStash::Filters::Ruby", {"code"=>"sleep 10000"}]=>[{"thread_id"=>15, "name"=>"|filterworker.0", "current_call"=>"(ruby filter code):1:in `sleep'"}]}}
The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information. {:level=>:error}
 {:level=>:warn, "INFLIGHT_EVENT_COUNT"=>{"input_to_filter"=>20, "total"=>20}, "STALLING_THREADS"=>{["LogStash::Filters::Ruby", {"code"=>"sleep 10000"}]=>[{"thread_id"=>15, "name"=>"|filterworker.0", "current_call"=>"(ruby filter code):1:in `sleep'"}]}}
 {:level=>:warn, "INFLIGHT_EVENT_COUNT"=>{"input_to_filter"=>20, "total"=>20}, "STALLING_THREADS"=>{["LogStash::Filters::Ruby", {"code"=>"sleep 10000"}]=>[{"thread_id"=>15, "name"=>"|filterworker.0", "current_call"=>"(ruby filter code):1:in `sleep'"}]}}
Forcefully quitting logstash.. {:level=>:fatal}

Twitter Input Enhancements

Numerous enhancements have been made in the popular twitter input which is based on Twitter's streaming API:

  • Added the ability to fetch data from the sample endpoint. This is very useful when you want to test your connection to twitter streaming API and validate your data flow. Using the sample option, you will read a small random of all public tweets.
  • Added an option to stream tweets from the authenticated user's timeline.
  • Twitter stream can be configured to filter by locations and language.
  • Capture hashtags, symbols and user_mentions as data for the non extended tweet event.

Bug Fixes

Below we highlight some of the bugs fixed in this release:

  • Multiline: Properly handle multiline events from multiple sources when using File input and Beats input. This means you no longer have to have separate file inputs to follow multiple files which emit logs in multiline format.
  • Eventlog: Changed the underlying library to capture Event Logs from Windows more reliably
  • Changed the default heap size from 500MB to 1GB. Many users use Logstash as a centralized logging server, and the default heap was resulting in out of memory errors. You can always alter the heap size using LS_HEAP_SIZE environment variable.
  • Adding --debug to --configtest now shows the configuration in blocks annotated by source config file. Very useful when using multiple config files in a directory.

Feedback

Many thanks to our users and contributors for making 2.1.0 a successful release. Please download the GA binaries and give it a spin! Let us know what you think on our forum, IRC, and GitHub.