Product release

Logstash 5.0.0-alpha1 released

We are excited to announce the availability of the first pre-release version for Logstash 5.0.0. Wait, what? 5.0.0? Yep, the next major version will be 5.0.0. In case you missed our announcement during Elastic{ON} 2016, we've decided to align all the components of the Elastic Stack to a single version. You can find the release notes here or head straight to the downloads page if you can't wait to try it out.

IMPORTANT: This is an alpha release and is intended for testing purposes only. There is no guarantee that Logstash 5.0.0-alpha1 will be compatible with other pre-releases, and 5.0.0 GA.

Monitor All The Things!

Over the last couple of releases, we've been working hard to make Logstash easier to manage operationally. Our goal is to make it a breeze to configure and deploy multiple Logstash instances, and in 5.x we'll continue to add features to support this initiative. In this release, we are pleased to introduce the first set of monitoring APIs which will provide more visibility into the Logstash pipeline.

Event Stats

Want to measure the number of events processed by Logstash? We've got an API for that now -

curl localhost:9600/_node/stats?pretty
{
  "events" : {
    "in" : 15000,
    "filtered" : 14875,
    "out" : 14000
  },
  "jvm" : {
    "timestamp" : 1459393492170,
    "uptime_in_millis" : 18731,
    "mem" : {
      "heap_used_in_bytes" : 245625232,
      "heap_used_percent" : 11,
      ....

Hot Threads

If you've used the hot threads API in Elasticsearch, you know how useful it can be for debugging hotspots in your cluster. Well, now, you can do the same thing in Logstash! The new hot threads API provides stacktrace from top 3 Java threads that are consuming the most CPU.

curl localhost:9600/_node/hot_threads?human
 Hot threads at 2016-03-30T20:08:22-07:00, busiestThreads=3:
     5.22 % of of cpu usage by waiting thread named '[main]>worker3'
        java.lang.Object.wait(Native Method)
        java.lang.Object.wait(Object.java:460)
        org.jruby.RubyThread$SleepTask.run(RubyThread.java:1050)
        org.jruby.RubyThread.executeBlockingTask(RubyThread.java:1066)
        org.jruby.RubyThread.wait_timeout(RubyThread.java:1414)
        org.jruby.ext.thread.Queue.pop(Queue.java:152)
        org.jruby.ext.thread.Queue.pop(Queue.java:127)
        org.jruby.ext.thread.SizedQueue.pop(SizedQueue.java:111)
        org.jruby.ext.thread.SizedQueue$INVOKER$i$pop.call(SizedQueue$INVOKER$i$pop.gen)
        org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
     2.44 % of of cpu usage by timed_waiting thread named '[main]-pipeline-manager'
        java.lang.Object.wait(Native Method)
        ....

We are big fans of APIs here at Elastic — we love providing access to all kind of metrics into the working of our software. Next up, in Logstash, we are targeting stats at a plugin level granularity. For example, it would be great to know how long (on average) an event spends in grok filter, geoip filter etc. You'll be able to do this soon. How 'bout a monitoring UI, you ask? A Kibana app to visualize all these metrics across a period of time is in the works! So, stay tuned!

Kafka 0.9 Support

Apache Kafka had released version 0.9 couple of months ago which brings in new security features (SSL, client based auth, access control), improved consumer API, and much more. This Logstash release provides support for SSL encryption and client auth features in Kafka. Some of the configuration options in the consumer have changed — as such this plugin is not backward compatible.

Note: To use these new security features, you'd need to upgrade the Kafka broker to 0.9

bin/plugin is now bin/logstash-plugin

We'd like to note that bin/plugin command — which is used to manage plugins — has been renamed to bin/logstash-plugin command. The main reason was to prevent PATH being polluted when other components of the Elastic Stack are installed on the same instance. Also, this new command will be enhanced in upcoming versions to work with cross-component plugin packs.

Elasticsearch 5.0.0 Compatibility

This alpha1 release works out of the box with Elasticsearch 5.0.0-alpha1! Also, you'll be able to use any 2.x Elasticsearch version with Logstash 5.0.0-alpha1. So, go ahead and give it a try!

Moar Features

In case you missed it, 5.0.0-alpha1 has all the goodies that came in version 2.3.0. We're excited about new features like dynamic config reloading and environment variables support in the configuration. Plus, there was a big bump in performance in 2.3.0! Read about all these features in our 2.3.0 release post.

Known Change

Be aware that if you're using the Ruby filter, your code may not execute as previously expected in this release. We're working on a fix and please consult the workarounds here for now.

Feedback

Please try this pre-release and let us know what you think! Your feedback and contribution is really important as we continue to iterate on 5.0.0! You can create issues on our GitHub repo, find us on our forum, or hang out with us on IRC (#logstash).