Product release

Logstash 5.0.0-alpha4 released

We are pleased to announce that Logstash 5.0.0-alpha4 has been released today. You can review the changes here, or jump directly to downloads.

Note: This is an alpha release. Please do not use this in your production environment.

Moar Metrics, Moar Visibility

Building on the systems level monitoring APIs that were exposed in previous alphas, we have now added per-plugin metrics. The question you've always wanted to ask Logstash: where are events spending the most time? In alpha4, every filter and output plugin has been enhanced to expose wall-clock time spent by events flowing through it. It doesn't stop with timing info — plugins like Grok, Date and GeoIP filters now expose number of failures, successes, cache stats and more. So, yes, debugging your four page Logstash configuration just got a lot easier!

curl localhost:9600/_node/stats/pipelines?pretty
{  
   "plugins":{  
      "filters":[  
         {  
            "id":"grok_8b1226e2-e714-444c-9774-0135833457d3",
            "name":"grok",
            "events":{  
               "in":122,
               "duration_in_millis":75,
               "out":122
            }
            "failures": 3
            "matches" : 1,
            "patterns_per_field" : {
                "message" : 1
            }
         }
      ],
      "output":[  
         {  
            "id":"elasticsearch_713f61e2-6d7a-436c-9159-b3a953b18451",
            "name":"elasticsearch",
            "events":{  
               "in":122,
               "duration_in_millis":175,
               "out":122
            }
         }
      ]
   }
}

Elasticsearch Output

  • ES output has been modified to be fully threadsafe, with the goal of optimizing for the pipeline design changes introduced in 2.2. This means internal resources can be efficiently shared among multiple output { elasticsearch {} } instances.
  • Sniffing improvements so any current connections don't have to be closed/reopened after a sniff round.
  • Introduced a connection pool to efficiently reuse connections to ES backends.
  • Added exponential backoff to connection retries with a ceiling of retry_max_interval which is the most time to wait between retries, and retry_initial_interval which is the initial amount of time to wait. retry_initial_interval will be increased exponentially between retries until a request succeeds.

Kafka 0.10 Support

Version 0.10 of Apache Kafka was released recently with many enhancements like rack awareness, better handling for protocol versions and timestamp in messages. Logstash input and output has been updated to work with 0.10.

Others

  • Made bin/logstash -V/--version fast on Unix platforms.
  • Created a new LS_HOME/data directory to store plugin states, Logstash instance UUID and more. This directory location is configurable via path.data.
  • Monitoring API: Added hostname, http_address, version as static fields for all APIs

Feedback

We welcome and appreciate all your feedback as we iterate through our pre-releases for 5.0.0! You can open issues on our GitHub page, or start a conversation on our forums. Also, do you know about our Pioneer Program