Product release

Logstash 5.0.0-alpha3 released

We are happy to announce the third pre-release (alpha3) in the 5.0.0 series! Please check the release notes for details, or jump directly to our downloads page if you can't wait to try it out!

IMPORTANT: This is an alpha release and is intended for testing purposes only. Please do not deploy in production.

This release is loaded with goodies that are highlighted below:

Improved Release Packages

Logstash release packages (Debian, RPM) has been given an overhaul in this release. Most importantly, this is a breaking change, so please read ahead! Previously, Logstash used /opt directory to install the binaries, whereas Elasticsearch used /usr/share, and /var. To make user experience consistent across our products — a theme you'll hear a lot in our 5.0.0 releases — we've changed Logstash to reflect Elasticsearch RPM and DEB layouts.

Secondly, we added systemd and upstart support to run Logstash as a service. Today, a lot of Linux distributions (Debian Jessie, Ubuntu 15.10+, and many of the SUSE derivatives) use systemd as the default service manager, and we received many requests from our users to support systemd. Oh, and all the JVM options have been extracted to a separate file, jvm.options à la Elasticsearch, so you can easily override or pass in new JVM options by modifying a single clean file, instead of digging through BASH scripts.

Settings File

Say hello to logstash.yml file. Yep, this is a new one-stop-place to configure all application-level settings for the Logstash process. With all the new features we've been adding, our list of CLI options kept growing! To stop cluttering the CLI, and to make bootstrap easier, we've introduced a new settings file which mirrors the CLI. Most of the long form options have been renamed to adhere to the yml dot notation, so this is filed under breaking changes as well. Short form options have remained the same. If you installed an RPM or DEB package, look for this file in /etc/logstash/config, or otherwise in $LS_HOME/config. Just to be clear, the pipeline configuration where you specify the input, filters and outputs is separate from the settings file.

Java Event API

Remember we released a new feature called Java Event — a rewrite of Event handling in pure Java — in version 2.3? Say you were using the Ruby Filter; in some cases this change could have been backward incompatible because this filter allows users to manipulate the Event object directly using Ruby’s hash paradigm. To mitigate this, we rolled back this feature when we released version 2.3.1. Now, in 5.0.0 — a major release — we've handled this correctly by introducing non-ambiguous APIs to interface with the Event object. This change mostly affects plugin developers who write and maintain custom plugins — all the default packaged plugins have been updated to use the new APIs. If you maintain a custom plugin, or plan to write a new one, this old style of accessing Events directly using the Ruby hash convention will not work anymore. Please use these setter/getter APIs if you need access to data inside of the Event object.

Plugin Generator

Kibana has it. Beats has it. And now Logstash has it too! We're talking about a plugin generator tool that makes it easier to develop new plugins for Logstash. Previously we've recommended developers to clone/fork the examples repositories, but now you can simply do:

bin/logstash-plugin generate --type input --name xkcd --path ~/ws/elastic/plugins

This subcommand bootstraps a new plugin logstash-input-xkcd with the right directory structure and all the required files (templates) for you to start developing this plugin right away. So, go on, create that xkcd input to stream those fine comic strips to Kibana!

Wait, you though we'd just talk about xkcd and not drop a strip here?!

regular_expressions.png

Reference: https://xkcd.com/208/

Want to be that superhero? Use Grok!

But Wait There's More!

We said this was a loaded release before..

  • JSON Logging: Back in the day, Logstash used to emit its logs in JSON format. Because of endless encoding issues and crashes in logger, we had to roll it back. We've worked through most of these issues, so we're bringing structured logging back. Starting with alpha3, you can tell Logstash to switch to JSON logging by setting log.format to json.
  • We've exposed more operational information to help run Logstash in production. _node/stats now shows file descriptors and cpu information.
"process" : {
   "peak_open_file_descriptors" : 48,
   "max_file_descriptors" : 10240,
   "open_file_descriptors" : 48,
   "mem" : {
     "total_virtual_in_bytes" : 5274738688
   },
   "cpu" : {
     "total_in_millis" : 20792844000,
     "percent" : 23
   }
 }
  • Keep Calm and CTRL-C Have you ever tried to shut down Logstash with CTRL-C when using the stdin input? You wouldn't have been able to. Previously to shut down Logstash in this configuration, you had to either use  CTRL-D or a CTRL-C followed by a ENTER key sequence. The  additional ENTER was needed to unblock the reads in stdin which  usually led to confused new users, many duplicate GitHub issues,  and not to forget the jammed/broken keyboards. Did we mention frustrated users? With alpha3, simply do a CTRL-C and Logstash will gracefully stop (#1769).

Feedback

Please try alpha3 and let us know what you think! You can even win a ticket for Elastic{ON} '17 when you help test our pre-releases! Your feedback and contribution is really important as we continue to iterate on 5.0.0. You can create issues on our GitHub repo, find us on our forum, or hang out with us on IRC (#logstash).