Product release

Logstash 5.0.0-beta1 released

We are pleased to announce that Logstash 5.0.0-beta1 has been released today. You can review the changes here or jump directly to downloads.

Note: This is a beta release. Please do not use this in your production environment.

Logstash's Logs

Did we say we <3 logs? The Elastic Stack is great for log management and analysis, and thousands of users use Logstash and the rest of the stack to crunch application logs of all kinds.

Logstash as a software is no exception — we emit internal logs that can be used by operators to troubleshoot issues. In this release, we've significantly improved the debugging capability of Logstash using its own logs. We've migrated to use the popular Log4j2 framework, and for our users, this means tons of new functionality:

  1. Finer logging configuration: You can now enable logs for a particular subsystem or a component. For example, if you are debugging slowness in the Elasticsearch output, you can increase logs for just this plugin.
  2. Log rotation by size and age criteria: In previous versions, Logstash wrote logs to a single log file, and users had to use an external tool called logrotate to keep this file at bay. Users can now use Log4j2's inbuilt mechanism to configure file rotation. By default, log files are rotated every day.
  3. Dynamically update log levels: To change log levels, you can modify the log4j2.properties file and restart Logstash, but that is both tedious and leads to unnecessary downtime. Instead, you can now dynamically update logging levels through the logging API. These settings are effective immediately and do not need a restart. To update logging levels, take the subsystem/module you are interested in and prepend logger. to it. For example:
PUT /_node/logging {
    "logger.logstash.outputs.elasticsearch" : "DEBUG"
}

Monitoring Enhancements

We've received good feedback about the new monitoring APIs from our pre-releases. In this release, we've incorporated most of the feedback we received:

  • Added JVM GC stats under /_node/stats/jvm
  • Added config reload stats under new resource type /_node/stats/pipeline/reloads
  • Added JVM GC strategy info under /_node/jvm

Breaking Change: New Elasticsearch Template

The index template for 5.0 has been changed to reflect Elasticsearch's mapping changes. Most importantly, the subfield for string multi-fields has changed from .raw to .keyword to match ES's default behavior. The impact of this change to various user groups are detailed below:

  1. New Logstash 5.0 and Elasticsearch 5.0 users - uses keyword from the outset for subfields. In Kibana, you would be using field.keyword to perform aggregations.
  2. Existing users with custom templates - most of these won't be impacted since you use a custom template.
  3. Existing users with default template: Logstash does not force upgrade templates if it already exists. In case you intend to move to the new template and want to use .keyword, you'll have to reindex existing data. Elasticsearch's reindexing API can help move your data from .raw subfield to .keyword.

Feedback

As we approach the 5.0 release, we welcome and appreciate all your feedback. You can open issues on our GitHub page, or start a conversation on our forums. Also, we would love for you to be a part of our Pioneer Program!