Logstash 7.9.0 Release Notesedit

New features and enhancementsedit

ECS support in Elasticsearch output pluginedit

This release is the first step toward Elastic Common Schema (ECS) support in Logstash. With 7.9, you can configure the Elasticsearch output plugin to manage index templates that are compatible with the Elastic Common Schema (ECS). The ECS compatibility setting in the Elasticsearch output plugin makes this possible.

See Compatibility with the Elastic Common Schema (ECS) in the Elasticsearch output plugin docs for more information.

Expanded API key supportedit

With this release, we’ve continued expanding support for Elasticsearch API keys. Support for API keys in the Elasticsearch output plugin arrived in Logstash 7.8.0. Logstash 7.9.0 introduces support for Elasticsearch API keys in the Elasticsearch input plugin, the Elasticsearch filter plugin, and Logstash monitoring and management.

Check out Grant access using API keys for more information about using API keys with Logstash and Elasticsearch. Implementation details are in #11953.

Aarch64 (ARM64) support (experimental)edit

Logstash runs on arm machines! We have tested Logstash against arm64, and we are looking to make docker and other images available soon.

ARM artifacts are not yet supported for production, and we’re offering them as "experimental" to early adopters.

Improved support in App Search outputedit

We replaced the deprecated Java client library for the Elastic App Search output plugin with the Ruby client library, and expanded integration testing. These changes provide a foundation for expanding App Search integration and quality assurance in future releases.

Improvements to persistent queue (PQ)edit

We’ve enhanced persistent queues to better manage exceptions and error handling which could sometimes result in a LockException when the queue file lock was not properly released. Under some conditions, a complex pipeline that is slower to initialize could be recreated when it was not done initializing, causing a LockException. Implementation details are in #12023.

These changes result in better stability of persistent queues.

Improvements to pipeline workers error handlingedit

Worker threads were not correctly monitored for a worker loop exception resulting in a complete logstash crash upon any exception even when multiple pipelines are running. Now only the failed pipeline is terminated. If pipeline reloading is enabled, you can edit the config and have the failed pipeline reloaded. Implementation details are in #12019 and #12038.

Performance improvement on startup and pipeline restartsedit

This release contains several optimizations to pipeline compilation, an essential step of the pipeline initialization process. These changes significantly improve startup and pipeline-restart performance for complex pipelines. (For technical details, check out this PR: #12060.)

From our tests in three different pipelines with eight workers each, we have seen times decrease from 9 - 28 minutes to around 1 minute.

To aid the development of pipelines, especially the performance impact of compilation, Logstash now reports the time taken to compile each pipeline as a log entry such as:

[2020-08-12T14:10:29,388][INFO ][logstash.javapipeline  ][main] Pipeline Java execution initialization time {"seconds"=>0.7}

Performance improvements and notable issues fixededit

  • Support white space as a delimiter on list-type params #12051. Resolves #6366 and #8157.
  • Support using unix pipe as local config file #11109
  • Logging improvements

    • Display Java pipeline initialization time to help with troubleshooting and diagnostics #11749
    • Logging framework enhancement to allow more finetuned logging #11853
    • Better logging after definition improvements and script routes in log4j #11929 and #11992
    • Improved Logstash startup logging to ensure that starting logstash entry happens before any other log entries #12086
  • Fix: Add back pipelines queue.data and queue.capacity subdocuments for _node/stats #11923
  • Fix: Avoid reloading pipelines that have no changes #12009
  • Fix: Removed unnecessary calls that, under some circumstances, could cause pipeline startup issues for pipelines that were slow to initialize #12034
  • Fix: Allow trailing newlines in config fragments to resolve an issue in which split configs were corrupted when merged #12161
  • Fix: Resolve issue in which pipeline init fails for a slow pipeline when monitoring is enabled #12034
  • Fix: Ignore default username when no password is set for monitoring and management #12094
  • Refactor code refactor to launch ruby thread from ruby code instead of java (as a workaround for jruby bug) #11900
  • Updates to dependencies

    • Update log4j dependency to 2.13.3
    • Update jruby to

Plugin releasesedit

Rubydebug Codec - 3.1.0

  • Replace stale awesome_print library with maintained fork called amazing_print #8

Elasticsearch Filter - 3.9.0

  • Add support to define a proxy with the proxy config option #134
  • Added api_key support #132
  • [DOC] Removed outdated compatibility notice #131

Memcached Filter - 1.1.0

  • Added better exception handling #25

Elasticsearch Input - 4.7.0

  • Added api_key support #131

File Input - 4.2.1

  • Fix: Skip sincedb eviction if read mode completion deletes file during flush #273
  • Fix: Watched files performance with huge filesets #268
  • Updated logging to include full traces in debug (and trace) levels

Imap Input - 3.1.0

  • Adds an option to recursively search the message parts for attachment and inline attachment filenames. If the save_attachments option is set to true, the content of attachments is included in the attachments.data field. The attachment data can then be used by the Elasticsearch Ingest Attachment Processor Plugin #48

Kafka Integration - 10.4.0

  • Added the input isolation_level to allow fine control of whether to return transactional messages #44
  • Added the input and output client_dns_lookup parameter to allow control of how DNS requests are made

Rabbitmq Integration - 7.1.0

  • Added support in Output plugin for sprintf templates in values provided to message_properties #8
  • Added support for extended metadata including raw payload to events generated by the Input Plugin #13
  • Fixes an issue with custom port assignment, in which the custom port was not being applied when more than one host was supplied #12
  • Fixes bug where attempting to read from undeclared exchange resulted in infinite retry loop #10
  • Fixes bug where failing to establish initial connection resulted in a pipeline that refused to shut down #11

Elastic_app_search Output - 1.1.0

  • Switched AppSearch client library from Java to Ruby #12
  • Covered with integration tests and dockerized local AppSearch server instance.

Elasticsearch Output - 10.6.1

  • Fixed an issue introduced in 10.6.0 that broke Logstash Core’s monitoring feature when this plugin is run in Logstash 7.7-7.8. #953
  • Added ecs_compatiblity mode, for managing ECS-compatable templates #952