Logstash 7.7.0 Release Notesedit

New features and improvementsedit

Improving Loggingedit

Continuing with the Logging improvements made in the past couple of releases, log messages have been expanded to more accurately display information about plugins. These changes allow users to better identify log messages with plugins or pipelines. #11078, #11593, #11567

Changes include:

  • Added plugin.name to all log entries.
  • Added plugin.id and pipeline.id to docker images.

Improving the Java (Pipeline) Execution Engineedit

The improvements available in Logstash 7.7 give users more control over event ordering for single worker pipelines.

Historically, Logstash event ordering between an input plugin and output plugin were preserved when a single worker executed a pipeline. This behaviour was never officially documented, but it was a feature that many users relied upon.

When the Java Execution engine was introduced, it broke the single worker ordering preservation. This has now been fixed with the introduction of the pipeline.ordered setting. #11552, #11710

The pipeline.ordered setting allows three modes:

  • auto (default) - to automatically enable ordering if there’s only one pipeline worker.
  • true - enforces ordering while preventing logstash from starting if there are multiple workers.
  • false - ordering will not be guaranteed.

Notable issues fixededit

  • Fixed: Add sudo into the rpm import of GPG-KEY #11684
  • Fixed: Use lightweight regex matching (which does not depend on frames) #11653
  • Fixed: Issue where monitoring API does not properly resolve ids including environment variable expansion #11592
  • Fixed: Change Javadoc to conform to updated requirements for JDK13 #11642

Known issueedit

Performance regression. A potential performance regression may affects some users. This issue can cause a slowdown on pipeline compilation when multiple pipelines are in use. We believe the issue was introduced in 7.7.0. This issue is currently being tracked and investigated in #12031

This issue seems to be affecting only big pipeline installations (that is, big pipeline definitions when multiple pipelines are defined). Symptoms include increased startup time and the appearance that Logstash is not responding to input events.

If you believe this issue is affecting you, we recommended that you downgrade to 7.6.2 while we continue to investigate and provide a resolution.

Logstash Plugin changesedit

Grok Filter

  • Feature: Added support for placing matches into a target namespace #156

Xml Filter

  • Feature: Added parser_options to allow for more control of the parsing process. Includes the option of strict parsing to force the XML parser to fail early when parsing invalid XML. #68

Azure_event_hubs Input

  • Fixed: Now honors max_batch_size setting - previously this was ignored #52
  • Changed: The default number of threads is now 16 instead of `4`to match the default number from the Azure-Sdk EventProcessorHost #54
  • Refactor: scope and review global java_imports #57
  • [DOC] Changed documentation to update the default number of threads #55
  • [DOC] Added clarification for threads parameter #50

Elasticsearch Input

  • Feature: Added option to specify proxy for Elasticsearch #114

S3 Input

  • Feature: Added support for including objects restored from Glacier or Glacier Deep #199
  • Feature: gzip_pattern option, enabling more flexible determination of whether a file is gzipped #165
  • Refactor: log exception: class + unify logging messages a bit #201

S3 Output

  • Feature: Added retry_count and retry_delay config. This allows opting out from infinite upload retries under error conditions. #218
  • [DOC] Updated setting descriptions for clarity #219 and #220