Logstash 7.7.0 Release Notesedit

New features and improvementsedit

Improving Loggingedit

Continuing with the Logging improvements made in the past couple of releases, log messages have been expanded to more accurately display information about plugins. These changes allow users to better identify log messages with plugins or pipelines. #11078, #11593, #11567

Changes include:

  • Added plugin.name to all log entries.
  • Added plugin.id and pipeline.id to docker images.

Improving the Java (Pipeline) Execution Engineedit

The improvements available in Logstash 7.7 give users more control over event ordering for single worker pipelines.

Historically, Logstash event ordering between an input plugin and output plugin were preserved when a single worker executed a pipeline. This behaviour was never officially documented, but it was a feature that many users relied upon.

When the Java Execution engine was introduced, it broke the single worker ordering preservation. This has now been fixed with the introduction of the pipeline.ordered setting. #11552, #11710

The pipeline.ordered setting allows three modes:

  • auto (default) - to automatically enable ordering if there’s only one pipeline worker.
  • true - enforces ordering while preventing logstash from starting if there are multiple workers.
  • false - ordering will not be guaranteed.

Notable issues fixededit

  • Fixed: Add sudo into the rpm import of GPG-KEY #11684
  • Fixed: Use lightweight regex matching (which does not depend on frames) #11653
  • Fixed: Issue where monitoring API does not properly resolve ids including environment variable expansion #11592
  • Fixed: Change Javadoc to conform to updated requirements for JDK13 #11642

Logstash Plugin changesedit

Grok Filter

  • Feature: Added support for placing matches into a target namespace #156

Xml Filter

  • Feature: Added parser_options to allow for more control of the parsing process. Includes the option of strict parsing to force the XML parser to fail early when parsing invalid XML. #68

Azure_event_hubs Input

  • Fixed: Now honors max_batch_size setting - previously this was ignored #52
  • Changed: The default number of threads is now 16 instead of `4`to match the default number from the Azure-Sdk EventProcessorHost #54
  • Refactor: scope and review global java_imports #57
  • [DOC] Changed documentation to update the default number of threads #55
  • [DOC] Added clarification for threads parameter #50

Elasticsearch Input

  • Feature: Added option to specify proxy for Elasticsearch #114

S3 Input

  • Feature: Added support for including objects restored from Glacier or Glacier Deep #199
  • Feature: gzip_pattern option, enabling more flexible determination of whether a file is gzipped #165
  • Refactor: log exception: class + unify logging messages a bit #201

S3 Output

  • Feature: Added retry_count and retry_delay config. This allows opting out from infinite upload retries under error conditions. #218
  • [DOC] Updated setting descriptions for clarity #219 and #220