The following list are changes in 8.0.0 as compared to 7.17.0, and combines release notes from the 8.0.0-alpha1, -alpha2, -beta1, -rc1 and -rc2 releases.
Many plugins can now be run in a mode that avoids implicit conflict with the Elastic Common Schema (ECS).
This mode is controlled individually with each plugin’s ecs_compatibility option, which defaults to the value of the Logstash pipeline.ecs_compatibility setting.
In Logstash 8, this compatibility mode will be on-by-default for all pipelines.
If you wish to lock in a pipeline’s behavior from Logstash 7.x before upgrading to Logstash 8,
you can set
pipeline.ecs_compatibility: disabledto its definition in
pipelines.yml(or globally in
- Starting from Logstash 8.0, the minimum required version of Java to run Logstash is Java 11. By default, Logstash will run with the bundled JDK, which has been verified to work with each specific version of Logstash, and generally provides the best performance and reliability.
Support for using
JAVA_HOMEto override the path to the JDK that Logstash runs with has been removed for this release. In the
8.xrelease, users should set the value of
LS_JAVA_HOMEto the path of their preferred JDK if they wish to use a version other than the bundled JDK. The value of
JAVA_HOMEwill be ignored.
- The Java Execution Engine has been the default engine since Logstash 7.0, and works with plugins written in either Ruby or Java. Removal of the Ruby Execution Engine will not affect the ability to run existing pipelines. #12517
- We have added support for UTF-16 and other multi-byte-character when reading log files. #9702
config.field_reference.parserhas been removed. The Field Reference parser interprets references to fields in your pipelines and plugins. Its behavior was configurable in 6.x, and since 7.0 allowed only a single option:
strict. 8.0 no longer recognizes the setting, but maintains the same behavior as the
strictsetting. Logstash rejects ambiguous and illegal inputs as standard behavior. #12466
For a more detailed view of these changes please check Breaking changes in 8.0.
- As processing times speed up, millisecond granularity is not always enough. Inbound data increasingly has sub-millisecond granularity timestamps. The pull request #12797 allows the internal mechanisms of Logstash that hold moment-in-time data - such as the Logstash Event, the Persistent Queue, the Dead Letter Queue and JSON encoding/decoding - to have nanosecond granularity.
- We have added another flag to the Benchmark CLI to allow passing a data file with previously captured data to the custom test case. This feature allows users to run the Benchmark CLI in a custom test case with a custom config and a custom dataset. #12437
Logstash 8.0.0 includes the same versions of all bundled plugins as Logstash 7.17.0. If you upgrade to 7.17 before upgrading to 8.0 (as recommended), you won’t see any changes to plugin versions.
Clone Filter - 4.2.0
- Added support for ECS v8 as alias for ECS v1 #27
Geoip Filter - 7.2.11
Improved compatibility with the Elastic Common Schema #206
Added support for ECS’s composite
US-WA), which replaces the non-ECS
WA) as a default field with City databases. To get the stand-alone
region_codein ECS mode, you must include it in the
- [DOC] Improve ECS-related documentation
- Added support for ECS’s composite
- [DOC] Air-gapped environment requires both ASN and City databases #204
Http Filter - 1.2.1
Ruby Filter - 3.1.8
- [DOC] Added doc to describe the option `tag_with_exception_message`https://github.com/logstash-plugins/logstash-filter-ruby/pull/62[#62]
- Fix SyntaxError handling so other pipelines can shut down gracefully #64
Useragent Filter - 3.3.3
- Docs: mention added fields in 3.3 with a note #78
Exec Input - 3.4.0
- Feat: adjust fields for ECS compatibility #28
Plugin will no longer override fields if they exist in the decoded payload (It no longer sets the
hostfield if decoded from the command’s output)
Gelf Input - 3.3.1
Fix: safely coerce the value of
_@timestampto avoid crashing the plugin #67
Generator Input - 3.1.0
- Feat: adjusted fields for ECS compatibility #22
- Fix: do not override the host field if it’s present in the generator line (after decoding)
- Fix: codec flushing when closing input
Imap Input - 3.2.0
- Feat: ECS compatibility #55
Fix: plugin should not close
$stdin, while being stopped
Jms Input - 3.2.1
- Fix: improve compatibility with MessageConsumer implementations #51, such as IBM MQ.
Test: Fix test failures due to ECS compatibility default changes in
8.xof logstash #53
- Feat: event_factory support + targets to aid ECS #49
- Fix: when configured to add JMS headers to the event, headers whose value is not set no longer result in nil entries on the event
Fix: when adding the
jms_reply_toheader to an event, a string representation is set instead of an opaque object.
Pipe Input - 3.1.0
- Feat: adjust fields for ECS compatibility #19
S3 Input - 3.8.3
Snmp Input - 1.3.1
Snmptrap Input - 3.1.0
- Feat: ecs_compatiblity support + (optional) target #37
Syslog Input - 3.6.0
- Add support for ECS v8 as alias to v1 implementation #68
Twitter Input - 4.1.0
- Feat: optional target + ecs_compatibility #72
Unix Input - 3.1.1
Jdbc Integration - 5.2.2
- Feat: name scheduler threads + redirect error logging #102
- Refactor: isolate paginated normal statement algorithm in a separate handler #101
jdbc_paging_modeoption to choose if use
explicitpagination in statements and avoid the initial count query or use
autoto delegate to the underlying library #95
Several improvements to Java driver loading
NOTE: A fatal driver error will no longer keep reloading the pipeline and now leads to a system exit.
- Fix: regression due returning the Java driver class #98
Kafka Integration - 10.9.0
Refactor: leverage codec when using schema registry
schema_registry_urlparsed the payload as JSON even if
codec => 'plain'was explicitly set, this is no longer the case. #106
Cloudwatch Output - 3.0.10
- Fix: an old undefined method error which would surface with load (as queue fills up)
- Deps: unpin rufus scheduler #20
Elasticsearch Output - 11.4.1
Feat: upgrade manticore (http-client) library #1063
- the underlying changes include latest HttpClient (4.5.13)
resolves an old issue with
ssl_certificate_verification => falsestill doing some verification logic
Updates ECS templates #1062
- Updates v1 templates to 1.12.1 for use with Elasticsearch 7.x and 8.x
- Updates BETA preview of ECS v8 templates for Elasticsearch 7.x and 8.x
- Feat: add support for traces data stream type #1057
- Refactor: review manticore error handling/logging, logging originating cause in case of connection related error when debug level is enabled. Java causes on connection related exceptions will now be extra logged when plugin is logging at debug level #1029
ECS-related fixes #1046
- Data Streams requirement on ECS is properly enforced when running on Logstash 8, and warned about when running on Logstash 7.
- ECS Compatibility v8 can now be selected
Core Patterns - 4.3.2
Fix: typo in
BIN9_QUERYLOGpattern (in ECS mode) #307