Certificate verification with internal monitoring. We fixed a bug in the monitoring pipeline that caused it to pass monitoring data to Elasticsearch with certificate verification disabled. Logstash internal monitoring had been sending monitoring metadata (such as pipeline throughput metrics) to Elasticsearch without verifying the recipient. #12749
For information: CVE-2021-22138.
We’ve done more work to help ease your transition to Elastic Common Schema (ECS). This release extends ECS work in previous releases. Here’s a recap:
- ECS support in Elasticsearch output plugin (7.9). The elasticsearch output plugin can manage index templates that are compatible with ECS. For more info, see Compatibility with the Elastic Common Schema (ECS).
Pipeline level ECS compatibility (7.10). The
pipeline.ecs_compatibilitysetting lets users control ECS compatibility for all plugins in a pipeline at once instead of configuring each instance manually. This setting lets users lock in a specific behavior in advance of their next major version upgrade.
ECS compatibility is off-by-default in Logstash 7.x, but will be on-by-default in Logstash 8.0.
The grok filter plugin offers a new set of patterns to make event field names ECS-compliant. (No worries if you’re not ready to transition yet. The complete set of legacy patterns is still available and continues to be the default for Logstash 7.x.)
The ECS pattern set has an equivalent for each pattern in the legacy set, and is a drop-in replacement. Use the ecs_compatibility setting when you’re ready to switch modes.
The beats input plugin is now ECS-compliant. It adds two fields related to the event: the deprecated host which contains the hostname, and the ip_address containing the remote address of the client’s connection. When ECS compatibility mode is enabled these fields are moved to ECS-compatible namespace.
Logstash introduces support for JDK 15! You need to update settings in
- you are upgrading from Logstash 7.11.x (or earlier) to 7.12 or later, AND
- you are using JDK 15 or later.
Unless both of these conditions apply, you don’t need to adjust settings because of the upgrade. See Using JDK 15 for more information.
We’ve added support for conditional settings and behavior, dependent on the JVM
version. Now you can configure different settings for different JVM versions.
Here is an example from the default
## GC configuration 8-13:-XX:+UseConcMarkSweepGC 8-13:-XX:CMSInitiatingOccupancyFraction=75 8-13:-XX:+UseCMSInitiatingOccupancyOnly
This example sets garbage collection (GC) values for JDK 8-13 only. Those settings don’t apply to JVM 14 and above.
This feature is available for any setting in the
jvm.options file, and aligns
more closely with the Elasticsearch implementation of jvm settings.
Support for 64-bit ARM architectures on Linux is now in beta, with downloadable artifacts and docker images available.
Pipeline loading and monitoring improvements
We’ve made changes to start the webserver that exposes the Logstash metrics API earlier in the startup process. For slow starting pipelines, this would cause error messages to appear in the Logstash logs, and cause delays to the availability of the metrics API. #12571
Windows startup fixes
We’ve fixed an issue where Logstash would crash when attempting to start using the bundled JDK when Logstash was located in a folder where the folder name contained spaces #12585
Elasticsearch Filter - 3.9.3
Geoip Filter - 6.0.5
Grok Filter - 4.4.0
- Feat: ECS compatibility support. Add (built-in) patterns definitions that are fully Elastic Common Schema compliant. #162
Metrics Filter - 4.0.7
- [DOC] Fixed typo in documentation
Beats Input - 6.1.0
Elasticsearch Input - 4.9.1
Http Input - 3.3.7
- Feat: improved error handling/logging/unwraping #133
Redis Input - 3.6.0
- Remove ruby pipeline dependency. Starting from Logstash 8, Ruby execution engine is not available. All pipelines should use Java pipeline #84
Syslog Input - 3.4.5
- Added support for listening on IPv6 addresses
Tcp Input - 6.0.7
- Fix: reduce error logging (to info level) on connection resets #168
- Refactor: only patch Socket classes once (on first input)
Refactor: use a proper log4j logger (in Java to avoid surprises when unwrapping
Udp Input - 3.4.0
Kafka Integration - 10.7.1
- Fix: dropped usage of SHUTDOWN event deprecated since Logstash 5.0 #71
Rabbitmq Integration - 7.2.0
- Remove ruby pipeline dependency. Starting from Logstash 8, Ruby execution engine is not available. All pipelines should use Java pipeline #39
Ecs_compatibility_support Mixin - 1.1.0
Support Mixin for ensuring a plugin has an
ecs_compatibilitymethod that is configurable from an
ecs_compatibilityoption that accepts the literal
disabledor a v-prefixed integer representing a major ECS version (e.g.,
v1), using the implementation from Logstash core if available.
Cloudwatch Output - 3.0.9
- Fix: dropped usage of SHUTDOWN event deprecated since Logstash 5.0 #18
Elasticsearch Output - 10.8.2
- [DOC] Update links to use shared attributes #985
Lumberjack Output - 3.1.8
- Fix: dropped usage of SHUTDOWN event deprecated since Logstash 5.0 #31
S3 Output - 4.3.3
- [DOC] Update links to use shared attributes #230
Core Patterns - 4.3.0
With 4.3.0 we’re introducing a new set of pattern definitions compliant with Elastic Common Schema (ECS), on numerous places patterns are capturing names prescribed by the schema or use custom namespaces that do not conflict with ECS ones.
Changes are backwards compatible as much as possible and also include improvements to some of the existing patterns.
Besides fields having new names, values for numeric (integer or floating point) types are usually converted to their
numeric representation to ease further event processing (e.g.
http.response.status_code is now stored as an integer).
to leverage the new ECS pattern set in Logstash a grok filter upgrade to version >= 4.4.0 is required.
in ECS mode we dropped the (incomplete) attempt to capture
S3_ACCESS_LOGwill handle up-to-date S3 access-log formats (6 new field captures at the end) Host Id → Signature Version → Cipher Suite → Authentication Type → Host Header → TLS version
ELB_ACCESS_LOGwill handle optional (
-) in legacy mode
null values such as
-1time values (e.g.
request_processing_time) are not captured in ECS mode
- in ECS mode we dropped the (incomplete) attempt to capture
Fix: improve matching of
BACULA_patterns to handle (optional) spaces
BACULA_LOGJob Id: X prefix as optional
- Fix: legacy matching of BACULA fatal error lines
- Fix: improve matching of
querytypewas further split into multiple fields as:
BIND9patterns (legacy as well) were adjusted to handle Bind9 >= 9.11 compatibility
BIND9_QUERYLOGBASEwas introduced for potential re-use
BRO_patterns are stricter in ECS mode - won’t mistakenly match newer BRO/Zeek formats
place holders such as
-null values won’t be captured
BRO_pattern has a newer
ZEEK_variant that supports latest Zeek 3.x versions e.g.
ZEEK_HTTPas a replacement for
BRO_HTTP(in ECS mode only), there’s a new file zeek where all of the
ZEEK_XXXpattern variants live
EXIM_MESSAGE_ARRIVAL) to match message arrival log lines - in ECS mode!
IPTABLESpattern which is re-used within
SHOREWALLnow supports IPv6 addresses (in ECS mode - due
timestampfields will be captured for
SFW2in legacy mode as well
SHOREWALLbecame less strict in containing the
NETSCREENSESSIONLOGproperly handles optional
xlate_type(legacy) CISCO fields are not captured in ECS mode
SYSLOGFACILITYtype casts facility code and priority in ECS mode
SYSLOGTIMESTAMPwill be captured (from
- Fix: e-mail address’s local part to match according to RFC (#273)
- several ECS-ified fields will be type-casted to integer in ECS mode e.g. haproxy.bytes_read
fields containing null value (
-) are no longer captured (e.g. in legacy mode
captured_request_cookiegets captured even if
optional fields (e.g.
user_agent) are only captured when not null (
clientportin legacy mode) is considered optional
dropped raw data (
rawrequestlegacy field) in ECS mode
- Fix: HTTPD_ERRORLOG should match when module missing (#299)
- optional fields (e.g.
JAVASTACKTRACEPART's matched line number will be converted to an integer
CATALINALOGmatching was updated to handle Tomcat 7/8/9 logging format
TOMCATLOGhandles the default Tomcat 7/8/9 logging format
old (custom) legacy TOMCAT format is handled by the added
TOMCAT_DATESTAMPstill match the legacy format, however this might change at a later point - if you rely on the old format use
integer fields (e.g.
juniper.srx.elapsed_time) are captured as integer values
- integer fields (e.g.
SYSLOG5424LINEcaptures (overwrites) the
messagefield instead of using a custom field name
regardless of the format used, in ECS mode, timestamps are always captured as
fields such as
process.pidare converted to integers
- mcollective-patterns file was removed, it’s all one mcollective in ECS mode
pidpreviously) is not type-casted to an integer
numeric fields such as
nagios.log.attemptare converted to integer values in ECS mode
- numeric fields such as
request duration times from
RAILS3log will be converted to floating point values
- request duration times from
bytesare type-casted to int
SQUID3pattern won’t capture null (-)
- Fix: allow to parse SQUID log with status 0 (#298)
- Fix: handle optional server address (#298)
- Fix: Java stack trace’s JAVAFILE to better match generated names
- Fix: match Information/INFORMATION in LOGLEVEL #274
- Fix: NAGIOS TIMEPERIOD unknown (from/to) field matching #275
- Fix: HTTPD access log parse failure on missing response #282
Fix: UNIXPATH to avoid DoS on long paths with unmatching chars #292
For longer paths, a non matching character towards the end of the path would cause the RegExp engine a long time to abort. With this change we're also explicit about not supporting relative paths (using the `PATH` pattern), these won't be properly matched.
- Feat: allow UNIXPATH to match non-ascii chars #291