We have improved test scripts infrastructure to accept a specific Java Development Kit (JDK) present in the host. The changes include exporting the path to a JDK home into an environment variable named BUILD_JAVA_HOME and its use by shell scripts to launch test and integration scripts. This work and other JDK14 fixes pave the way for Logstash to support both AdoptOpenJDK 11 and 14 in the near future . #11786, #11839, #11935
Logstash runs on many different operating platforms and various flavors of the Java Development Kit (JDK), and we’ve expanded this coverage in 7.8.0. Logstash has introduced support for running on CentOS/RHEL 8.x and Ubuntu 20.04. We’ve added new JDK support for Zulu 11, AdoptOpenJDK 11, and Oracle/OpenJDK/AdoptOpenJDK 14.
Support for API keys was added to Elasticsearch in 6.7.0. With 7.8.0 Logstash introduces support for Elasticsearch API keys in the Elasticsearch output plugin #934.
Authentication in Elasticsearch can be done in different ways, from LDAP to SAML and others. User/password authentication makes sense for discrete users accessing Elasticsearch. For machine-to-machine communication, API key access is more common. Check out Grant access using API keys for more information about using API keys with Logstash and Elasticsearch.
Support for API keys in the Elasticsearch input and filter plugins, and the monitoring and management features will be added in upcoming releases.
Many of our users deploy Logstash and the Elastic Stack in segmented networks where one component may not be able to directly reach out to another or to the Internet. Logstash plugins, such as the elasticsearch, http and SNS outputs, support the configuration of proxy servers. Version 7.8.0 brings proxy support to monitoring and central management #11799.
Configure the proxy’s URL in your
logstash.yml file using
"xpack.monitoring.elasticsearch.proxy" (for monitoring) or
"xpack.management.elasticsearch.proxy" (for central management).
Azure and Netflow modules in Logstash have been deprecated and replaced by the Azure modules in Filebeat and Metricbeat, and the Netflow module in Filebeat. The Filebeat and Metricbeat modules are compliant with the Elastic Common Schema (ECS).
Performance regression. A potential performance regression may affect some users. This issue can cause a slowdown on pipeline compilation when multiple large pipelines are in use. We believe the issue was introduced in 7.7.0. This issue is currently being tracked and investigated in #12031
This issue seems to be affecting only big pipeline installations (that is, big pipeline definitions when multiple pipelines are defined). Symptoms include increased startup time and the appearance that Logstash is not responding to input events.
If you believe this issue is affecting you, we recommended that you downgrade to 7.6.2 while we continue to investigate and provide a resolution.
Cef Codec - 6.1.1
- Improved encoding performance, especially when encoding many extension fields #81
- Fixed CEF short to long name translation for ahost/agentHostName field, according to documentation #75
- Fixed support for deep dot notation #73
- Fixed minor doc inconsistencies (added reverse_mapping to options table, moved it to alpha order in option descriptions, fixed typo) #60
- Added reverse_mapping option, which can be used to make encoder compliant to spec #51
- Fix handling of malformed inputs that have illegal unescaped-equals characters in extension field values (restores behaviour from ⇐ v5.0.3 in some edge-cases) #56
- Fix bug in parsing headers where certain legal escape sequences could cause non-escaped pipe characters to be ignored.
- Fix bug in parsing extension values where a legal unescaped space in a field’s value could be interpreted as a field separator #54
Add explicit handling for extension key names that use array-like syntax that isn’t legal with the strict-mode field-reference parser (e.g.,
- Fix handling of higher-plane UTF-8 characters in message body
deprecated_v1_fieldsfields from deprecated to obsolete
- added mapping for outcome = eventOutcome from CEF whitepaper (ref:p26/39)
- changed rt from receiptTime to deviceReceiptTime (ref:p27/39)
- changed tokenizer to include additional fields (ad.fieldname)
delimitersetting. This allows the decoder to be used with inputs like the TCP input where event delimiters are used.
- Implements the dictionary translation for abbreviated CEF field names from chapter Chapter 2: ArcSight Extension Dictionary page 3 of 39 of the CEF specification.
_cefparsefailuretag on failed decode
- breaking: Updated plugin to use new Java Event APIs
Switch in-place sub! to sub when extracting
cef_version. new Logstash Java Event does not support in-place String changes.
- Depend on logstash-core-plugin-api instead of logstash-core, removing the need to mass update plugins on major releases of logstash
- New dependency requirements for logstash-core for the 5.0 release
encodewith escaping according to the CEF specification
sevis deprecated, use
- Plugins were updated to follow the new shutdown semantic. This allows Logstash to instruct input plugins to terminate gracefully, instead of using Thread.raise on the plugins' threads. #3895
- Dependency on logstash-core update to 2.0
Elasticsearch Filter - 3.7.1
- Fix: solves an issue where non-ascii unicode values in a template were not handled correctly #128
File Input - 4.1.18
- Fix: release watched files on completion (in read-mode) #271
Added configuration setting
check_archive_validitysettings to enable gzipped files verification. Fixes: #261
[DOC] Added clarification for settings available with
[DOC] Rearranged text and fixed formatting for
Syslog Input - 3.4.2
Tcp Input - 6.0.5
- Fix potential startup crash that could occur when multiple instances of this plugin were started simultaneously #155
Kafka Integration - 10.2.0
- Changed: config defaults to be aligned with Kafka client defaults #30
- updated kafka client (and its dependencies) to version 2.4.1 #16
added the input
client_rackparameter to enable support for follower fetching
added the output
partitionerparameter for tuning partitioning strategy
- Refactor: normalized error logging a bit - make sure exception type is logged
- Fix: properly handle empty ssl_endpoint_identification_algorithm #8
Refactor : made
partition_assignment_strategyoption easier to configure by accepting simple values from an enumerated set instead of requiring lengthy class paths #25
Elasticsearch Output - 10.5.1
File Output - 4.3.0