Logstash 5.1.1 Release Notesedit

  • Persistent queues is a new beta feature that provides infrastructure in Logstash for buffering event data to disk. By default, Logstash still uses an in-memory, bounded queue between the input and filter stages without any guarantees for data delivery. Users can now opt-in to enable disk-based persistence that can prevent event loss in certain scenarios. The setting queue.type: persisted enables this feature (Issue 2605, Issue 2606).
  • Filters can be configured to log event data and related context when filter execution exceeds a specified time threshold. These slowlogs will be collected in a separate file called logstash-slowlog-plain-YYYY-MM-dd.log (Issue 5733).
  • Added two new fields id, and name to the base metadata for API requests. The id field is the persisted UUID. The name field is the custom name the user has passed in (defaults to the hostname).
  • The _node/stats API will now report uptime in milliseconds (Issue 6214).
  • Enhanced the /_node/stats API to report the CPU load average information. Load average stats are not reported on Windows (Issue 6215).
  • Fixed heap dump path for Windows installs.
  • When using the JSON logging format, exception messages are now handled correctly (Issue 6082).
  • We now log an error message when a plugin specified in the config is not found in the installed list (Issue 6020).

Input Pluginsedit

Beats:

  • The congestion_threshold configuration is now deprecated as the new Java implementation uses an alternate keep-alive mechanism.
  • Fixed an issue where data shipped in intervals greater than five seconds could cause the connection between Filebeat and Logstash to be closed (Issue 163).

Kafka:

  • Added Kerberos authentication support (Issue 124).

Syslog:

TCP:

UDP:

  • Added receive_buffer_bytes config setting to optionally set the socket receive buffer size. Allowing a user to set this can prevent data loss on uneven traffic flow (Issue 22).

RabbitMQ

  • A dropped connection between Logstash and RabbitMQ is now re-tried when an intermediary proxy reports that there are no live backends (#76)

Filter Pluginsedit

Grok:

  • The timeout enforcer can now be disabled by setting timeout_millis to nil. The default for timeout_millis is now 30s (Issue 99).

Date:

  • Improved performance: The date filter is now 2.8x faster for common case (first pattern matches), and 14x faster for events where multiple patterns are attempted (Issue 74).

Output Pluginsedit

Elasticsearch:

  • A health check is now performed against Elasticsearch hosts right after startup and sniffing for the first time. Previously, a health check was issued after an error was received from a request (Issue 507).
  • You can now specify event dependent configuration when configuring the ingest pipeline (Issue 491).