29 March 2018

Logstash Lines: Improvements in filters and plugins

By Monica Sarbu

Did you know that Logstash 6.2 is already available? Try it and let us know what you think.

Logstash core

Java Execution Will Be Default in 6.4

We’ve decided to ship our java execution engine as our default execution engine in 6.4. This is currently only enabled if you use the special --experimental-java-execution flag. We’ve made a number of improvements to the engine since 6.2.3, including better flush handling #9289, removal of a possible memory leak in #9282, and much improved parsing speed for larger pipelines in #9278.

Publishing Java Plugins Discussion Underway

We’re just starting the discussion about how future Logstash plugins written in pure java will be packaged and published. Got an opinion? Help us out by commenting on #9227.

Drop Filter / Event#cancel now Significantly Faster

We found a surprising performance regression in our code for dropping events. This was fixed in #9284. Instead of returning a very cheap boolean object we were accidentally returning a very very expensive one. For those of you dropping a large number of events this should be a significant performance improvement. We’re currently targeting 6.3 for its release.

Logstash integrations

Redis Input 3.2.0 now supports Unix sockets and SSL

If you intend to use Redis in the same box where Logstash is installed, now there's the possibility to use Unix sockets. Redis claims in some situations this can improve performance by up to 50%.

On the other hand, if you're using Redis across the wire, you can enable SSL by passing " ssl => true". At this point, it's only possible to enable the use of SSL and not parametrize its settings such as paths for certificates and verifications modes.

UDP input supports IPv6 starting with 3.3.0

IPv6 support for this plugin has been broken in Logstash for a while due to a JRuby issue in the UDPSocket class and ability to guess if an IP was IPv4 or IPV6.

Filter kv 4.1.1 fixes handling of empty values

In previous versions, an empty value in a sequence of key values like "a=1|b=|c=3" would cause incorrect interpretation of the key/value pairs, where key "b" would have the value "|c=3". This has been fixed, and now keys without values will have a value of nil, allowing the rest of the pairs to be correctly interpreted (i.e. "c" will be "3").

Documentation

Repository: elastic/logstash-docs

Changes in versioned_plugin_docs:

  • Fix asciidoc error #500
  • Update link to support matrix #499

Other changes

Repository: elastic/logstash

  • Fix race condition that could stall pipeline shutdowns #9285
  • Improve cleanup of build server when using docker #9279

Repository: logstash-plugins

logstash-codec-json_lines - 3.0.6

  • Support flush method, see #35

logstash-codec-netflow - 3.11.3

  • Renamed some unknown VMware VDS fields

logstash-input-http - 3.0.9

  • Make sure default codec is also cloned for thread safety. #80
  • Always flush codec after each request and codec decoding. #81

logstash-input-jdbc - 4.3.6

  • Fix regression with 4.3.5 that can result in NULL :sql_last_value depending on timestamp format #274

logstash-input-tcp - 5.0.5

  • Restore SSLSUBJECT field when ssl_verify is enabled. #115

logstash-input-udp - 3.3.0

  • Add metrics support for events, operations, connections and errors produced during execution. #3
  • Fix support for IPv6 #31

logstash-output-file - 4.2.2

  • Add feature write_behavior to the documentation #58