29 January 2019 Releases

Logstash 6.6.0 Released

By Guy Boertje

Logstash 6.6.0 has been released with new features and enhancements. This release introduces the HTTP filter plugin, the Memcached filter plugin and the experimental Java Plugin API.


Java Execution Engine

The Java execution engine is GA in this release after months of being experimental. Since we last blogged about this feature we have had feedback from the field that confirms higher performance, reduced memory usage, and lower config startup and reload times. You still need to opt-in to (turn on) Java execution and we have updated the Logstash documentation to explain this better. Java execution will become the default engine in 7.0.


Index Lifecycle Management  (ILM)

In 6.6.0, Elasticsearch introduces the Index Lifecycle Management feature.  You can control how indices are managed as they age, and specify, for example, a maximum size or age before an index should be automatically rolled over into a new index.
We updated the Elasticsearch output to suit. It can automatically create a default lifecycle policy that will automatically rollover an index after it reaches 50GB or is 30 days old. It can also work with more complex policies that have been previously created, which can perform more advanced lifecycle operations, such as deleting the index when it is no longer needed for retention, or moving the index to less performant hardware after a certain amount of time.


Worker Thread Naming

We fixed a regression whereby the worker thread names were not being set in the same way that they were in 6.3. This made interpretation of "hot threads" and stalled shutdown reports really difficult, with this bug fix it is much easier to see that workers could be "hot" or stalled on shutdown.


Native support for Java plugins

This long awaited feature is released as experimental. Dan Hermann has written a detailed blog post on this feature detailing how, as a Java developer, you can create Logstash plugins without needing to learn Ruby. With this API, Java developers should be able to write Logstash plugins using idiomatic Java code free of any dependencies on Ruby and its toolchain and ecosystem that seamlessly inter-operate with Ruby plugins and yield high performance in the Java execution engine.


HTTP Filter

This is an enrichment filter, it allows you to invoke HTTP calls with parameters from an event to external APIs and have the response inserted into that event. Thanks go to Lucas Henning who created the logstash-filter-rest, upon which this filter is based.

Here is an example calling the convert function on the neutrinoapi service.


 http {
    url => "https://neutrinoapi.com/convert"
    body => {
      "user-id" => "auser"
      "api-key" => "anapikey"
      "from-value" => "%{millimeters}"
      "from-type" => "mm"
      "to-type" => "inch"
    }
    verb => "POST"
    body_format => "json"
    target_body => "neutrinoapi_convert_mm_to_in"
    add_tag => ["from_api"]
    id => "http-neutrino-lookup"
  }
  mutate {
    # needed because the neutrino response JSON object includes unneeded extra key/values
    copy => {"[neutrinoapi_convert_mm_to_in][result]" => "[inches]"}
    remove_field => ["neutrinoapi_convert_mm_to_in"]
    id => "copy-to-inches"
  }

For more details on the plugin and usage instructions, check the documentation page.


Memcached Filter

This is the enrichment filter you have all been waiting for. Why? It enables K/V set as well as get, meaning, for example, you can have one pipeline extracting and loading memcached with key/values and another pipeline enriching events via the same keys.

An example of setting:

 memcached {
    hosts => ["localhost"]
    namespace => "neutrinoapi_convert"
    set => {
      "[neutrinoapi_convert_mm_to_in][result]" => "%{millimeters}mm"
    }
    id => "memcached-set"
  }


An example of getting:

 memcached {
    hosts => ["localhost"]
    namespace => "neutrinoapi_convert"
    get => {
      "%{millimeters}mm" => "[inches]"
    }
    add_tag => ["from_cache"]
    id => "memcached-get"
  }


For more details on the plugin and usage instructions, check the documentation page.


Other Plugin Improvements

Logstash 6.6.0 includes a number of improvements to some of our bundled plugins. Some highlights:


  • The Aggregate filter has some improvements such as custom timeouts, metrics, metadata additions in the code block.
  • The Elasticsearch filter has improvements in the client connection.
  • The Gelf input has had a general overhaul around shutdown, socket handling json parsing and code quality.
  • The Kafka input and output have version 2.1.0 of the Kafka Client and a new setting "ssl.endpoint.identification.algorithm".
  • The Elasticsearch output now supports Index Lifecycle Management for Elasticsearch versions 6.6.0 and above running with a Basic License(or higher). 
  • The Tcp output no longer needs a certificate/key pair when enabling ssl


Download and Release Notes

You can find the full list of the changes in the release notes.



Feedback

If you want to try the new features added in Logstash 6.6.0, please download it, install it, and let us know what you think on Twitter (@elastic) or in our forum. Enjoy!