Logstash 6.1.0 Released

Logstash 6.1.0 has launched! We've got some great new features to talk about! Read about it here, or just head straight over to our downloads page and give it a shot! However, you may want to take a minute to read about breaking changes and release notes first. Read on for what's new in Logstash 6.1.0.

File Based Ruby Scripting Support

We’re proud to announce a great new way to extend Logstash functionality in 6.1.0. Complex modification of events in Logstash is now much easier due to our new feature, file based Ruby scripting via the Logstash Ruby filter. While the Ruby filter already lets you use custom ruby code to modify events, that code must live inside the Logstash configuration file itself, which doesn’t work well for longer pieces of code, and is hard to debug. Additionally, there’s never been a good way to reuse or test that code. File based Ruby scripts can take arguments, letting you reuse code within your Logstash configs easily.

This new feature lets you write Ruby code in an external file, with tests inline in that same file, and reuse that anywhere you’d like. Another nice feature here is that we can generate accurate line numbers in stack traces for exceptions in file based Ruby scripts, making them much easier to debug. The full details for configuring this are available in the Ruby filter docs. For a short example see below:

To configure the ruby filter with a file use:

filter {
  ruby {
    # Cancel 90% of events
    path => "/etc/logstash/drop_percentage.rb"
    script_params => { "percentage" => 0.9 }
  }
}

The file 'drop_percentage.rb' would look like:

def register(params)
  @should_reject = params["reject"]
end

def filter(event)
  return [] if event.get("message") == @should_reject
  event.set("field_test", rand(10))
  extra_processing(event)
  [event]
end

def extra_processing(event)
  # ..
end

test "non rejected events" do
  parameters do
    { "reject" => "hello" }
  end

  in_event { { "message" => "not hello" } }

  expect("events to flow through") do |events|
    events.size == 1
  end

  expect("events to mutate") do |events|
    events.first.get("field_test").kind_of?(Numeric)
  end
end

Experimental New Execution Engine

Logstash 6.1.0 brings some exciting new changes to the Logstash internals. We’ve been working on a full rewrite of the internal execution engine in Logstash. This rewrite moves the core execution logic from JRuby to Java/JVM Bytecode. With this approach we’ll be able to pave the way to more performance improvements in the future, as well as the ability to write optimized Logstash plugins in any JVM language.

This feature is currently disabled by default, and users should note that it is experimental and not yet ready for production. To enable this feature, you’ll need to use the '--experimental-java-execution' flag. We encourage users to try this flag out in test and staging environments and report any bugs found. Our hope is to make this the default execution method sometime in the 6.x timeframe.