Breaking changes in 7.0

Here are the breaking changes for Logstash 7.0.

These changes can affect any instance of Logstash that uses impacted features. Changes to Logstash Core are plugin agnostic.

The new Java execution engine is enabled by default. It features faster performance, reduced memory usage, and lower config startup and reload times.

For more information, see the blog post about the initial release of the Java execution engine.

We went to considerable lengths to make this change seamless. Still, it’s a big change. If you notice different behaviors that might be related, please open a GitHub issue to let us know.

As of 7.0, Beats fields conform to the Elastic Common Schema (ECS).

If you upgrade Logstash before you upgrade Beats, the payloads continue to use the pre-ECS schema. If you upgrade your Beats before you upgrade Logstash, then you’ll get payloads with the ECS schema in advance of any Logstash upgrade.

If you see mapping conflicts after upgrade, that is an indication that the Beats/ECS change is influencing the data reaching existing indices.

The Field Reference parser, which is used to interpret references to fields in your pipelines and plugins, was made to be more strict and will now reject inputs that are either ambiguous or illegal. Since 6.4, Logstash has emitted warnings when encountering input that is ambiguous, and allowed an early opt-in of strict-mode parsing either by providing the command-line flag --field-reference-parser STRICT or by adding config.field_reference.parser: STRICT to logstash.yml.

Here’s an example.

Before

logstash-6.7.0 % echo "hello"| bin/logstash -e 'filter { mutate { replace => { "message" => "%{[[]]message]} you" } } }'
[2019-04-05T16:52:18,691][WARN ][org.logstash.FieldReference] Detected ambiguous Field Reference `[[]]message]`, which we expanded to the path `[message]`; in a future release of Logstash, ambiguous Field References will not be expanded.
{
       "message" => "hello you",
      "@version" => "1",
    "@timestamp" => 2019-04-05T15:52:18.546Z,
          "type" => "stdin",
          "host" => "overcraft.lan"
}

After

logstash-7.0.0 % echo "hello"| bin/logstash -e 'filter { mutate { replace => { "message" => "%{[[]]message]} you" } } }'
[2019-04-05T16:48:09,135][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>java.lang.IllegalStateException: org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `[[]]message]`
[2019-04-05T16:48:09,167][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

With 7.0.0, we have taken the opportunity to upgrade a number of bundled plugins to their newest major version, absorbing their breaking changes into the Logstash distribution.

While these upgrades included new features and important fixes, only the breaking changes are called out below.

Here are the breaking changes for codec plugins.

CEF Codec

Netflow Codec

Here are the breaking changes for filter plugins.

Clone Filter

Geoip Filter

HTTP Filter

Here are the breaking changes for input plugins.

Beats Input

HTTP Input

HTTP Poller Input

Tcp Input

Here are the breaking changes for output plugins.

Elasticsearch Output

HTTP Output

Kafka Output

Redis Output

Sqs Output

Tcp Output