NOTE: Alpha releases cannot be used in production. They are for evaluation purposes only
Mo' pipelines mo' fun
We've added support for running multiple, self-contained pipelines in the same Logstash instance (on the same JVM). Logstash configurations can be complex with multiple data flows built using conditionals. This feature will help simplify that — if your data flow and processing can be separated, you can run them in an isolated fashion as separate pipelines. This separation means that a blocked output in one pipeline won’t exert backpressure in the other pipeline.
A new configuration file called
pipeline.yml file has been added to define multiple pipelines that can be run on the same instance. Users can dynamically add, modify and remove pipeline configurations. This feature also provides an option to run pipelines with their own run-time settings like workers size, batch size etc. This feature isn't complete yet — the monitoring UI in x-pack currently reports metrics from a single pipeline; we're working on enhancing the UI to support multiple pipelines as well, so stay tuned for updates!
Breaking Change: glob only *.conf in conf.d folder
Currently, when Logstash is installed and set up via package managers, it loads all files found in
/etc/logstash/conf.d as configuration. This can be problematic if there are non-configuration files in this folder. For example, user makes a copy of a configuration file and names it with a
.bak extension which results in unexpected behavior. Starting from alpha2, Logstash will only glob files ending with
.conf extension in
We would love for you to try 6.0.0-alpha2 and the new multi-pipeline feature! Let us know what you think!