Hi all! With Elasticon wrapped up we're back with our weekly updates! Read on for what's new here!
New Chart to Visualize PQ Size and Capacity
We merged in a patch that will provide a chart that shows the total size and capacity across all persistent queues on a Logstash node. Once this is released, you'll be able to access the chart in the Advanced tab of the Logstash Node Monitoring UI if you have x-pack installed.
Inter-pipeline Comms are Closer
As announced at Elasticon, inter-pipeline communications are a big deal for the future of Logstash. This feature lets users easily connect pipelines running on the same Logstash creating arbitrary internal topologies. There are two main patterns where this is quite useful:
- "Distributor": One pipeline runs a beats input , classifies the data, then sends it to a specific Logstash pipeline for that business purpose. This massively DRYs up the configs, while still letting users use a single port for beats.
- "Output Decoupler": Users often complain that if one output temporarily blocks so does the other. This feature lets users insert a PQ between pipelines, allowing for temporary buffering if an output does disappear.
With the core PR (logstash#9206) open we now have all the code up we'd need to support this. With that PR and logstash-integration-internal#1 inter-pipeline communications work. We do need to put in work to merge this code and ensure that the quality the code is up to par, but we are close.
Exec Filter can Now Be Run on a Schedule
With logstash-input-exec#14 the Logstash exec filter can now be run on a schedule! This means Logstash can repeatedly execute a command a-la-cron (with cron syntax) and ingest its output. This really opens up a lot of use cases for Logstash, and is a welcome improvement!
Fixed Memory Leak in Logstash 5.x (JRuby 1.7)
In Jruby 1.7 some ruby constructs can leak memory. This affected the grok filter, which was patched here: logstash-filter-grok#136 .
Reduce Memory Usage in Beats Input
The Beats input can use excessive memory when there are many clients, esp. when pipelining values in the beats client are set to larger values. logstash-input-beats#301 reduces memory usage mitigating this issue.
Java API Proposal, Early Stages
We're hard at work on our new Java API in logstash#9137. We're scoping that PR quite small, bypassing some more complex features and dependency management, but it's an important first step.
Other Patches and Improvements
- Make '-v' fast on windows: logstash#8508
- Eliminate Ruby calls in worker loop: logstash#9178
- Config validation fixes for AWS based plugins: logstash-mixin-aws#34
- Fix millisecond handling in JDBC input: logstash-input-jdbc#260
- Fix interpolation when 'update' used in ES output: logstash-output-elasticsearch#385
- Github input stability fixes: logstash-input-github#1
- Fix missing logs in beats input: logstash-input-beats#247
- Add sophos UTM def to netflow codec: logstash-codec-netflow#105