27 June 2016

Logstash Lines: More Monitoring Info, Beats Input Improvements

By Suyog Rao

Welcome back to The Logstash Lines! In these weekly posts, we'll share the latest happenings in the world of Logstash and its ecosystem.

Moar Metrics, Moar Visibility (5.0)

For key filters like Grok, Date and GeoIP, we added per-plugin metrics like number of matches, failures, cache stats, and some more. This, coupled with execution timings we added recently, should make it easier to debug these popular filters.

  "name": "grok"
  "failures": 3
  "matches" : 1,
  "patterns_per_field" : {
     "message" : 1
  },

Persistent Queues

This week saw good progress in this area! Colin and Guy pushed initial commits for the low level data structures and classes that implement ack-able file based queues. Follow the feature branch for updates.

Beats Input Improvement

Beats input is being rewritten in Java, using the Netty library. This week we are doing reviews and end-to-end testing.

Curator v4

Big day for Curator! Aaron released a new version with tons of enhancements. The headlining feature is adding support for snapshot restore, and a new yaml based config file to manage execution.

Others:

  • Often, its simple things that annoys users. And we don't like when users are not happy. Checking version in Logstash was sloooow. Now its super fast thanks to a sweet hack.
/tmp/logstash-5.0.0 % time bin/logstash --version
logstash 5.0.0
27.79s user 1.34s system 270% cpu 10.773 total
....
/tmp/logstash-5.0.0 % time bin/logstash --version
logstash 5.0.0
0.01s user 0.02s system 101% cpu 0.029 total
  • Created a new LS_HOME/data directory to store plugin states, Logstash instance UUID and more. This directory location is configurable via path.data (#5404).
  • Kafka: Much requested 0.10 support has been published as version 5.0.0 of plugin. It will be shipped with 5.0.0-alpha4 and work on 2.4 as well.
  • Kinesis: This input plugin authored by Brian Palmer (codekitchen) has been moved to logstash-plugins org.
  • Pere has been investigating why plugin install is slow. This makes our test suite slow, and a speedup would also help users.
  • We are hardening the version constraints in our Gemfile — a manifest for default plugins and its versions — so we don't accidentally ship major upgrades in plugins in LS minor releases.