Logstash Lines: GeoIP filter enhancements

Por
Suyog Rao

Welcome back to The Logstash Lines! In these weekly posts, we'll share the latest happenings in the world of Logstash and its ecosystem.

Java Event

Last week, we made changes in core to expose the new setter and getter APIs to the Event object. To propagate this change to all the 200+ plugins, we've taken a phased approach. First up, we've updated all the plugins that are used in the core unit tests with the new API, and tests are now passing. Next step, which will be this week, we move to those plugins (70+) which are packaged in Logstash artifacts. We have a meta issue to track that, and work will be split by the team. Once these plugin tests are passing, we can start publishing gems for all the updated plugins, and move on to non-packaged plugins

Plugin Manager

Added --preserve option to bin/logstash-plugin install command. This allows us to preserve gem options which are already specified in Gemfile, which would have been previously overwritten. This will also help with the unified build effort by avoiding the unnecessary publication of core plugins snapshots.

Verbose Output: When running any plugin related command you can use DEBUG=1, which will give the user a bit more information about what bundler is doing.

Acceptance Testing

Work continues on building the framework for acceptance tests on multiple artifacts/platforms using Vagrant. At this point, we've written plugin manager validation tests using the new framework. This week, we'll focus on deb/rpm package tests and running these on Jenkins (thanks to our infra team).

Plugins

GeoIP Filter

Supporting MaxMind's GeoIP2 city database has been a longstanding enhancement request! Over the past couple of weeks we've been working with our community members Gary Gao (@garyelephant) and Thomas Decaux (@ebuildy) on a pull request to bring these changes to this filter. This plugin now directly uses the GeoIP2 Java API, adds support for IPv6 lookups and brings in all the goodness that is GeoIP2. Many thanks to Gary and Thomas! These changes will be packaged with 5.0.0-alpha2.

Elasticsearch Output: This output now supports ingest pipelines while indexing docs. Very simply, you can:

{     
  "hosts" => "localhost",
  "pipeline" => "apache-logs",
  ...
}

Others

  • Set the heap size to 1gb when running rake tasks. Without this, some of the plugin installs were failing in our CI environment. This started showing up after we upgraded to JRuby 1.7.25 (#5179).
  • Kafka: Fixed an issue were snappy and LZ4 compression were not working with 0.9 broker (#50).
  • Added static analysis rake task for i18n usage. This will be used to catch descriptions that are missing for user facing errors.