Tech Topics

Exciting Logstash Plugin Ecosystem Changes

With the release of Logstash 1.5.0 Beta 1 (try it out!), we are changing the way plugins are installed, maintained, and published. We have taken the feedback from our loverly users and community over time and our goal is to make it easier to use and develop plugins. This project is only the beginning — we will be iterating on this idea to provide a one-stop solution to discover and share a community of plugins! In this blog we would like to explain our reasons behind this decision, take you through the new workflow and future roadmap we have in mind.

There’s a plugin for that!

Logstash has a rich collection of plugins (inputs, filters, outputs, and codecs) which are both developed by Elasticsearch and contributed to by the community. One of the core strengths of Logstash is the availability of these plugins and ease of adding new ones to extend behavior. Today, there are over 165 plugins in the ecosystem which are split across two projects:

  • logstash-core which included most-commonly-used plugins which were shipped by default in Logstash
  • logstash-contrib which included plugins contributed to by the community and available for download separately

New Plugin Ecosystem Changes

All the plugins will now be separated out from Logstash core into their own self-contained packages using rubygems. We chose rubygems because of its powerful and standard way to package and distribute libraries with dependencies. Plus, we will be leveraging the rubygems.org platform for publishing and discovery. We have also added infrastructure to easily install, update, and remove plugins on top of Logstash. The contrib project will slowly cease to exist — all plugins will have a single home.

Reasons to move to the Plugin Ecosystem

Having a large set of plugins exposes unique challenges in distribution and publishing. Some of the pain points which inspired us to make these change were:

  • Today, we have to release a new version of Logstash every time a plugin needs an update.
  • Developers would like to release new versions and bug fixes for plugins outside of the release cadence of Logstash
  • Plugin developers would want to be able to describe external dependencies
  • Reduce the download size of the core Logstash distributable package and allow users to only install plugins they require
  • logstash-contrib was hard to maintain as one repository

The Details:

Source Code Location

Logstash source code will continue to exist in its current GitHub location but will not have any code or tests source related to plugins. Having this separation allows us to focus on core as well as to iterate quickly on individual plugins, which will improve the overall quality of Logstash project.

The source code for all plugins will be located in a new GitHub organization logstash-plugins and each plugin will exist as individual repositories under it. At first blush, this may seem hard to maintain, but it provides clear isolation for tests, issues and dependencies. Our aim is to automate testing, documentation, and gem publishing and provide additional tooling to ease this move.

However, it is not necessary for developers to host their plugin source under this organization — it is only necessary to publish it to rubygems.org to make it available to our community.

Workflow

Below we describe the interaction/workflow with the new plugin ecosystem from various perspectives:

Logstash Users:

Users will download the Logstash binary similar to previous releases. Logstash 1.5.0 will ship with the same set of plugins packaged with 1.4.2 to ease the migration to the new system. Additionally, users will be able to install and upgrade any Logstash plugin after the initial deployment.

$LS_HOME/bin/plugin script will be used for all plugin lifecycle interaction

Installing a plugin

Most of the plugins will have their gems uploaded to rubygems.org. For example, if the user has to install the Apache Kafka output plugin -

bin/plugin install logstash-output-kafka

Alternatively, using a file location:

bin/plugin install /path/to/logstash-output-kafka-1.0.0.gem

Removing a plugin

bin/plugin uninstall logstash-output-kafka

Updating one or all plugins

bin/plugin update

bin/plugin update logstash-output-kafka

Listing plugins

bin/plugin list

bin/plugin list elasticsearch ( List all plugins containing a name )

bin/plugin list --group output ( list all outputs )

Documentation

Even though the plugins are separated, documentation for all plugins will be in one central location

Logstash Plugin Developers:

Plugin developers and authors will be able to publish plugins for the Logstash ecosystem. Plugins will be able to declare external dependencies on gems and/or java libraries. More importantly, developers will be able to release improvements to plugins outside of the release cadence of Logstash.

Rubygems technology was chosen for the packaging system, dependency management, and hosting facility. Developers familiar with publishing regular Ruby gems will be able to easily publish Logstash plugins. Elasticsearch provides and maintains the tooling to aid developers with these functions.

Developing and testing locally

JRuby 1.7.16 is the only prerequisites for developing the plugin. Providing a patch to a plugin is like before. For example, to patch logstash-output-kafka:

  1. git clone https://github.com/logstash-plugins/logstash-outpu...
  2. Make changes
  3. Test locally from within the plugin:
    • bundle install
    • bundle exec rspec
    • To test against another version of Logstash or a local Logstash, you can edit the Gemfile and add alternative locations like gem "logstash", :github => "elasticsearch/logstash", :ref => "master"
  4. Create a new pull request against the repository logstash-output-kafka
  5. Community code reviews and Elasticsearch accepts the patch

Versioning

Version information is maintained in the .gemspec of the respective plugins. For example, for the Apache Kafka output, the gemspec is here. Versioning should follow semantic versioning rules and is maintained by the developer of the plugin outside of the Logstash versioning system. When Logstash 1.5.0 is released, plugins with milestone 1 will have version 1.0.0; plugins with milestone 2 will have version 2.0.0.

Publishing

When the developer is happy with the changes and wish to publish the plugins, the version number is changed at .gemspec. When all tests pass, Elasticsearch will manually publish the plugin to rubygems.org. If the tests fail, the gem is not published. Longer term we will be moving to automated publishing of the plugins. Since these changes are new, we would like to understand more about it and improve our testing infrastructure for plugins before we are able to provide automated publishing.

Issues

Issues should be opened against respective github repositories for each plugin. Logstash core repo will continue to be the home for issues that are pertaining to core pipeline and common functionality.

Documentation

Plugin documentation is generated from the source code itself, so each plugin’s documentation is contained in its own repository. Elasticsearch will provide the infrastructure to generate and consolidate the documentation for all plugins in to a centralised location under elasticsearch.org/guide.

Migration

All new pull requests and issues need to be opened against the respective plugins under the logstash-plugins github organisation.

What happens to my existing PR?

Fear not, existing pull requests does not need to be migrated to by developers. Logstash team will be merging existing PR’s targeted to Logstash core repo into their respective plugins repo by doing this:

git clone … # clone the specific plugin repo
# now apply the patch
curl -s https://github.com/elasticsearch/logstash/pull/XXX... | git am --3way
git push

Note: this process will maintain git history for existing PR’s.

GitHub Issues

Issues currently open against Logstash repo will be moved to their respective plugins. Logstash team will be automating this process using the github.com API. So, rest assured, we will migrate existing issues against the individual plugins.

Future Roadmap

While it is a first step, these changes provides a solid foundation for us to make the ecosystem even better for our users and developers.

Short term, we will be adding infrastructure for developers to provide automatic testing feedback on pull requests. We will continue to provide more tools to bootstrap and manage plugin repositories.

Longer term, we would like to provide a community portal for discovering and publishing all Logstash plugins. This idea is similar to the works of Puppet Forge and AWS marketplace.

We just released Logstash 1.5.0 Beta 1, which has support for the new ecosystem. Please try it out and let us know what you think about these changes. Your feedback (on Twitter or GitHub) is valuable to us!!