Logstash Lines: Introduce integration plugins

Integration plugins are coming soon!

We are introducing the idea of Integration Plugins to the upcoming Logstash release. The main goal is to reduce the maintenance effort and improve the user experience of many plugins that ship with Logstash.

Logstash contains plugins that connect to Elasticsearch, RabbitMQ, Kafka, Cloud Services and many more. For many of these technologies we provide with both an Input plugin (to consume data from) and an Output plugin (to produce data to), and for some we have one or more Filter plugins (to manipulate or enrich data as it flows through a pipeline).

As an example, Logstash has the logstash-input-rabbitmq and logstash-output-rabbitmq plugins to connect to the RabbitMQ message broker.

With separate repositories, it's difficult to ensure the two plugins have a consistent experience: some bugs may be reported and fixed in one and not the other; documentation improvements may also only land in only one of the repositories; and both development and use of these plugins required awkward lock-step effort.

Other plugin sets like the Elasticsearch input, filter and output were not created at the same time and even depend on different libraries, so they have different settings for things like encryption and report errors in different ways.

As of 7.5.0, Logstash will be capable of handling integration plugins. An integration plugin is packaged as Ruby Gem like individual plugins and is created from a single repository that contains one or more Logstash plugins.

This release will ship with the first two integration plugins: kafka and rabbitmq and you will be able to see them in the familiar plugin list:

logstash-7.5.0 % bin/logstash-plugin list integration
 ├── logstash-input-kafka
 └── logstash-output-kafka
 ├── logstash-input-rabbitmq
 └── logstash-output-rabbitmq

Each of pairs of input/output plugins for message brokers can now be fixed, enhanced and released in tandem, and as development continues we can gradually reduce code and make them more and more consistent. The individual plugin repositories will be gradually archived and issues moved to the respective integration repository.

At this point you may be wondering "What is the impact of my deployment of Logstash?". And the answer is "None!" as this transition does not break existing configurations.

Finally, what is the next step? We'll continue to convert sets of plugins into integration plugins (elasticsearch input/filter/output, jdbc input/filters, etc) and include them in the next releases.