How to write a Logstash input plugin
Logstash is an event processing pipeline, which features a rich ecosystem of plugins, allowing users to push data in, manipulate it, and then send it to various backends.
While there is a multitude of plugins currently available for Logstash, perhaps, the one that fits your exact needs has yet to be created. That's where you come in…To that end, this tutorial is to help walk you through the process of building your own.
We're going to use the source code from the stdin
plugin to go through the process of creating an input plugin for Logstash. This tutorial expects a certain amount of Ruby knowledge, but, hopefully, you will find it fairly easy to follow.
Anatomy of an input plugin
For this example, we'll be using the Logstash stdin input plugin. While this is a basic plugin, the design principles and requirements apply to any input plugin. This document will refer to the source code by line (with links to the relevant block) and will show inline examples.
The following is a line-by-line breakdown of the example plugin. Clicking the links will take you to highlighted parts of the code.
Encoding
It seems like a small thing, but please don't omit adding
encoding: utf-8
to the top of your input plugin. Logstash depends on things being in UTF-8, so we put this here to tell the Ruby interpreter that we're going to be using the UTF-8 encoding.
Require
A Logstash input plugin requires some parent classes that can be referenced through the indicated statements. The following require
statements are mandatory:
require "logstash/inputs/base"<br>
require "logstash/namespace"
You may also need the socket
require statement if your plugin is going to obtain the local hostname by way of a Socket.gethostname
call (more here).
Of course, the plugin you build may depend on other code, or even gems. Just put them here along with these Logstash dependencies.
Plugin Body
Here, we'll cover many subsections one by one.
Defining the Plugin
- The class name.
- Your class will need to be a sub-class of
LogStash::Inputs
- Your class needs to extend the
LogStash::Inputs::Base
class - You should name your class in a way that closely mirrors the plugin name, e.g.
LogStash::Inputs::Stdin
- Your class will need to be a sub-class of
config_name
- This is the name your input plugin will call inside the input block.
For example, if I set
config_name "my_plugin"
, my corresponding Logstash block would look like this:
<code>input { my_plugin {...} }</code>
- This is the name your input plugin will call inside the input block.
For example, if I set
milestone
- Milestones are defined here. Basically,
- Milestone 1: it's new and under development and may change
- Milestone 2: more stable, more backwards-compatible, more-widely used
- Milestone 3: even more backward compatible & this is likely enforced by automated tests.
- Milestone 0: Infrequently used, means that the plugin is under-supported.
- If it's a new plugin, please start out by setting
milestone 1
- Milestones are defined here. Basically,
Setting defaults
- Default codec
- Codecs are for processing input (and output) streams. You can serialize (or deserialize) the stream into a format Logstash expects. You may even want to write a codec if you have a specialized format to go with your input (not in the scope of this document).
- Setting a default codec is advised, especially if you expect only one kind of input format
- Codecs currently in code are here.
- Plugin initialization
- The register method is like the
initialize
method. It was originally created to enforce havingsuper
called, preventing headaches for newbies. (Note: It may go away in favor ofinitialize
and with some enforced testing to ensuresuper
is called.) public
This means the method can be called anywhere, not just within the class. This is the default behavior for methods in Ruby, but it is called explicitly here anyway.- Assign instance variables here. In the example,
@host
is defined here. - Calling
fix_streaming_codecs
will automatically override the assigned codec if you use theplain
orjson
codecs. This may or may not be needed for your plugin.
- The register method is like the
The run
method: Where the action is
- The flow here is fairly straightforward:
- Read data in
- Decode data by way of a codec
- Loop through resulting events
- For each event:
decorate
(Add@timestamp
,@version
fields, if not present)- Any other processing happens here…
- e.g. Add/overwrite any other fields, as needed
- Custom processing, tokenizing, anything needed!
- Append that event to the queue
- Catch errors, if any (the
rescue
call)
Cleanup
How do I make my own plugin from this?
Since you now have a good overview of how a plugin is built, and what the flow looks like, you should be able to envision your path to a plugin that does what you want.
Most of what you will want to do will be in the run
method, or, at least, will be accessed from the run
method. You can write other methods, include other required gems or code, and basically get your plugin to do anything you want so long as you:
- Read the data in
- Decode that data with a codec (even if you use the
noop
, don't-do-anything codec). - Loop through the resulting events and:
decorate
them with@timestamp
and@version
, in addition to your other fields- Do your magic here!
- Append your event to the queue. This is how your event makes it into the rest of the Logstash pipeline.
- Catch any errors that may come up
- Teardown when it's closing time.
Testing
Write unit and integration tests to ensure that your plugin behaves as expected. Tests for existing plugins are in the Logstash code in the path spec/ in the input, codec, filter, and output directories. These files provide excellent examples from which to derive your own tests.
If you would like to submit your plugin to the greater Logstash community, please be sure to include tests! A few examples of input plugin tests (ranging from simple to more complex) are:
Summary
This is a simple example of how you could write your own Logstash input plugin. Your final product can be as simple or as complex as your needs require. Once you find how easy it can be to write your own input plugin, you are empowered to make Logstash work for you in new and exciting ways! And once you get your new plugins working, we'd love to hear about them? Just drop us a line on Twitter so we can share in your awesomeness.
Happy Logstashing!