26 September 2014 Engineering

How to write a Logstash input plugin

By Aaron Mildenstein

Logstash is an event processing pipeline, which features a rich ecosystem of plugins, allowing users to push data in, manipulate it, and then send it to various backends.

While there is a multitude of plugins currently available for Logstash, perhaps, the one that fits your exact needs has yet to be created. That’s where you come in…To that end, this tutorial is to help walk you through the process of building your own.

We’re going to use the source code from the stdin plugin to go through the process of creating an input plugin for Logstash. This tutorial expects a certain amount of Ruby knowledge, but, hopefully, you will find it fairly easy to follow.

Anatomy of an input plugin

For this example, we’ll be using the Logstash stdin input plugin. While this is a basic plugin, the design principles and requirements apply to any input plugin. This document will refer to the source code by line (with links to the relevant block) and will show inline examples.

The following is a line-by-line breakdown of the example plugin. Clicking the links will take you to highlighted parts of the code.

Encoding

It seems like a small thing, but please don’t omit adding

encoding: utf-8

to the top of your input plugin. Logstash depends on things being in UTF-8, so we put this here to tell the Ruby interpreter that we’re going to be using the UTF-8 encoding.

Require

A Logstash input plugin requires some parent classes that can be referenced through the indicated statements. The following require statements are mandatory:

require "logstash/inputs/base"
require "logstash/namespace"

You may also need the socket require statement if your plugin is going to obtain the local hostname by way of a Socket.gethostname call (more here).

Of course, the plugin you build may depend on other code, or even gems. Just put them here along with these Logstash dependencies.

Plugin Body

Here, we’ll cover many subsections one by one.

Defining the Plugin

  1. The class name.
    • Your class will need to be a sub-class of LogStash::Inputs
    • Your class needs to extend the LogStash::Inputs::Base class
    • You should name your class in a way that closely mirrors the plugin name, e.g. LogStash::Inputs::Stdin
  2. config_name
    • This is the name your input plugin will call inside the input block.

      For example, if I set config_name "my_plugin", my corresponding Logstash block would look like this:

    input {
        my_plugin {...}
    }
  3. milestone
    • Milestones are defined here. Basically,
      • Milestone 1: it’s new and under development and may change
      • Milestone 2: more stable, more backwards-compatible, more-widely used
      • Milestone 3: even more backward compatible & this is likely enforced by automated tests.
      • Milestone 0: Infrequently used, means that the plugin is under-supported.
    • If it’s a new plugin, please start out by setting milestone 1

Setting defaults

  1. Default codec
    • Codecs are for processing input (and output) streams. You can serialize (or deserialize) the stream into a format Logstash expects. You may even want to write a codec if you have a specialized format to go with your input (not in the scope of this document).
    • Setting a default codec is advised, especially if you expect only one kind of input format
    • Codecs currently in code are here.
  2. Plugin initialization
    • The register method is like the initialize method. It was originally created to enforce having super called, preventing headaches for newbies. (Note: It may go away in favor of initialize and with some enforced testing to ensure super is called.)
    • public This means the method can be called anywhere, not just within the class. This is the default behavior for methods in Ruby, but it is called explicitly here anyway.
    • Assign instance variables here. In the example, @host is defined here.
    • Calling fix_streaming_codecs will automatically override the assigned codec if you use the plain or json codecs. This may or may not be needed for your plugin.

The run method: Where the action is

Cleanup

How do I make my own plugin from this?

Since you now have a good overview of how a plugin is built, and what the flow looks like, you should be able to envision your path to a plugin that does what you want.

Most of what you will want to do will be in the run method, or, at least, will be accessed from the run method. You can write other methods, include other required gems or code, and basically get your plugin to do anything you want so long as you:

  1. Read the data in
  2. Decode that data with a codec (even if you use the noop, don’t-do-anything codec).
  3. Loop through the resulting events and:
    1. decorate them with @timestamp and @version, in addition to your other fields
    2. Do your magic here!
    3. Append your event to the queue. This is how your event makes it into the rest of the Logstash pipeline.
  4. Catch any errors that may come up
  5. Teardown when it’s closing time.

Testing

Write unit and integration tests to ensure that your plugin behaves as expected. Tests for existing plugins are in the Logstash code in the path spec/ in the input, codec, filter, and output directories. These files provide excellent examples from which to derive your own tests.

If you would like to submit your plugin to the greater Logstash community, please be sure to include tests! A few examples of input plugin tests (ranging from simple to more complex) are:

Summary

This is a simple example of how you could write your own Logstash input plugin. Your final product can be as simple or as complex as your needs require. Once you find how easy it can be to write your own input plugin, you are empowered to make Logstash work for you in new and exciting ways! And once you get your new plugins working, we’d love to hear about them? Just drop us a line on Twitter so we can share in your awesomeness.

Happy Logstashing!