News

Watcher 1.0 Goes GA

We are thrilled to announce the first GA release of Watcher 1.0.

Watcher was built to play a key role in the Elastic product stack and bring alerting and notification functionality to Elasticsearch. Elasticsearch itself is an amazing system that enables fine grained exploration of data... massive amounts of data. And now with Watcher, much of this data exploration can be automated.

Watcher enables you to register Watches to watch your data and act on events of interest. A Watch consists of four main parts - Trigger, Input, Condition and Actions. The following pseudocode describes the roles of these different parts in the watch execution:

Trigger fires an event e
watch.execute(e)
    payload = input.load(e)
    if condition.eval(payload)
        for each action in actions
            action.execute(e, payload)

Lets look at at how this simple execution model translates to a real-world scenario. In this example, we assume you use Elasticsearch to index and store events. These events can be log entries from various systems that are fed by Logstash or perhaps network monitoring events fed by Beats.

You don't index these events just for the sake of storing them; in most cases, the main reason for indexing them is to gain insight into what exactly goes on in the various systems you are monitoring. Part of this overview is to pick up on misbehaving systems, in which case, you want to do it as close to real time as possible.

So lets look at a simple example for a watch that looks for error events and if/when found, sends a notification to the appropriate stakeholders.

Assuming you have Elasticsearch already installed, the first thing you need to do is install Watcher. Watcher is a standard plugin to Elasticsearch and as such, you install it in the same way you install any other plugin:

bin/plugin -i elasticsearch/license/latest

For Watcher to work, you will also need to install the License plugin, but don't worry, you don't need a license to test watcher and play around with it, a 30 days trial license is automatically installed and opens up Watcher's full functionality. If you would like to extend the trial license, just contact us and we will be happy to extend it.

bin/plugin -i elasticsearch/watcher/latest

Now that you have Watcher installed, start Elasticsearch and add your first watch:

<span class="pln">curl -XPUT 'http://localhost:9200/_watcher/watch/error_status'
{</span>
    "trigger"<span class="pln">: {</span>
        "schedule"<span class="pln"> : { </span>"interval"<span class="pln"> : </span>"5m"<span class="pln"> }
    },</span>
    "input"<span class="pln"> : {</span>
        "search"<span class="pln"> : {</span>
            "request"<span class="pln"> : {</span>
                "indices"<span class="pln"> : [</span> "<events-{now/d}>"<span class="pln">,</span> "<events-{now/d-1d}>"<span class="pln"> ],</span>
                "body"<span class="pln"> : {</span>
                    "query"<span class="pln"> : {</span>
                        "filtered"<span class="pln"> : {</span>
                            "query"<span class="pln"> : { </span>"match"<span class="pln"> : {</span> "status"<span class="pln"> :</span> "error"<span class="pln"> }},</span>
                            "filter"<span class="pln"> : {</span> "range"<span class="pln"> : {</span> "_timestamp"<span class="pln"> : {</span> "from"<span class="pln"> :</span> "now-5m" <span class="pln">}}}
                        }
                    }
                }
            }
        }
    },</span>
    "condition"<span class="pln"> : {</span>
        "compare" <span class="pln">: { </span>"ctx.payload.hits.total"<span class="pln"> : { </span>"gt"<span class="pln"> : 0 }}
    },</span>
    "actions"<span class="pln"> : {</span>
        "email_admin"<span class="pln"> : {</span>
            "email"<span class="pln"> : {</span>
                "to"<span class="pln"> : </span>"admin@domain"<span class="pln">,</span>
                "subject"<span class="pln"> : </span>"Error Events"<span class="pln">,</span>
                "priority"<span class="pln"> : </span>"high"<span class="pln">,</span>
                "body"<span class="pln"> :</span> "Found {{ctx.payload.hits.total}} erroneous events"<span class="pln">
            }
        }
    }
}</span>

Once registered, the above watch will be triggered and be executed every 5 minutes. When executed the input searches for events with an error status (note the index name pattern that enables the input to only search the last 2 daily event indices). The condition checks if any errors were found, and if so the configured actions will be executed. In this watch we defined a single action that sends a high priority email to the administrator, indicating the number of errors found. Note that the body (as well as many other fields) supports the mustache template language, enabling you to freely customize its content.

As you can see, it's pretty straight forward. Now, lets take a closer look at the different parts that make a watch.

Trigger

The trigger determines when a watch will be executed. A trigger may have different implementations with different strategies around the decision whether or not a watch should be executed.

Watcher 1.0 ships with the schedule trigger - a time/calendar based trigger that can be configured to execute the watch periodically (e.g. "every 5 seconds", "every 10 minutes", "every 2 days", etc...) or based on the calendar using cron expressions (e.g. "on Monday at noon, every second week of the month").

Input

The input loads data for the watch to work with. There are different input types associated with different sources of data, most notably: search and http.

The search input loads data indexed in Elasticsearch leveraging its full search capabilities, including the full Query DSL and Aggregations. The returned search response (which includes the hits and the computed aggregations) will be loaded as the initial watch execution data.

The http input enables loading the initial data from remote web services. This can be a 3rd party system exposing RESTful endpoints or it can be an external Elasticsearch cluster with which you communicate via its REST API. The http input effectively enables you to set up a dedicated "watching" cluster that will watch all your other Elasticsearch clusters. An additional value in in this generic http input is that enables accessing all Elasticsearch APIs (beyond search) which means you can actually watch the health of your elasticsearch cluster itself.

The data loaded by the watch input loads will be used by the watch condition and actions during their execution. This data is commonly referred to as the watch execution Payload

Condition

The condition determines whether the actions should be executed. This predicate will typically evaluate the payload that was loaded by the input (as described above). For example, when using a search input, the condition can choose to execute the actions if the the number of returned search hits is greater than zero. Watcher supports different types of conditions, most notably: compare and script.

The compare condition simply compares a value in the payload to another value. For example, if the http input is used to load Elasticsearch's cluster health, the compare condition may look at the health status and decide to execute the actions if it is "red".

The script condition enables you to utilize the full scripting capabilities in Elasticsearch to define more complex logic around these decisions.

Actions

Finally, if the condition is met the actions execute. Again, like all other watch constructs, different types of actions are available. Watcher 1.0 ships with three main action types - email, webhook and index.

The email action enables you to notify about the events by sending an email to one or more recipients. We also made sure the email infrastructure in Watcher works well with all major email services out there.

The webhook action enables the watch to call a 3rd party system via a webhook. For example, one can call a paging service to notify whoever needs to be notified or automatically open a high priority ticket in your organization's ticketing system.

The index action simply takes the watch execution payload and indexes it back into Elasticsearch. It can be indexed as is, or can be transformed to another document that will be indexed instead.

Watch History

Watcher keeps a history of all watch executions in dedicated watch history time based indices. This enables continuous insight on Watcher's runtime and serves as a great tool to explore the behaviour of the registered watches. You can search these indices just like any other index in elasticsearch, run analytics on them and even build Kibana dashboards to help visually monitor Watcher. We have built such a dashboard ourselves that you can use by importing it into Kibana (requires Kibana 4.1 and above). You can also just use this dashboard as a starting point and further customise it as you see fit.

What's next?

With Watcher 1.0 we focused on building a solid future proof infrastructure that will serve as the foundations for many features and enhancements in the future. We tried to nail down the most basic constructs that make a Watch and as described above provide set of basic implementations to all the Watch constructs. This set of inputs, conditions, actions can go a long way to help you build both the simplest and the most advanced watching tasks.

But we certainly won't stop here. In future releases expect to see additional inputs, actions, and perhaps even predefined watches that will be optimized for the most common watching tasks and with that simplifying the way you interact with Watcher.

To learn more about watcher, we encourage you to download it and read the online documentation. We also encourage you to participate in the discussions in our Watcher Forum, ask questions, report bugs or simply share your experience with Watcher - we LOVE feedback and we act on it.

Enjoy!