13 September 2017 Engineering

Creating a threshold alert in Elasticsearch is simpler than ever.

By Jim UngerChris Roberson

Simple things should be simple was one of the themes at Elastic{ON} ‘17, our annual user conference where we connect with our users. During and prior to the user conference, we received many requests for a simple and easy to use UI to create alerts. As it turns out, creating a single UI to work effectively for all types of alerts is pretty hard. For example, a UI that can create an alert when the average CPU utilization goes over 50% looked pretty different from a UI that can create an alert when there are many concurrent logins from the same IP address.

Since it is hard to build a universal UI for all types of alerts, we decided to tackle the UI for the most commonly requested alert first: a simple, threshold alert that is triggered when a metric goes above or below a given threshold.

Before we can get into examples, make sure that you have the minimum of version of 6.0.0-beta 2 of Elasticsearch and Kibana installed with X-Pack installed on both. Also, ensure that you have Elasticsearch configured with a user with sufficient rights. Now, we need some interesting data with which to build our alerts. Metricbeat is a great beat for monitoring system and user processes on a machine. To set up Metricbeat, refer to our documentation here.

Once Metricbeat is installed and running, open up a browser and navigate to Kibana. Click on the Management app in the side navigation bar, and then click on Watcher under the Elasticsearch heading.

Now, click on the ‘Create New Watch’ button and select ‘Threshold Alert’. This will bring you to the new threshold alert UI.

In the ‘Create a new threshold alert’ panel, configure the name, index, time field and the trigger interval. This will reveal the condition panel with a simple condition already populated, as well as a visualization of the data that the condition is based upon.

For our first example, we want a threshold alert that is triggered when the maximum of the total CPU usage on a machine ever goes above 50% during the past 5 minutes. This is a very common type of alerts because it can help understand when and where the spikes are happening to take an action. We will choose 50% as a threshold value. Modify the condition expression so that it matches the image below. As you change the expression, the visualization is updated automatically to show the threshold value and data as red and blue lines respectively.

Now that we have configured the condition to trigger the alert, let’s configure what we want to happen when that condition is met. In this case, we want to send an E-mail to the person to which it matters most. Click on the ‘Add new action’ select box, and choose ‘E-mail’. A sample configuration for an e-mail action is shown below.

Note that you can use mustache syntax to access the {{ctx}} object which contains information about the alert when it triggers. We are using that here to embed the max CPU value into the body of our e-mail. If you have an e-mail account configured in Elasticsearch, then you can click on the ‘Test fire an e-mail now’ and watcher will send out a sample e-mail.

Click on the ‘Save’ button to save your new alert. There is now a threshold alert running on your Elasticsearch cluster.

Want to see this feature in a live demo?

But of course there’s more, like what’s coming up next. We have plans to allow you to use Kibana's saved searches to be the input for a threshold alert since those are fairly common and currently used in other areas of Kibana. We also plan to build out more UIs for other alerting use cases, such as comparing today’s value with last week’s value and alerting when the change goes beyond a certain threshold.

Go use this feature and check out our documentation for more details. As you dive into the 6.0 preview releases, we'd love to hear your feedback as part of our Pioneer Program. Your insights make our software better!