Watcher is an Elasticsearch feature that you can use to create actions based on conditions, which are periodically evaluated using queries on your data. Watches are helpful for analyzing mission-critical and business-critical streaming data. For example, you might watch application logs for performance outages or audit access logs for security threats.
To get started with the Watcher UI, go to Management > Elasticsearch > Watcher. With this UI, you can:
Alerting on cluster and index events is a good source for detailed information on how watches work. If you are using the UI to create a threshold watch, take a look at the different watcher actions. If you are creating an advanced watch, you should be familiar with the parts of a watch—input, schedule, condition, and actions.
If the Elasticsearch security features are enabled, you must have the
cluster privileges to use Watcher in Kibana.
Alternately, you can have the built-in
and either of these watcher roles:
watcher_admin. You can perform all Watcher actions, including create and edit watches.
watcher_user. You can view watches, but not create or edit them.
You can manage roles in Management > Security > Roles, or use the Kibana Role Management API. Watches are shared between all users with the same role.
If you are creating a threshold watch, you must also have index management privileges. See Managing Indices for detailed information.
A threshold alert is one of the most common types of watches that you can create. This alert periodically checks when your data is above, below, equals, or is in between a certain threshold within a given time interval.
The following example walks you through creating a threshold alert. The alert is triggered when the maximum total CPU usage on a machine goes above a certain percentage. The example uses Metricbeat to collect metrics from your systems and services. Learn more on how to install and get started with Metricbeat.
Click Create and then select Create threshold alert.
You’re navigated to a page where you’re asked to define the watch name, the data that you want to evaluate, and how often you want to trigger the watch.
Enter a name that you want to call the alert, for example,
In the Indices to query field, enter
@timestampas the time field.
Use the default schedule to run the watch every 1 minute.
You should now see a panel with default conditions and a visualization of the data based on those conditions. The condition evaluates the data you’ve loaded into the watch and determines if any action is required.
WHENexpression and change the value to
OFexpression now appears.
system.process.cpu.total.norm.pctand select it from the list.
IS ABOVEexpression and change the value to
.25to trigger an alert whenever the CPU is above 25%.
As you change the condition, the visualization is automatically updated. The black line represents the threshold (25%), while the green fluctuating line represents the change in CPU over the set time period.
Now that the condition is set, you must add an action. The action triggers when the watch condition is met. For a complete list of actions and how to configure them, see Adding conditions to actions.
In this example, you’ll configure an email action. You must have an email account configured in Elasticsearch for this example to work.
- Click Add action and select Email.
- In the To email address field, enter one or more email addresses to whom you want to send the message when the condition is met.
Enter a subject and body for the email.
To test the action before saving the watch, click Send test email.
A sample email is sent using the configuration you set up.
Click Create alert.
The alert appears on the Watcher overview page, where you can drill down into the watch history and status.
In this example, you set the threshold to 25% so you can see the watch fire. In a real-world scenario, this threshold is likely too low because the alerts are so frequent. Once you are done experimenting, you should delete the alert. Find the alert on the Watcher overview page and click the trash icon in the Actions column.
Alternatively, you can keep the alert and adjust the threshold value. To edit an alert, find the alert on the Watcher overview page and click the pencil icon in the Actions column.
The Watcher overview page lists your watches and includes details such as state, last fired, and last triggered. A watch has one of four states:
- Firing. The watch is triggered and actively performing the associated actions.
- Error. The watch is not working properly.
- OK. The watch is not actively firing but working properly.
- Disabled. The watch will not fire under any circumstances.
From this page you can drill down into a watch to investigate its history and status.
The Execution history tab shows each time the watch is triggered and the results of the query, whether the condition was met, and what actions were taken.
The Action statuses tab lists all actions associated with the watch and the state of each action. If the action is firing, you can acknowledge the watch to prevent too many executions of the same action for the same watch. See Acknowledgement and throttling for details.
Actions for deactivating and deleting a watch are on each watch detail page:
- Deactivate a watch if you know a situation is planned that will cause a false alarm. You can reactivate the watch when the situation is resolved.
- Delete a watch to permanently remove it from the system. You can delete the watch you are currently viewing, or go to the Watcher overview, and delete watches in bulk.
Advanced watches are for users who are more familiar with Elasticsearch query syntax and the Watcher framework. The UI is aligned with using the REST APIs. For more information, see Query DSL.
On the Watch overview page, click Create and choose Create advanced watch. An advanced watch requires a name and ID. Name is a user-friendly way to identify the watch, and ID refers to the identifier used by Elasticsearch. Refer to Watch definition for how to input the watch JSON.
The Simulate tab allows you to override parts of the watch, and then run a simulation. Be aware of these implementation details on overrides:
After starting the simulation, you’ll see a results screen. For more information on the fields in the response, see the Execute watch API.
Refer to these examples for creating an advanced watch: