Enable Alerting (Watcher)edit

If you are looking for the Alerting and Actions documentation, see Alerting and Actions in the Kibana Guide, version 7.8 and later.

Alerting lets you take action based on changes in your data. It is designed around the principle that, if you can query something in Elasticsearch, you can alert on it. Simply define a query, condition, schedule, the actions to take, and Alerting will do the rest.

Alerting (via Watcher) can be enabled when configuring your cluster, available for clusters with version 1.7.2 or higher. You can run Alerting on a separate cluster from the cluster whose data you are actually watching.

In Elasticsearch 5.x, Watcher was renamed to Alerting and became a part of X-Pack. If you’re using a version of Elasticsearch before 5.0, think Watcher every time you read about Alerting.

Before you beginedit

Some restrictions apply when adding alerts. To learn more, see Restrictions for alerts (via Watcher).

To enable alerting on a cluster, you also need to:

  • On the Edit page, enable Automatic index creation in Elasticsearch plugins and settings, if it isn’t enabled already
  • Enable scripting for most uses of Alerting, as Alerting uses the Elasticsearch script infrastructure
  • For Elasticsearch versions before 5.0: Enable authentication
  • To receive default Elasticsearch X-pack watcher alerts (cluster status, nodes changed, version mismatch), you need to have monitoring enabled send to the Admin email specified in Kibana by going to Advanced Settings > Admin email. Then whitelist that same email in the Elasticsearch Add-On for Heroku console by updating the monitoring email whitelist.

To learn more about Alerting and how to use it, see Alerting on cluster and index events (version 6.3 and later), Alerting on cluster and index events (version 5.0 to 6.2), or Elasticsearch Watcher (all versions before 5.0).

Send alerts by emailedit

Alerting can send alerts by email.

To send alerts by email:

  1. Log in to the Elasticsearch Add-On for Heroku console.
  2. Go to Account and then Contacts, or visit the contacts page directly.
  3. Enter a recipient to be whitelisted under Monitoring email whitelist and click Add.

    An email is sent to the email address.

  4. The recipient must acknowledge the request by clicking Confirm Email Whitelisting in the email.

    After the whitelist request is acknowledged, you are able to send alerts to the recipient address by email.

  5. Optional: You can add the xpack.notification.email.html.sanitization.* setting under Elasticsearch User setting overrides in the Elasticsearch Add-On for Heroku console so that HTML elements are sanitized in the email notification.

For more information on sending alerts by email, see Email action.

Elasticsearch Add-On for Heroku lets you add destination email addresses to the whitelist as described in this section, but other configuration options are not supported. Specifically, configuring email accounts is not applicable to Elasticsearch Add-On for Heroku.

Advanced usageedit

Slack, HipChat, and PagerDuty integrationedit

Under the hood, Alerting is configured via elasticsearch.yml. If you want to customize your Alerting settings, you can provide custom elasticsearch.yml snippet which is appended to your configuration.

To provide the custom snippet, you can use the console Elasticsearch settings editor for your deployment.

For example if you want to use the Slack integration:

There are three steps to integrate Elasticsearch 7.0 or greater with Slack:

  1. Generate a Webhook URL in Slack. It will look similar to https://hooks.slack.com/services/..
  2. Add a Slack account name to your Elasticsearch User settings
  3. Associate the Slack account with the Slack Webhook in the Elasticsearch keystore

To add a webhook in Slack, click on the settings icon, select Add an app, and then search for webhook.

The following example shows a configuration with multiple Slack accounts (account1, account2, and account3) specified in elasticsearch.yml:

xpack.notification.slack:
  default_account: account1
  account:
    account1:
      message_defaults:
        from: account1
        to: channel1
    account2:
      message_defaults:
        from: account2
        to: channel2
    account3:
      message_defaults:
        from: account3
        to: channel3

In Elasticsearch version 7.x and later, you can no longer configure Slack Webhooks using elasticsearch.yml settings. Your Slack account name and secure URL settings need to be configured in the Elasticsearch keystore, with a setting like the following:

Slack Webhook account settingsedit

The Slack Webhook is set for each account in the Elasticsearch Keystore with the following settings:

Setting name
xpack.notification.slack.account.ACCOUNT_NAME.secure_url where ACCOUNT_NAME is the Slack account, such as account1.
Type
Single string
Secret
The Webhook URL you generated in Slack earlier.

To specify a Slack account to use for a Watcher Alert that isn’t set as default_account, you must create an Advanced Watch and explicitly define which Slack account to use in the actions section.

If you have a Slack account that is not currently set as default_account, and you want to use this account for a Watcher Alert, you must create an Advanced Watch and explicitly define in the Actions section of the UI which Slack account to use.

PUT _watcher/watch/test-alarm
{
  "metadata" : {
    ...
  },
  "trigger" : {
    ...
  },
  "input" : {
    ...
  },
  "actions" : {
    "notify-slack" : {
      "throttle_period" : "10s",
      "slack" : {
        "account" : "account2",
        "message" : {
          "to" : [ "#testing-channel" ],
          "text" : "You Know, for Search"
        }
      }
    }
  }
}

In Elasticsearch versions before 7.0:, you are not required to use the Elasticsearch keystore. Instead, you can use the console Elasticsearch settings editor for your deployment.

Advanced Alerting configuration