From version 5.0 onward, Watcher is part of X-Pack. For more information, see Alerting on cluster and index events.
Elastic Docs ›Elasticsearch Watcher [1.0] ›Reference
« Condition Transform »

Actionsedit

The actions associated with a watch are executed whenever the watch is executed, its condition is met, and the watch is not throttled. The actions are executed one at a time and each action executes independently from the others. Any failures encountered while executing an action are recorded in the action result and in the watch history.

If no actions are defined for a watch, no actions are executed. However, a watch_record is still written to the Watch History.

Actions have access to the payload in the execution context. They can use it to support their execution in any way they need. For example, the payload might serve as a model for a templated email body.

Acknowledgement and Throttlingedit

During the watch execution, once the condition is met, a decision is made per configured action as to whether it should be throttled. The main purpose of action throttling is to prevent too many executions of the same action for the same watch.

For example, suppose you have a watch that detects errors in an application’s log entries. The watch is triggered every five minutes and searches for errors during the last hour. In this case, if there are errors, there is a period of time where the watch is checked and its actions are executed multiple times based on the same errors. As a result, the system administrator receives multiple notifications about the same issue, which can be annoying.

To address this issue, Watcher supports time-based throttling. You can define a throttling period as part of the action configuration to limit how often the action is executed. When you set a throttling period, Watcher prevents repeated execution of the action if it has already executed within the throttling period time frame (now - throttling period).

The following snippet shows a watch for the scenario described above - associating a throttle period with the email_administrator action:

Watch Definition Example.

PUT _watcher/watch/log_event_watch
{
  "metadata" : {
    "color" : "red"
  },
  "trigger" : {
    "schedule" : {
      "interval" : "5m"
    }
  },
  "input" : {
    "search" : {
      "request" : {
        "search_type" : "count",
        "indices" : "log-events",
        "body" : {
          "query" : { "match" : { "status" : "error" } }
        }
      }
    }
  },
  "condition" : {
    "script" : "return ctx.payload.hits.total > 5"
  },
  "actions" : {
    "email_administrator" : {
      "throttle_period": "15m", 
      "email" : { 
        "to" : "sys.admino@host.domain",
        "subject" : "Encountered {{ctx.payload.hits.total}} errors",
        "body" : "Too many error in the system, see attached data",
        "attach_data" : true,
        "priority" : "high"
      }
    }
  }
}

There will be at least 15 minutes between subsequent email_administrator action executions.

See Email Action for more information.

You can also define a throttle period at the watch level. The watch-level throttle period serves as the default throttle period for all of the actions defined in the watch:

Watch Definition Example.

PUT _watcher/watch/log_event_watch
{
  "trigger" : {
    ...
  },
  "input" : {
    ...
  },
  "condition" : {
    ...
  },
  "throttle_period" : "15m", 
  "actions" : {
    "email_administrator" : {
      "email" : {
        "to" : "sys.admino@host.domain",
        "subject" : "Encountered {{ctx.payload.hits.total}} errors",
        "body" : "Too many error in the system, see attached data",
        "attach_data" : true,
        "priority" : "high"
      }
    },
    "notify_pager" : {
      "webhook" : {
        "method" : "POST",
        "host" : "pager.service.domain",
        "port" : 1234,
        "path" : "/{{watch_id}}",
        "body" : "Encountered {{ctx.payload.hits.total}} errors"
      }
    }
  }
}

There will be at least 15 minutes between subsequent action executions (applies to both email_administrator and notify_pager actions)

If you do not define a throttle period at the action or watch level, the global default throttle period is applied. Initially, this is set to 5 seconds. To change the global default, configure the watcher.execution.default_throttle_period setting in elasticsearch.yml:

watcher.execution.default_throttle_period: 15m

Watcher also supports acknowledgement-based throttling. You can acknowledge a watch using the Ack Watch API to prevent the watch actions from being executed again while the watch condition remains true. This essentially tells Watcher "I received the notification and I’m handling it, please do not notify me about this error again". An acknowledged watch action remains in the acked state until the watch’s condition evaluates to false. When that happens, the action’s state changes to awaits_successful_execution.

To acknowledge an action, you use the ack API:

PUT _watcher/watch/<id>/_ack?actions=<action_ids>

Where <id> is the id of the watch and <action_ids> is a comma-separated list of the action ids you want to acknowledge. To acknowledge all actions, omit the actions parameter.

The following diagram illustrates the throttling decisions made for each action of a watch during its execution:

action throttling

Watcher supports four action types: Email, Webhook, Index and Logging.

Email Actionedit

A watch action that sends email notifications. To use the email action, you must configure at least one email account. For instructions, see Configuring Email Accounts.

See Table 11, “Email Action Attributes” for the supported attributes. Any attributes that are missing from the email action definition are looked up in the configuration of the account from which the email is being sent. The required attributes must either be set in the email action definition or the account’s email_defaults.

Configuring Email Actionsedit

You configure email actions in a watch’s actions array. Action-specific attributes are specified using the email keyword.

The following snippet shows a basic email action definition:

"actions" : {
  "email_admin" : { 
    "transform" : { ... }, 
    "email": {
      "to": "'John Doe <john.doe@example.com>'", 
      "subject": "{{ctx.watch_id}} executed", 
      "body": "{{ctx.watch_id}} executed with {{ctx.payload.hits.total}} hits" 
    }
  }
}

The id of the action.

An optional transform to transform the payload before processing the email.

One or more addresses to send the email to. If not specified, the to address is read from the account’s email_defaults.

The subject of the email (static text or a Mustache template).

The body of the email (static text or a Mustache template).

Table 11. Email Action Attributes

Name Required Default Description

account

no

the default account

The account to use to send the email.

from

no

-

The email address from which the email will be sent. The from field can contain Mustache templates as long as it resolves to a valid email address.

to

yes

-

The email addresses of the to recipients. The to field can contain Mustache templates as long as it resolves to a valid email address.

cc

no

-

The email addresses of the cc recipients. The cc field can contain Mustache templates as long as it resolves to a valid email address.

bcc

no

-

The email addresses of the bcc recipients. The bcc field can contain Mustache templates as long as it resolves to a valid email address.

reply_to

no

-

The email addresses that will be set on the message’s Reply-To header. The reply_to field can contain Mustache templates as long as it resolves to a valid email address.

subject

no

-

The subject of the email. The subject can be static text or contain Mustache templates.

body

no

-

The body of the email. When this field holds a string, it will default to the text body of the email. Set as an object to specify either the text or the html body or both (using the fields bellow)

body.text

yes*

-

The plain text body of the email. The body can be static text or contain Mustache templates.

body.html

yes*

-

The html body of the email. The body can be static text or contain Mustache templates. This body will be sanitized to remove dangerous content such as scripts. This behavior can be disabled by setting watcher.actions.email.sanitize_html: false in elasticsearch.yaml.

priority

no

-

The priority of this email. Valid values are: lowest, low, normal, high and highest. The priority can contain a Mustache template as long as it resolves to one of the valid values.

attach_data

no

false

Indicates whether the watch execution data should be attached to the email. You can specify a Boolean value or an object. If attach_data is set to true, the data is attached as a YAML file called data.yml. If it’s set to false, no data is attached. To control the format of the attached data, specify an object that contains a format field`.

attach_data.format

no

yaml

When attach_data is specified as an object, this field controls the format of the attached data. The supported formats are json and yaml.
  • When setting the body object, at least one of its text or html fields must be defined.
Email Address
An email address can contain two possible parts—​the address itself and an optional personal name as described in RFC 822. The address can be represented either as a string of the form user@host.domain or Personal Name <user@host.domain>. You can also specify an email address as an object that contains name and address fields.
Address List
A list of addresses can either be specified as a comma-delimited string or as an array: 'Personal Name <user1@host.domain>, user2@host.domain' or [ 'Personal Name <user1@host.domain>', 'user2@host.domain' ]

Webhook Actionedit

A watch action that connects to a web server and listens on a specific port. The webhook action supports both HTTP and HTTPS connections. See Table 12, “Webhook Action Attributes” for the supported attributes.

Configuring Webhook Actionsedit

You configure webhook actions in a watch’s actions array. Action-specific attributes are specified using the webhook keyword.

The following snippet shows a simple webhook action definition:

"actions" : {
  "my_webhook" : { 
    "transform" : { ... }, 
    "throttle_period" : "5m", 
    "webhook" : {
      "method" : "POST", 
	    "host" : "mylisteningserver", 
	    "port" : 9200, 
      "path": ":/{{ctx.watch_id}", 
	    "body" : "{{ctx.watch_id}}:{{ctx.payload.hits.total}}" 
    }
  }
}

The id of the action

An optional transform to transform the payload before executing the webhook action

An optional throttle period for the action (5 minutes in this example)

The HTTP method to use when connecting to the host

The host to connect to

The port to connect to

The path (URI) to use in the HTTP request

The body to send with the request

You can use basic authentication when sending a request to a secured webservice. For example:

"actions" : {
  "my_webhook" : {
    "webhook" : {
      "auth" : {
        "basic" : {
          "username" : "<username>", 
          "password" : "<password>" 
        }
      }
      "method" : "POST",
      "host" : "mylisteningserver",
      "port" : 9200,
      "path": ":/{{ctx.watch_id}",
      "body" : "{{ctx.watch_id}}:{{ctx.payload.hits.total}}"
    }
  }
}

The username

The corresponding password

By default, both the username and the password are stored in the .watches index in plain text. When Shield is installed, Watcher can be configured to encrypt the password before storing it.

Query Parametersedit

You can specify query parameters to send with the request with the params field. This field simply holds an object where the keys serve as the parameter names and the values serve as the parameter values:

"actions" : {
  "my_webhook" : {
    "webhook" : {
      "method" : "POST",
      "host" : "mylisteningserver",
      "port" : 9200,
      "path": ":/alert",
      "params" : {
        "watch_id" : "{{ctx.watch_id}}" 
      }
    }
  }
}

The parameter values can contain templated strings.

Custom Request Headersedit

You can specify request headers to send with the request with the headers field. This field simply holds an object where the keys serve as the header names and the values serve as the header values:

"actions" : {
  "my_webhook" : {
    "webhook" : {
      "method" : "POST",
      "host" : "mylisteningserver",
      "port" : 9200,
      "path": ":/alert/{{ctx.watch_id}}",
      "headers" : {
        "Content-Type" : "application/yaml" 
      },
      "body" : "count: {{ctx.payload.hits.total}}"
    }
  }
}

The header values can contain templated strings.

Table 12. Webhook Action Attributes

Name Required Default Description

scheme

no

http

The connection scheme. Valid values are: http or https.

host

yes

-

The host to connect to.

port

yes

-

The port the HTTP service is listening on.

path

no

-

The URL path. The path can be static text or include Mustache templates. URL query string parameters must be specified via the request.params attribute.

method

no

get

The HTTP method. Valid values are: head, get, post, put and delete.

headers

no

-

The HTTP request headers. The header values can be static text or include Mustache templates.

params

no

-

The URL query string parameters. The parameter values can be static text or include Mustache templates.

auth

no

-

Authentication related HTTP headers. Currently, only basic authentication is supported.

body

no

-

The HTTP request body. The body can be static text or include Mustache templates. When not specified, an empty body is sent.

connection_timeout

no

10s

The timeout for setting up the http connection. If the connection could not be set up within this time, the action will timeout and fail. It is also possible to configure the default connection timeout for all http connection timeouts.

Index Actionedit

A watch Action that enable you to index data in Elasticsearch. See Table 13, “Index Action Attributes” for the supported attributes.

Configuring Index Actionsedit

The following snippet shows a simple index action definition:

"actions" : {
  "index_payload" : { 
    "transform": { ... },
    "index" : {
      "index" : "my-index", 
      "doc_type" : "my-type" 
    }
  }
}

The id of the action

An optional transform to transform the payload and prepare the data that should be indexed

The elasticsearch index to store the data to

The document type to store the data as

Table 13. Index Action Attributes

Name Required Default Description

index

yes

-

The Elasticsearch index to index into. Dynamic index names are supported

doc_type

yes

-

The type of the document the data will be indexed as.

execution_time_field

no

_timestamp

The field that will store/index the watch execution time. When not set or when set to _timestamp, the execution time will serve as the document’s timestamp.

timeout

no

60s

The timeout for waiting for the index api call to return. If no response is returned within this time, the index action times out and fails. This setting overrides the default internal index/bulk operations timeouts.

dynamic_name_timezone

no

-

The time zone to use for resolving the index name based on Dynamic Index Names. The default time zone also can be configured globally.

Multi-Document Supportedit

Like with all other actions, you can use a transform to replace the current execution context payload with another and by that change the document that will end up indexed.

The index action plays well with transforms with its support for the special _doc payload field.

When resolving the document to be indexed, the index action first looks up for a _doc field in the payload. When not found, the payload is indexed as a single document.

When a _doc field exists, if the field holds an object, it is extracted and indexed as a single document. If the field holds an array of objects, each object is treated as a document and the index aciton indexes all of them in a bulk.

Logging Actionedit

A watch action that simply logs text to the standard Elasticsearch logs. See Table 14, “Logging Action Attributes” for the supported attributes.

This action is primarily used during development and debugging.

Configuring Logging Actionsedit

You configure logging actions in a watch’s actions array. Action-specific attributes are specified using the logging keyword.

The following snippet shows a simple logging action definition:

"actions" : {
  "log" : { 
    "transform" : { ... }, 
    "logging" : {
      "text" : "executed at {{ctx.execution_time}}" 
    }
  }
}

The id of the action.

An optional transform to transform the payload before executing the logging action.

The text to be logged.

Table 14. Logging Action Attributes

Name Required Default Description

text

yes

-

The text that should be logged. Can be static text or include Mustache templates.

category

no

watcher.actions.logging

The category under which the text will be logged.

level

no

info

The logging level. Valid values are: error, warn, info, debug and trace.
« Condition Transform »