Inputsedit

A watch input loads data into a watch’s execution context as the initial payload. Watcher supports four input types: simple , search, http and chain

If you don’t define an input for a watch, an empty payload is loaded into the execution context.

Simple Inputedit

An input that enables you to load static data into a watch’s execution context as the initial payload. The simple input is useful when the data you want to work with doesn’t need to be loaded dynamically, but for maintainability you want to store the data centrally and reference it with templates.

You can define the static data as a string (str), numeric value (num), or an object (obj):

{
  "input" : {
    "simple" : {
      "str" : "val1",
      "num" : 23,
      "obj" : {
        "str" : "val2"
      }
    }
  }
  ...
}

For example, the following watch uses the simple input to set the recipient name for a reminder email that’s sent every day at noon.

{
  "trigger" : {
    "schedule" : {
      "daily" : { "at" : "noon" }
    }
  },
  "input" : {
    "simple" : {
      "name" : "John"
    }
  },
  "actions" : {
    "reminder_email" : {
      "email" : {
        "to" : "to@host.domain",
        "subject" : "Reminder",
        "body" : "Dear {{ctx.payload.name}}, by the time you read these lines, I'll be gone"
      }
    }
  }
}

Search Inputedit

An input that enables you to search the Elasticsearch cluster that Watcher is running on and load the response into a watch’s execution context as the initial payload. See Table 2, “Search Input Attributes” for the supported attributes.

Conditions, transforms, and actions can access the search results through the watch execution context. For example:

  • To load all of the search hits into an email body, use ctx.payload.hits.
  • To reference the total number of hits, use ctx.payload.hits.total.
  • To access a particular hit, use its zero-based array index. For example, to get the third hit, use ctx.payload.hits.hits.2.
  • To get a field value from a particular hit, use ctx.payload.hits.hits.<index>.fields.<fieldname>. For example, to get the message field from the first hit, use ctx.payload.hits.hits.0.fields.message.

Table 2. Search Input Attributes

Name Required Default Description

request.search_type

no

query_then_fetch

The type of search request to perform. Valid values are: dfs_query_and_fetch, dfs_query_then_fetch, query_and_fetch, query_then_fetch, and scan. The Elasticsearch default is query_then_fetch.

request.indices

no

-

The indices to search. If omitted, all indices are searched, which is the default behaviour in Elasticsearch.

request.types

no

-

The document types to search for. If omitted, all document types are are searched, which is the default behaviour in Elasticsearch.

request.body

no

-

The body of the request. The request body follows the same structure you normally send in the body of a REST _search request. The body can be static text or include mustache templates.

request.template

no

-

The body of the search template. See configure templates for more information.

request.indices_options.expand_wildcards

no

open

How to expand wildcards. Valid values are: all, open, closed, and none See expand_wildcards for more information.

request.indices_options.ignore_unavailable

no

true

Whether the search should ignore unavailable indices. See ignore_unavailable for more information.

request.indices_options.allow_no_indices

no

true

Whether to allow a search where a wildcard indices expression results in no concrete indices. See allow_no_indices for more information.

extract

no

-

A array of JSON keys to extract from the search response and load as the payload. When a search generates a large response, you can use extract to select the relevant fields instead of loading the entire response.

timeout

no

30s

The timeout for waiting for the search api call to return. If no response is returned within this time, the search input times out and fails. This setting overrides the default internal search operations timeouts.

You can reference the following variables in the execution context when specifying the request body:

Name Description

ctx.watch_id

The id of the watch that is currently executing.

ctx.execution_time

The time execution of this watch started.

ctx.trigger.triggered_time

The time this watch was triggered.

ctx.trigger.scheduled_time

The time this watch was supposed to be triggered.

ctx.metadata.*

Any metadata associated with the watch.

Submitting Searchesedit

You can use the search input to submit any valid search request to your Elasticsearch cluster. For example, the following snippet returns all event documents in the logs index.

"input" : {
  "search" : {
    "request" : {
      "indices" : [ "logs" ],
      "types" : [ "event" ],
      "body" : {
        "query" : { "match_all" : {}}
      }
    }
  }
}

Extracting Specific Fieldsedit

You can specify which fields in the search response you want to load into the watch payload with the extract attribute. This is useful when a search generates a large response and you are only interested in particular fields.

For example, the following input loads only the total number of hits into the watch payload:

"input": {
    "search": {
      "request": {
        "indices": [".watch_history*"]
      },
      "extract": ["hits.total"]
    }
  },

Using Templatesedit

The search input supports search templates. For example, the following snippet references the indexed template called my_template and passes a value of 23 to fill in the template’s value parameter.

{
  "input" : {
    "search" : {
      "request" : {
        "indices" : [ "logs" ],
        "template" : {
          "id" : "my_template",
          "params" : {
            "value" : 23
          }
        }
      }
    }
  }
  ...
}

Applying Conditionsedit

The search input is often used in conjunction with the script condition. For example, the following snippet adds a condition to check if the search returned more than five hits

{
  "input" : {
    "search" : {
      "request" : {
        "indices" : [ "logs" ],
        "body" : {
          "query" : { "match_all" : {} }
        }
      }
    }
  },
  "condition" : {
    "script" : "return ctx.payload.hits.total > 5"
  }
  ...
}

HTTP Inputedit

An input that enables you to query an HTTP endpoint and load theresponse into a watch’s execution context as the initial payload. See Table 3, “HTTP Input Attributes” for the supported attributes.

For example, you can use the http input to:

  • Query an external Elasticsearch cluster. The query that one can define in the search input can also defined in the http input. This in particular interesting as one can query clusters that are version wise incompatible with the cluster where Watcher is running on. With the search input these cluster would otherwise not be accessible. Also with the http input it is straight forward to run a dedicated Watcher cluster that queries other clusters.
  • Query Elasticsearch APIs other than the search API. For example, the Nodes Stats, Cluster Health or Cluster State APIs.
  • Query an external webservice. Any service that exposes an HTTP endpoint can be queried by the http input. This can be very useful when you need to bridge between an Elasticsearch cluster and other systems.

Conditions, transforms, and actions can access the JSON response through the watch execution context. For example, if the response contains a message object, you could use ctx.payload.message to get the message from the payload.

If the body of the response from the HTTP endpoint is in the JSON or YAML format it will be parsed and used as the initial payload. Any other value that is returned will be assigned and accessible to/via the _value variable of the payload.

Table 3. HTTP Input Attributes

Name Required Default Description

request.scheme

no

http

Url scheme. Valid values are: http or https.

request.host

yes

-

The host to connect to.

request.port

yes

-

The port the http service is listening on.

request.path

no

-

The URL path. The path can be static text or contain mustache templates. URL query string parameters must be specified via the request.params attribute.

request.method

no

get

The HTTP method. Supported values are: head, get, post, put and delete.

request.headers

no

-

The HTTP request headers. The header values can be static text or include mustache templates.

request.params

no

-

The URL query string parameters. The parameter values can be static text or contain mustache templates.

request.url

no

-

Allows you to set request.scheme, request.host, request.port and request.params add once by specifying a real URL, like https://www.example.org:1234/mypath?foo=bar. May not be combined with on of those four parameters. As those parameters are set, specifying them individually might overwrite them.

request.auth

no

-

Authentication related HTTP headers. Currently, only basic authentication is supported.

request.proxy.host

no

-

The proxy host to use when connecting to the host.

request.proxy.port

no

-

The proxy port to use when connecting to the host.

request.connection_timeout

no

10s

The timeout for setting up the http connection. If the connection could not be set up within this time, the input will timeout and fail. It is also possible to configure the default connection timeout for all http connection timeouts.

request.read_timeout

no

10s

The timeout for reading data from http connection. If no response was received within this time, the input will timeout and fail. It is also possible to configure the default read timeout for all http connection timeouts.

request.body

no

-

The HTTP request body. The body can be static text or include mustache templates.

extract

no

-

A array of JSON keys to extract from the input response and use as payload. In cases when an input generates a large response this can be used to filter the relevant piece of the response to be used as payload.

response_content_type

no

json

The expected content type the response body will contain. Supported values are json, yaml and text. If the format is text the extract attribute cannot exist. Note that this overrides the header that is returned in the HTTP response. If this is set to text the body of the response will be assigned and accessible to/via the _value variable of the payload.

You can reference the following variables in the execution context when specifying the path, params, headers and body values:

Name Description

ctx.watch_id

The id of the watch that is currently executing.

ctx.execution_time

The time execution of this watch started.

ctx.trigger.triggered_time

The time this watch was triggered.

ctx.trigger.scheduled_time

The time this watch was supposed to be triggered.

ctx.metadata.*

Any metadata associated with the watch.

Accessing the HTTP Responseedit

If the response body is formatted in JSON or YAML, it is parsed and loaded into the execution context. If the response body is not formatted in JSON or YAML, it is loaded into the payload’s _value field.

Conditions, transforms, and actions access the response data through the execution context. For example, if the response contains a message object, you can use ctx.payload.message to access the message data.

In addition all the headers from the response can be accessed using the ctx.payload._headers field as well as the HTTP status code of the response using ctx.payload._status_code.

Querying External Elasticsearch Clustersedit

The following snippet shows a basic http input that searches for all documents in the idx index in an external cluster:

"input" : {
  "http" : {
    "request" : {
      "host" : "example.com",
      "port" : 9200,
      "path" : "/idx/_search"
    }
  }
}

You can use the full Elasticsearch Query DSL to perform more sophisticated searches. For example, the following snippet retrieves all documents that contain event in the category field.

"input" : {
  "http" : {
    "request" : {
      "host" : "host.domain",
      "port" : 9200,
      "path" : "/idx/_search",
      "body" :  "{\"query\" :  {  \"match\" : { \"category\" : \"event\"}}}"
    }
  }
}

Using Templatesedit

The http input supports templating. You can use templates when specifying the path, body, header values, and parameter values.

For example, the following snippet uses templates to specify what index to query and restrict the results to documents added within the last five minutes.

"input" : {
  "http" : {
    "request" : {
      "host" : "host.domain",
      "port" : 9200,
      "path" : "/{{ctx.watch_id}}/_search",
      "body" : "{\"query\" : {\"range\": {\"@timestamp\" : {\"from\": \"{{ctx.trigger.triggered_time}}||-5m\",\"to\": \"{{ctx.trigger.triggered_time}}\"}}}}"
      }
    }
  }

Calling Elasticsearch APIsedit

You can use http input load the data returned by any Elasticsearch API. For example, the following snippet calls the Cluster Stats API and passes in the human query string argument.

Http Input.

"input" : {
  "http" : {
    "request" : {
      "host" : "host.domain",
      "port" : "9200",
      "path" : "/_cluster/stats",
      "params" : {
        "human" : "true" 
      }
    }
  }
}

Enabling this attribute returns the bytes values in the response in human readable format.

Calling External Webservicesedit

You can use http input to get data from any external webservice. The http input supports basic authentication. For example, the following snippet calls myservice and uses basic authentication:

Http Input.

"input" : {
  "http" : {
    "request" : {
      "host" : "host.domain",
      "port" : "9200",
      "path" : "/myservice",
      "auth" : {
        "basic" : {
          "username" : "user",
          "password" : "pass"
        }
      }
    }
  }
}

Chain Inputedit

An input that enables you to string together multiple inputs to load data into the watch execution context. Each input loads data into the execution context and that data is available to subsequent inputs in the chain.

The chain input is useful when you want to perform an action based on data from multiple sources. You can also use the data collected by one input to load data from another source.

To reference data loaded by a particular input, you use the input’s name, ctx.payload.<input-name>.<value>.

For example, the following chain input loads data from an HTTP server using the path set by a simple input.

{
  "input" : {
    "chain" : {
      "inputs" : [ 
        {
          "first" : {
            "simple" : { "path" : "/_search" }
          }
        },
        {
          "second" : {
            "http" : {
              "request" : {
                "host" : "localhost",
                "port" : 9200,
                "path" : "{{ctx.payload.first.path}}" 
              }
            }
          }
        }
      ]
    }
  }
  ...
}

The inputs in a chain are specified as an array to guarantee the order in which the inputs are processed. (JSON does not guarantee the order of arbitrary objects.)

Loads the path set by the first input.