Create a datafeed
Added in 5.4.0
Datafeeds retrieve data from Elasticsearch for analysis by an anomaly detection job.
You can associate only one datafeed with each anomaly detection job.
The datafeed contains a query that runs at a defined interval (frequency
).
If you are concerned about delayed data, you can add a delay (query_delay') at each interval.
By default, the datafeed uses the following query:
{"match_all": {"boost": 1}}`.
When Elasticsearch security features are enabled, your datafeed remembers which roles the user who created it had
at the time of creation and runs the query using those same roles. If you provide secondary authorization headers,
those credentials are used instead.
You must use Kibana, this API, or the create anomaly detection jobs API to create a datafeed. Do not add a datafeed
directly to the .ml-config
index. Do not give users write
privileges on the .ml-config
index.
Path parameters
-
datafeed_id
string Required A numerical character string that uniquely identifies the datafeed. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters.
Query parameters
-
allow_no_indices
boolean If true, wildcard indices expressions that resolve into no concrete indices are ignored. This includes the
_all
string or when no indices are specified. -
expand_wildcards
string | array[string] Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values.
-
ignore_throttled
boolean Deprecated If true, concrete, expanded, or aliased indices are ignored when frozen.
Body
Required
-
aggregations
object If set, the datafeed performs aggregation searches. Support for aggregations is limited and should be used only with low cardinality data.
-
chunking_config
object -
delayed_data_check_config
object -
frequency
string A duration. Units can be
nanos
,micros
,ms
(milliseconds),s
(seconds),m
(minutes),h
(hours) andd
(days). Also accepts "0" without a unit and "-1" to indicate an unspecified value. -
indices
string | array[string] -
indices_options
object -
job_id
string -
max_empty_searches
number If a real-time datafeed has never seen any data (including during any initial training period), it automatically stops and closes the associated job after this many real-time searches return no documents. In other words, it stops after
frequency
timesmax_empty_searches
of real-time operation. If not set, a datafeed with no end time that sees no data remains started until it is explicitly stopped. By default, it is not set. -
query
object An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.
-
query_delay
string A duration. Units can be
nanos
,micros
,ms
(milliseconds),s
(seconds),m
(minutes),h
(hours) andd
(days). Also accepts "0" without a unit and "-1" to indicate an unspecified value. -
runtime_mappings
object -
script_fields
object Specifies scripts that evaluate custom expressions and returns script fields to the datafeed. The detector configuration objects in a job can contain functions that use these script fields.
-
scroll_size
number The size parameter that is used in Elasticsearch searches when the datafeed does not use aggregations. The maximum value is the value of
index.max_result_window
, which is 10,000 by default. -
headers
object
curl \
--request PUT http://api.example.com/_ml/datafeeds/{datafeed_id} \
--header "Authorization: $API_KEY" \
--header "Content-Type: application/json" \
--data '{"aggregations":{},"chunking_config":{"mode":"auto","time_span":"string"},"delayed_data_check_config":{"check_window":"string","enabled":true},"frequency":"string","indices":"string","indices_options":{"allow_no_indices":true,"expand_wildcards":"string","ignore_unavailable":true,"ignore_throttled":true},"job_id":"string","max_empty_searches":42.0,"query":{},"query_delay":"string","runtime_mappings":{"additionalProperty1":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"},"additionalProperty2":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"}},"script_fields":{"additionalProperty1":{"script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true},"additionalProperty2":{"script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true}},"scroll_size":42.0,"headers":{}}'
{
"aggregations": {},
"chunking_config": {
"mode": "auto",
"time_span": "string"
},
"delayed_data_check_config": {
"check_window": "string",
"enabled": true
},
"frequency": "string",
"indices": "string",
"indices_options": {
"allow_no_indices": true,
"expand_wildcards": "string",
"ignore_unavailable": true,
"ignore_throttled": true
},
"job_id": "string",
"max_empty_searches": 42.0,
"query": {},
"query_delay": "string",
"runtime_mappings": {
"additionalProperty1": {
"fields": {
"additionalProperty1": {
"type": "boolean"
},
"additionalProperty2": {
"type": "boolean"
}
},
"fetch_fields": [
{
"field": "string",
"format": "string"
}
],
"format": "string",
"input_field": "string",
"target_field": "string",
"target_index": "string",
"script": {
"source": "string",
"id": "string",
"params": {
"additionalProperty1": {},
"additionalProperty2": {}
},
"": "painless",
"options": {
"additionalProperty1": "string",
"additionalProperty2": "string"
}
},
"type": "boolean"
},
"additionalProperty2": {
"fields": {
"additionalProperty1": {
"type": "boolean"
},
"additionalProperty2": {
"type": "boolean"
}
},
"fetch_fields": [
{
"field": "string",
"format": "string"
}
],
"format": "string",
"input_field": "string",
"target_field": "string",
"target_index": "string",
"script": {
"source": "string",
"id": "string",
"params": {
"additionalProperty1": {},
"additionalProperty2": {}
},
"": "painless",
"options": {
"additionalProperty1": "string",
"additionalProperty2": "string"
}
},
"type": "boolean"
}
},
"script_fields": {
"additionalProperty1": {
"script": {
"source": "string",
"id": "string",
"params": {
"additionalProperty1": {},
"additionalProperty2": {}
},
"": "painless",
"options": {
"additionalProperty1": "string",
"additionalProperty2": "string"
}
},
"ignore_failure": true
},
"additionalProperty2": {
"script": {
"source": "string",
"id": "string",
"params": {
"additionalProperty1": {},
"additionalProperty2": {}
},
"": "painless",
"options": {
"additionalProperty1": "string",
"additionalProperty2": "string"
}
},
"ignore_failure": true
}
},
"scroll_size": 42.0,
"headers": {}
}
{
"aggregations": {},
"authorization": {
"api_key": {
"id": "string",
"name": "string"
},
"roles": [
"string"
],
"service_account": "string"
},
"chunking_config": {
"mode": "auto",
"time_span": "string"
},
"delayed_data_check_config": {
"check_window": "string",
"enabled": true
},
"datafeed_id": "string",
"frequency": "string",
"indices": [
"string"
],
"job_id": "string",
"indices_options": {
"allow_no_indices": true,
"expand_wildcards": "string",
"ignore_unavailable": true,
"ignore_throttled": true
},
"max_empty_searches": 42.0,
"query": {},
"query_delay": "string",
"runtime_mappings": {
"additionalProperty1": {
"fields": {
"additionalProperty1": {
"type": "boolean"
},
"additionalProperty2": {
"type": "boolean"
}
},
"fetch_fields": [
{
"field": "string",
"format": "string"
}
],
"format": "string",
"input_field": "string",
"target_field": "string",
"target_index": "string",
"script": {
"source": "string",
"id": "string",
"params": {
"additionalProperty1": {},
"additionalProperty2": {}
},
"": "painless",
"options": {
"additionalProperty1": "string",
"additionalProperty2": "string"
}
},
"type": "boolean"
},
"additionalProperty2": {
"fields": {
"additionalProperty1": {
"type": "boolean"
},
"additionalProperty2": {
"type": "boolean"
}
},
"fetch_fields": [
{
"field": "string",
"format": "string"
}
],
"format": "string",
"input_field": "string",
"target_field": "string",
"target_index": "string",
"script": {
"source": "string",
"id": "string",
"params": {
"additionalProperty1": {},
"additionalProperty2": {}
},
"": "painless",
"options": {
"additionalProperty1": "string",
"additionalProperty2": "string"
}
},
"type": "boolean"
}
},
"script_fields": {
"additionalProperty1": {
"script": {
"source": "string",
"id": "string",
"params": {
"additionalProperty1": {},
"additionalProperty2": {}
},
"": "painless",
"options": {
"additionalProperty1": "string",
"additionalProperty2": "string"
}
},
"ignore_failure": true
},
"additionalProperty2": {
"script": {
"source": "string",
"id": "string",
"params": {
"additionalProperty1": {},
"additionalProperty2": {}
},
"": "painless",
"options": {
"additionalProperty1": "string",
"additionalProperty2": "string"
}
},
"ignore_failure": true
}
},
"scroll_size": 42.0
}