Get field data cache information

GET /_cat/fielddata
Api key auth Basic auth Bearer auth

Get the amount of heap memory currently used by the field data cache on every data node in the cluster.

IMPORTANT: cat APIs are only intended for human consumption using the command line or Kibana console. They are not intended for use by applications. For application consumption, use the nodes stats API.

Query parameters

  • bytes string

    The unit used to display byte values.

    Values are b, kb, mb, gb, tb, or pb.

  • fields string | array[string]

    Comma-separated list of fields used to limit returned information.

  • h string | array[string]

    List of columns to appear in the response. Supports simple wildcards.

  • s string | array[string]

    List of columns that determine how the table should be sorted. Sorting defaults to ascending and can be changed by setting :asc or :desc as a suffix to the column name.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
GET /_cat/fielddata 
curl \
 --request GET 'http://api.example.com/_cat/fielddata' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response from `GET /_cat/fielddata?v=true&fields=body&format=json`. You can specify an individual field in the request body or URL path. This example retrieves heap memory size information for the `body` field. 
[
  {
    "id": "Nqk-6inXQq-OxUfOUI8jNQ",
    "host": "127.0.0.1",
    "ip": "127.0.0.1",
    "node": "Nqk-6in",
    "field": "body",
    "size": "544b"
  }
]
A successful response from `GET /_cat/fielddata/body,soul?v=true&format=json`. You can specify a comma-separated list of fields in the request body or URL path. This example retrieves heap memory size information for the `body` and `soul` fields. To get information for all fields, run `GET /_cat/fielddata?v=true`. 
[
  {
    "id": "Nqk-6inXQq-OxUfOUI8jNQ",
    "host": "1127.0.0.1",
    "ip": "127.0.0.1",
    "node": "Nqk-6in",
    "field": "body",
    "size": "544b"
  },
  {
    "id": "Nqk-6inXQq-OxUfOUI8jNQ",
    "host": "127.0.0.1",
    "ip": "127.0.0.1",
    "node": "Nqk-6in",
    "field": "soul",
    "size": "480b"
  }
]

Get thread pool statistics

GET /_cat/thread_pool
Api key auth Basic auth Bearer auth

Get thread pool statistics for each node in a cluster. Returned information includes all built-in thread pools and custom thread pools. IMPORTANT: cat APIs are only intended for human consumption using the command line or Kibana console. They are not intended for use by applications. For application consumption, use the nodes info API.

Query parameters

  • h string | array[string]

    List of columns to appear in the response. Supports simple wildcards.

  • s string | array[string]

    List of columns that determine how the table should be sorted. Sorting defaults to ascending and can be changed by setting :asc or :desc as a suffix to the column name.

  • time string

    The unit used to display time values.

    Values are nanos, micros, ms, s, m, h, or d.

  • local boolean

    If true, the request computes the list of selected nodes from the local cluster state. If false the list of selected nodes are computed from the cluster state of the master node. In both cases the coordinating node will send requests for further information to each selected node.

  • master_timeout string

    Period to wait for a connection to the master node.

Responses

GET /_cat/thread_pool 
curl \
 --request GET 'http://api.example.com/_cat/thread_pool' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response from `GET /_cat/thread_pool?format=json`. 
[
  {
    "node_name": "node-0",
    "name": "analyze",
    "active": "0",
    "queue": "0",
    "rejected": "0"
  },
  {
    "node_name": "node-0",
    "name": "fetch_shard_started",
    "active": "0",
    "queue": "0",
    "rejected": "0"
  },
  {
    "node_name": "node-0",
    "name": "fetch_shard_store",
    "active": "0",
    "queue": "0",
    "rejected": "0"
  },
  {
    "node_name": "node-0",
    "name": "flush",
    "active": "0",
    "queue": "0",
    "rejected": "0"
  },
  {
    "node_name": "node-0",
    "name": "write",
    "active": "0",
    "queue": "0",
    "rejected": "0"
  }
]
A successful response from `GET /_cat/thread_pool/generic?v=true&h=id,name,active,rejected,completed&format=json`. It returns the `id`, `name`, `active`, `rejected`, and `completed` columns. It also limits returned information to the generic thread pool. 
[
  {
    "id": "0EWUhXeBQtaVGlexUeVwMg",
    "name": "generic",
    "active": "0",
    "rejected": "0",
    "completed": "70"
  }
]

Get data stream stats Added in 7.9.0

GET /_data_stream/{name}/_stats
Api key auth Basic auth Bearer auth

Get statistics for one or more data streams.

Path parameters

  • name string Required

    Comma-separated list of data streams used to limit the request. Wildcard expressions (*) are supported. To target all data streams in a cluster, omit this parameter or use *.

Query parameters

  • expand_wildcards string | array[string]

    Type of data stream that wildcard patterns can match. Supports comma-separated values, such as open,hidden.

    Supported values include:

    • all: Match any data stream or index, including hidden ones.
    • open: Match open, non-hidden indices. Also matches any non-hidden data stream.
    • closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.
    • hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.
    • none: Wildcard expressions are not accepted.

Responses

GET /_data_stream/{name}/_stats 
curl \
 --request GET 'http://api.example.com/_data_stream/{name}/_stats' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response for retrieving statistics for a data stream. 
{
  "_shards": {
    "total": 10,
    "successful": 5,
    "failed": 0
  },
  "data_stream_count": 2,
  "backing_indices": 5,
  "total_store_size": "7kb",
  "total_store_size_bytes": 7268,
  "data_streams": [
    {
      "data_stream": "my-data-stream",
      "backing_indices": 3,
      "store_size": "3.7kb",
      "store_size_bytes": 3772,
      "maximum_timestamp": 1607512028000
    },
    {
      "data_stream": "my-data-stream-two",
      "backing_indices": 2,
      "store_size": "3.4kb",
      "store_size_bytes": 3496,
      "maximum_timestamp": 1607425567000
    }
  ]
}

Update data stream lifecycles Added in 8.11.0

PUT /_data_stream/{name}/_lifecycle
Api key auth Basic auth Bearer auth

Update the data stream lifecycle of the specified data streams.

Path parameters

  • name string | array[string] Required

    Comma-separated list of data streams used to limit the request. Supports wildcards (*). To target all data streams use * or _all.

Query parameters

  • expand_wildcards string | array[string]

    Type of data stream that wildcard patterns can match. Supports comma-separated values, such as open,hidden. Valid values are: all, hidden, open, closed, none.

    Supported values include:

    • all: Match any data stream or index, including hidden ones.
    • open: Match open, non-hidden indices. Also matches any non-hidden data stream.
    • closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.
    • hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.
    • none: Wildcard expressions are not accepted.
  • master_timeout string

    Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

  • timeout string

    Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

application/json

Body

  • data_retention string

    A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

  • downsampling object
    Hide downsampling attribute Show downsampling attribute object
    • rounds array[object] Required

      The list of downsampling rounds to execute as part of this downsampling configuration

      Hide rounds attributes Show rounds attributes object
      • after string Required

        A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • config object Required
        Hide config attribute Show config attribute object
        • fixed_interval string Required

          A date histogram interval. Similar to Duration with additional units: w (week), M (month), q (quarter) and y (year)

  • enabled boolean

    If defined, it turns data stream lifecycle on/off (true/false) for this data stream. A data stream lifecycle that's disabled (enabled: false) will have no effect on the data stream.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • acknowledged boolean Required

      For a successful response, this value is always true. On failure, an exception is returned instead.
PUT /_data_stream/{name}/_lifecycle 
curl \
 --request PUT 'http://api.example.com/_data_stream/{name}/_lifecycle' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '"{\n  \"data_retention\": \"7d\"\n}"'
Request examples 
{
  "data_retention": "7d"
}
This example configures two downsampling rounds. 
{
    "downsampling": [
      {
        "after": "1d",
        "fixed_interval": "10m"
      },
      {
        "after": "7d",
        "fixed_interval": "1d"
      }
    ]
}
Response examples (200)
A successful response for configuring a data stream lifecycle. 
{
  "acknowledged": true
}

Get a document by its ID

GET /{index}/_doc/{id}
Api key auth Basic auth Bearer auth

Get a document and its source or stored fields from an index.

By default, this API is realtime and is not affected by the refresh rate of the index (when data will become visible for search). In the case where stored fields are requested with the stored_fields parameter and the document has been updated but is not yet refreshed, the API will have to parse and analyze the source to extract the stored fields. To turn off realtime behavior, set the realtime parameter to false.

Source filtering

By default, the API returns the contents of the _source field unless you have used the stored_fields parameter or the _source field is turned off. You can turn off _source retrieval by using the _source parameter:

GET my-index-000001/_doc/0?_source=false

If you only need one or two fields from the _source, use the _source_includes or _source_excludes parameters to include or filter out particular fields. This can be helpful with large documents where partial retrieval can save on network overhead Both parameters take a comma separated list of fields or wildcard expressions. For example:

GET my-index-000001/_doc/0?_source_includes=*.id&_source_excludes=entities

If you only want to specify includes, you can use a shorter notation:

GET my-index-000001/_doc/0?_source=*.id

Routing

If routing is used during indexing, the routing value also needs to be specified to retrieve a document. For example:

GET my-index-000001/_doc/2?routing=user1

This request gets the document with ID 2, but it is routed based on the user. The document is not fetched if the correct routing is not specified.

Distributed

The GET operation is hashed into a specific shard ID. It is then redirected to one of the replicas within that shard ID and returns the result. The replicas are the primary shard and its replicas within that shard ID group. This means that the more replicas you have, the better your GET scaling will be.

Versioning support

You can use the version parameter to retrieve the document only if its current version is equal to the specified one.

Internally, Elasticsearch has marked the old document as deleted and added an entirely new document. The old version of the document doesn't disappear immediately, although you won't be able to access it. Elasticsearch cleans up deleted documents in the background as you continue to index more data.

Path parameters

  • index string Required

    The name of the index that contains the document.

  • id string Required

    A unique document identifier.

Query parameters

  • preference string

    The node or shard the operation should be performed on. By default, the operation is randomized between the shard replicas.

    If it is set to _local, the operation will prefer to be run on a local allocated shard when possible. If it is set to a custom value, the value is used to guarantee that the same shards will be used for the same custom value. This can help with "jumping values" when hitting different shards in different refresh states. A sample value can be something like the web session ID or the user name.

  • realtime boolean

    If true, the request is real-time as opposed to near-real-time.

  • refresh boolean

    If true, the request refreshes the relevant shards before retrieving the document. Setting it to true should be done after careful thought and verification that this does not cause a heavy load on the system (and slow down indexing).

  • routing string

    A custom value used to route operations to a specific shard.

  • _source boolean | string | array[string]

    Indicates whether to return the _source field (true or false) or lists the fields to return.

  • _source_excludes string | array[string]

    A comma-separated list of source fields to exclude from the response. You can also use this parameter to exclude fields from the subset specified in _source_includes query parameter. If the _source parameter is false, this parameter is ignored.

  • _source_includes string | array[string]

    A comma-separated list of source fields to include in the response. If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the _source_excludes query parameter. If the _source parameter is false, this parameter is ignored.

  • stored_fields string | array[string]

    A comma-separated list of stored fields to return as part of a hit. If no fields are specified, no stored fields are included in the response. If this field is specified, the _source parameter defaults to false. Only leaf fields can be retrieved with the stored_field option. Object fields can't be returned;​if specified, the request fails.

  • version number

    The version number for concurrency control. It must match the current version of the document for the request to succeed.

  • version_type string

    The version type.

    Supported values include:

    • internal: Use internal versioning that starts at 1 and increments with each update or delete.
    • external: Only index the document if the specified version is strictly higher than the version of the stored document or if there is no existing document.
    • external_gte: Only index the document if the specified version is equal or higher than the version of the stored document or if there is no existing document. NOTE: The external_gte version type is meant for special use cases and should be used with care. If used incorrectly, it can result in loss of data.
    • force: This option is deprecated because it can cause primary and replica shards to diverge.

    Values are internal, external, external_gte, or force.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • _index string Required
    • fields object

      If the stored_fields parameter is set to true and found is true, it contains the document fields stored in the index.

      Hide fields attribute Show fields attribute object
      • * object Additional properties
    • _ignored array[string]
    • found boolean Required

      Indicates whether the document exists.

    • _id string Required
    • _primary_term number

      The primary term assigned to the document for the indexing operation.

    • _routing string

      The explicit routing, if set.

    • _seq_no number
    • _source object

      If found is true, it contains the document data formatted in JSON. If the _source parameter is set to false or the stored_fields parameter is set to true, it is excluded.

    • _version number
GET /{index}/_doc/{id} 
curl \
 --request GET 'http://api.example.com/{index}/_doc/{id}' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response from `GET my-index-000001/_doc/0`. It retrieves the JSON document with the `_id` 0 from the `my-index-000001` index. 
{
  "_index": "my-index-000001",
  "_id": "0",
  "_version": 1,
  "_seq_no": 0,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "@timestamp": "2099-11-15T14:12:12",
    "http": {
      "request": {
        "method": "get"
      },
      "response": {
        "status_code": 200,
        "bytes": 1070000
      },
      "version": "1.1"
    },
    "source": {
      "ip": "127.0.0.1"
    },
    "message": "GET /search HTTP/1.1 200 1070000",
    "user": {
      "id": "kimchy"
    }
  }
}
A successful response from `GET my-index-000001/_doc/1?stored_fields=tags,counter`, which retrieves a set of stored fields. Field values fetched from the document itself are always returned as an array. Any requested fields that are not stored (such as the counter field in this example) are ignored. 
{
  "_index": "my-index-000001",
  "_id": "1",
  "_version": 1,
  "_seq_no" : 22,
  "_primary_term" : 1,
  "found": true,
  "fields": {
      "tags": [
        "production"
      ]
  }
}
A successful response from `GET my-index-000001/_doc/2?routing=user1&stored_fields=tags,counter`, which retrieves the `_routing` metadata field. 
{
  "_index": "my-index-000001",
  "_id": "2",
  "_version": 1,
  "_seq_no" : 13,
  "_primary_term" : 1,
  "_routing": "user1",
  "found": true,
  "fields": {
      "tags": [
        "env2"
      ]
  }
}

Check a document

HEAD /{index}/_doc/{id}
Api key auth Basic auth Bearer auth

Verify that a document exists. For example, check to see if a document with the _id 0 exists:

HEAD my-index-000001/_doc/0

If the document exists, the API returns a status code of 200 - OK. If the document doesn’t exist, the API returns 404 - Not Found.

Versioning support

You can use the version parameter to check the document only if its current version is equal to the specified one.

Internally, Elasticsearch has marked the old document as deleted and added an entirely new document. The old version of the document doesn't disappear immediately, although you won't be able to access it. Elasticsearch cleans up deleted documents in the background as you continue to index more data.

Path parameters

  • index string Required

    A comma-separated list of data streams, indices, and aliases. It supports wildcards (*).

  • id string Required

    A unique document identifier.

Query parameters

  • preference string

    The node or shard the operation should be performed on. By default, the operation is randomized between the shard replicas.

    If it is set to _local, the operation will prefer to be run on a local allocated shard when possible. If it is set to a custom value, the value is used to guarantee that the same shards will be used for the same custom value. This can help with "jumping values" when hitting different shards in different refresh states. A sample value can be something like the web session ID or the user name.

  • realtime boolean

    If true, the request is real-time as opposed to near-real-time.

  • refresh boolean

    If true, the request refreshes the relevant shards before retrieving the document. Setting it to true should be done after careful thought and verification that this does not cause a heavy load on the system (and slow down indexing).

  • routing string

    A custom value used to route operations to a specific shard.

  • _source boolean | string | array[string]

    Indicates whether to return the _source field (true or false) or lists the fields to return.

  • _source_excludes string | array[string]

    A comma-separated list of source fields to exclude from the response. You can also use this parameter to exclude fields from the subset specified in _source_includes query parameter. If the _source parameter is false, this parameter is ignored.

  • _source_includes string | array[string]

    A comma-separated list of source fields to include in the response. If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the _source_excludes query parameter. If the _source parameter is false, this parameter is ignored.

  • stored_fields string | array[string]

    A comma-separated list of stored fields to return as part of a hit. If no fields are specified, no stored fields are included in the response. If this field is specified, the _source parameter defaults to false.

  • version number

    Explicit version number for concurrency control. The specified version must match the current version of the document for the request to succeed.

  • version_type string

    The version type.

    Supported values include:

    • internal: Use internal versioning that starts at 1 and increments with each update or delete.
    • external: Only index the document if the specified version is strictly higher than the version of the stored document or if there is no existing document.
    • external_gte: Only index the document if the specified version is equal or higher than the version of the stored document or if there is no existing document. NOTE: The external_gte version type is meant for special use cases and should be used with care. If used incorrectly, it can result in loss of data.
    • force: This option is deprecated because it can cause primary and replica shards to diverge.

    Values are internal, external, external_gte, or force.

Responses

HEAD /{index}/_doc/{id} 
HEAD my-index-000001/_doc/0

curl -I "localhost:9200/my-index-000001/_doc/0?pretty"

const response = await client.exists({
  index: "my-index-000001",
  id: 0,
});
console.log(response);

resp = client.exists(
  index="my-index-000001",
  id="0",
)
print(resp)

response = client.exists(
  index: 'my-index-000001',
  id: 0
)
puts response

Get enrich stats Added in 7.5.0

GET /_enrich/_stats
Api key auth Basic auth Bearer auth

Returns enrich coordinator statistics and information about enrich policies that are currently executing.

Query parameters

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • coordinator_stats array[object] Required

      Objects containing information about each coordinating ingest node for configured enrich processors.

      Hide coordinator_stats attributes Show coordinator_stats attributes object
    • executing_policies array[object] Required

      Objects containing information about each enrich policy that is currently executing.

      Hide executing_policies attributes Show executing_policies attributes object
      • name string Required
      • task object Required Additional properties
        Hide task attributes Show task attributes object
        • action string Required
        • cancelled boolean
        • cancellable boolean Required
        • description string

          Human readable text that identifies the particular request that the task is performing. For example, it might identify the search request being performed by a search task. Other kinds of tasks have different descriptions, like _reindex which has the source and the destination, or _bulk which just has the number of requests and the destination indices. Many requests will have only an empty description because more detailed information about the request is not easily available or particularly helpful in identifying the request.

        • headers object Required
          Hide headers attribute Show headers attribute object
          • * string Additional properties
        • id number Required
        • node string Required
        • running_time string

          A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • running_time_in_nanos number

          Time unit for nanoseconds

        • start_time_in_millis number

          Time unit for milliseconds

        • status object

          The internal status of the task, which varies from task to task. The format also varies. While the goal is to keep the status for a particular task consistent from version to version, this is not always possible because sometimes the implementation changes. Fields might be removed from the status for a particular request so any parsing you do of the status might break in minor releases.

        • type string Required

        • parent_task_id string | number

          One of:
          TaskId string TaskId number
    • cache_stats array[object]

      Objects containing information about the enrich cache stats on each ingest node.

      Hide cache_stats attributes Show cache_stats attributes object
GET /_enrich/_stats 
curl \
 --request GET 'http://api.example.com/_enrich/_stats' \
 --header "Authorization: $API_KEY"

Import a dangling index Added in 7.9.0

POST /_dangling/{index_uuid}
Api key auth Basic auth Bearer auth

If Elasticsearch encounters index data that is absent from the current cluster state, those indices are considered to be dangling. For example, this can happen if you delete more than cluster.indices.tombstones.size indices while an Elasticsearch node is offline.

Path parameters

  • index_uuid string Required

    The UUID of the index to import. Use the get dangling indices API to locate the UUID.

Query parameters

  • accept_data_loss boolean Required

    This parameter must be set to true to import a dangling index. Because Elasticsearch cannot know where the dangling index data came from or determine which shard copies are fresh and which are stale, it cannot guarantee that the imported data represents the latest state of the index when it was last in the cluster.

  • master_timeout string

    Specify timeout for connection to master

  • timeout string

    Explicit operation timeout

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • acknowledged boolean Required

      For a successful response, this value is always true. On failure, an exception is returned instead.
POST /_dangling/{index_uuid} 
curl \
 --request POST 'http://api.example.com/_dangling/{index_uuid}?accept_data_loss=true' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response from `POST /_dangling/zmM4e0JtBkeUjiHD-MihPQ?accept_data_loss=true`. 
{
  "acknowledged": true
}

Flush data streams or indices

GET /_flush
Api key auth Basic auth Bearer auth

Flushing a data stream or index is the process of making sure that any data that is currently only stored in the transaction log is also permanently stored in the Lucene index. When restarting, Elasticsearch replays any unflushed operations from the transaction log into the Lucene index to bring it back into the state that it was in before the restart. Elasticsearch automatically triggers flushes as needed, using heuristics that trade off the size of the unflushed transaction log against the cost of performing each flush.

After each operation has been flushed it is permanently stored in the Lucene index. This may mean that there is no need to maintain an additional copy of it in the transaction log. The transaction log is made up of multiple files, called generations, and Elasticsearch will delete any generation files when they are no longer needed, freeing up disk space.

It is also possible to trigger a flush on one or more indices using the flush API, although it is rare for users to need to call this API directly. If you call the flush API after indexing some documents then a successful response indicates that Elasticsearch has flushed all the documents that were indexed before the flush API was called.

Query parameters

  • allow_no_indices boolean

    If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices.

  • expand_wildcards string | array[string]

    Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as open,hidden. Valid values are: all, open, closed, hidden, none.

    Supported values include:

    • all: Match any data stream or index, including hidden ones.
    • open: Match open, non-hidden indices. Also matches any non-hidden data stream.
    • closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.
    • hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.
    • none: Wildcard expressions are not accepted.
  • force boolean

    If true, the request forces a flush even if there are no changes to commit to the index.

  • ignore_unavailable boolean

    If false, the request returns an error if it targets a missing or closed index.

  • wait_if_ongoing boolean

    If true, the flush operation blocks until execution when another flush operation is running. If false, Elasticsearch returns an error if you request a flush when another flush operation is running.

Responses

GET /_flush 
curl \
 --request GET 'http://api.example.com/_flush' \
 --header "Authorization: $API_KEY"

Get index statistics Added in 1.3.0

GET /_stats/{metric}
Api key auth Basic auth Bearer auth

For data streams, the API retrieves statistics for the stream's backing indices.

By default, the returned statistics are index-level with primaries and total aggregations. primaries are the values for only the primary shards. total are the accumulated values for both primary and replica shards.

To get shard-level statistics, set the level parameter to shards.

NOTE: When moving to another node, the shard-level statistics for a shard are cleared. Although the shard is no longer part of the node, that node retains any node-level statistics to which the shard contributed.

Path parameters

  • metric string | array[string] Required

    Limit the information returned the specific metrics.

Query parameters

  • completion_fields string | array[string]

    Comma-separated list or wildcard expressions of fields to include in fielddata and suggest statistics.

  • expand_wildcards string | array[string]

    Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as open,hidden.

    Supported values include:

    • all: Match any data stream or index, including hidden ones.
    • open: Match open, non-hidden indices. Also matches any non-hidden data stream.
    • closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.
    • hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.
    • none: Wildcard expressions are not accepted.
  • fielddata_fields string | array[string]

    Comma-separated list or wildcard expressions of fields to include in fielddata statistics.

  • fields string | array[string]

    Comma-separated list or wildcard expressions of fields to include in the statistics.

  • forbid_closed_indices boolean

    If true, statistics are not collected from closed indices.

  • groups string | array[string]

    Comma-separated list of search groups to include in the search statistics.

  • include_segment_file_sizes boolean

    If true, the call reports the aggregated disk usage of each one of the Lucene index files (only applies if segment stats are requested).

  • include_unloaded_segments boolean

    If true, the response includes information from segments that are not loaded into memory.

  • level string

    Indicates whether statistics are aggregated at the cluster, index, or shard level.

    Values are cluster, indices, or shards.

Responses

GET /_stats/{metric} 
curl \
 --request GET 'http://api.example.com/_stats/{metric}' \
 --header "Authorization: $API_KEY"

Simulate data ingestion Technical preview

GET /_ingest/{index}/_simulate
Api key auth Basic auth Bearer auth

Run ingest pipelines against a set of provided documents, optionally with substitute pipeline definitions, to simulate ingesting data into an index.

This API is meant to be used for troubleshooting or pipeline development, as it does not actually index any data into Elasticsearch.

The API runs the default and final pipeline for that index against a set of documents provided in the body of the request. If a pipeline contains a reroute processor, it follows that reroute processor to the new index, running that index's pipelines as well the same way that a non-simulated ingest would. No data is indexed into Elasticsearch. Instead, the transformed document is returned, along with the list of pipelines that have been run and the name of the index where the document would have been indexed if this were not a simulation. The transformed document is validated against the mappings that would apply to this index, and any validation error is reported in the result.

This API differs from the simulate pipeline API in that you specify a single pipeline for that API, and it runs only that one pipeline. The simulate pipeline API is more useful for developing a single pipeline, while the simulate ingest API is more useful for troubleshooting the interaction of the various pipelines that get applied when ingesting into an index.

By default, the pipeline definitions that are currently in the system are used. However, you can supply substitute pipeline definitions in the body of the request. These will be used in place of the pipeline definitions that are already in the system. This can be used to replace existing pipeline definitions or to create new ones. The pipeline substitutions are used only within this request.

Path parameters

  • index string Required

    The index to simulate ingesting into. This value can be overridden by specifying an index on each document. If you specify this parameter in the request path, it is used for any documents that do not explicitly specify an index argument.

Query parameters

  • pipeline string

    The pipeline to use as the default pipeline. This value can be used to override the default pipeline of the index.

application/json

Body Required

  • docs array[object] Required

    Sample documents to test in the pipeline.

    Hide docs attributes Show docs attributes object
  • component_template_substitutions object

    A map of component template names to substitute component template definition objects.

    Hide component_template_substitutions attribute Show component_template_substitutions attribute object
  • index_template_substitutions object

    A map of index template names to substitute index template definition objects.

    Hide index_template_substitutions attribute Show index_template_substitutions attribute object
  • mapping_addition object
    Hide mapping_addition attributes Show mapping_addition attributes object
    • all_field object
      Hide all_field attributes Show all_field attributes object
    • date_detection boolean
    • dynamic string

      Values are strict, runtime, true, or false.

    • dynamic_date_formats array[string]
    • dynamic_templates array[object]
    • _field_names object
      Hide _field_names attribute Show _field_names attribute object
    • index_field object
      Hide index_field attribute Show index_field attribute object
    • _meta object
      Hide _meta attribute Show _meta attribute object
      • * object Additional properties
    • numeric_detection boolean
    • properties object
    • _routing object
      Hide _routing attribute Show _routing attribute object
    • _size object
      Hide _size attribute Show _size attribute object
    • _source object
      Hide _source attributes Show _source attributes object
    • runtime object
      Hide runtime attribute Show runtime attribute object
      • * object Additional properties
        Hide * attributes Show * attributes object
        • fields object

          For type composite

          Hide fields attribute Show fields attribute object
          • * object Additional properties
            Hide * attribute Show * attribute object
            • type string Required

              Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

        • fetch_fields array[object]

          For type lookup

          Hide fetch_fields attributes Show fetch_fields attributes object
          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • format string
        • format string

          A custom format for date type runtime fields.

        • input_field string

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • target_field string

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • target_index string
        • script object
          Hide script attributes Show script attributes object

          • source string | object

            One of:
            ScriptSource string SearchRequestBody object
          • id string
          • params object

            Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

            Hide params attribute Show params attribute object
            • * object Additional properties

          • lang string

            Any of:
            ScriptLanguage string ScriptLanguage string

            Values are painless, expression, mustache, or java.

          • options object
            Hide options attribute Show options attribute object
            • * string Additional properties
        • type string Required

          Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

    • enabled boolean
    • subobjects string

      Values are true or false.

    • _data_stream_timestamp object
      Hide _data_stream_timestamp attribute Show _data_stream_timestamp attribute object
  • pipeline_substitutions object

    Pipelines to test. If you don’t specify the pipeline request path parameter, this parameter is required. If you specify both this and the request path parameter, the API only uses the request path parameter.

    Hide pipeline_substitutions attribute Show pipeline_substitutions attribute object
    • * object Additional properties
      Hide * attributes Show * attributes object
      • description string

        Description of the ingest pipeline.

      • on_failure array[object]

        Processors to run immediately after a processor failure.

        Hide on_failure attributes Show on_failure attributes object
        • append object
          Hide append attributes Show append attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • value
          • allow_duplicates boolean

            If false, the processor does not append values already present in the field.

        • attachment object
          Hide attachment attributes Show attachment attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • indexed_chars number

            The number of chars being used for extraction to prevent huge fields. Use -1 for no limit.

          • indexed_chars_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Array of properties to select to be stored. Can be content, title, name, author, keywords, date, content_type, content_length, language.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • remove_binary boolean

            If true, the binary field will be removed from the document

          • resource_name string

            Field containing the name of the resource to decode. If specified, the processor passes this resource name to the underlying Tika library to enable Resource Name Based Detection.

        • bytes object
          Hide bytes attributes Show bytes attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • circle object
          Hide circle attributes Show circle attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • error_distance number Required

            The difference between the resulting inscribed distance from center to side and the circle’s radius (measured in meters for geo_shape, unit-less for shape).

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • shape_type string Required

            Values are geo_shape or shape.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • community_id object
          Hide community_id attributes Show community_id attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • source_ip string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • source_port string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • destination_ip string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • destination_port string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • iana_number string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • icmp_type string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • icmp_code string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • transport string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • seed number

            Seed for the community ID hash. Must be between 0 and 65535 (inclusive). The seed can prevent hash collisions between network domains, such as a staging and production network that use the same addressing scheme.

          • ignore_missing boolean

            If true and any required fields are missing, the processor quietly exits without modifying the document.

        • convert object
          Hide convert attributes Show convert attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • type string Required

            Values are integer, long, double, float, boolean, ip, string, or auto.

        • csv object
          Hide csv attributes Show csv attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • empty_value object

            Value used to fill empty fields. Empty fields are skipped if this is not provided. An empty field is one with no value (2 consecutive separators) or empty quotes ("").

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • quote string

            Quote used in CSV, has to be single character string.

          • separator string

            Separator used in CSV, has to be single character string.

          • target_fields string | array[string] Required
          • trim boolean

            Trim whitespaces in unquoted fields.

        • date object
          Hide date attributes Show date attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • formats array[string] Required

            An array of the expected date formats. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • locale string

            The locale to use when parsing the date, relevant when parsing month names or week days. Supports template snippets.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • timezone string

            The timezone to use when parsing the date. Supports template snippets.

          • output_format string

            The format to use when writing the date to target_field. Must be a valid java time pattern.

        • date_index_name object
          Hide date_index_name attributes Show date_index_name attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • date_formats array[string]

            An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • date_rounding string Required

            How to round the date when formatting the date into the index name. Valid values are: y (year), M (month), w (week), d (day), h (hour), m (minute) and s (second). Supports template snippets.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • index_name_format string

            The format to be used when printing the parsed date into the index name. A valid java time pattern is expected here. Supports template snippets.

          • index_name_prefix string

            A prefix of the index name to be prepended before the printed date. Supports template snippets.

          • locale string

            The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.

          • timezone string

            The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.

        • dissect object
          Hide dissect attributes Show dissect attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • append_separator string

            The character(s) that separate the appended fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to apply to the field.

        • dot_expander object
          Hide dot_expander attributes Show dot_expander attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • override boolean

            Controls the behavior when there is already an existing nested object that conflicts with the expanded field. When false, the processor will merge conflicts by combining the old and the new values into an array. When true, the value from the expanded field will overwrite the existing value.

          • path string

            The field that contains the field to expand. Only required if the field to expand is part another object field, because the field option can only understand leaf fields.

        • drop object
          Hide drop attributes Show drop attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • enrich object
          Hide enrich attributes Show enrich attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • max_matches number

            The maximum number of matched documents to include under the configured target field. The target_field will be turned into a json array if max_matches is higher than 1, otherwise target_field will become a json object. In order to avoid documents getting too large, the maximum allowed value is 128.

          • override boolean

            If processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • policy_name string Required

            The name of the enrich policy to use.

          • shape_relation string

            Values are intersects, disjoint, within, or contains.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • fail object
          Hide fail attributes Show fail attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • message string Required

            The error message thrown by the processor. Supports template snippets.

        • fingerprint object
          Hide fingerprint attributes Show fingerprint attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • fields string | array[string] Required
          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • salt string

            Salt value for the hash function.

          • method string

            Values are MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.

          • ignore_missing boolean

            If true, the processor ignores any missing fields. If all fields are missing, the processor silently exits without modifying the document.

        • foreach object
          Hide foreach attributes Show foreach attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true, the processor silently exits without changing the document if the field is null or missing.

          • processor object Required
        • ip_location object
          Hide ip_location attributes Show ip_location attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • database_file string

            The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found IP location data will be returned, even if the field contains an array.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the IP location lookup.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • download_database_on_pipeline_creation boolean

            If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • geo_grid object
          Hide geo_grid attributes Show geo_grid attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            The field to interpret as a geo-tile.= The field format is determined by the tile_type.

          • tile_type string Required

            Values are geotile, geohex, or geohash.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • parent_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • children_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • non_children_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • precision_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • target_format string

            Values are geojson or wkt.

        • geoip object
          Hide geoip attributes Show geoip attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • database_file string

            The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found geoip data will be returned, even if the field contains an array.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the geoip lookup.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • download_database_on_pipeline_creation boolean

            If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • grok object
          Hide grok attributes Show grok attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • ecs_compatibility string

            Must be disabled or v1. If v1, the processor uses patterns with Elastic Common Schema (ECS) field names.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern_definitions object

            A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition.

          • patterns array[string] Required

            An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches.

          • trace_match boolean

            When true, _ingest._grok_match_index will be inserted into your matched document’s metadata with the index into the pattern found in patterns that matched.

        • gsub object
          Hide gsub attributes Show gsub attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to be replaced.

          • replacement string Required

            The string to replace the matching patterns with.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • html_strip object
          Hide html_strip attributes Show html_strip attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document,

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • inference object
          Hide inference attributes Show inference attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • model_id string Required
          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field_map object

            Maps the document field names to the known field names of the model. This mapping takes precedence over any default mappings provided in the model configuration.

          • inference_config object
          • input_output
          • ignore_missing boolean

            If true and any of the input fields defined in input_ouput are missing then those missing fields are quietly ignored, otherwise a missing field causes a failure. Only applies when using input_output configurations to explicitly list the input fields.

        • join object
          Hide join attributes Show join attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • separator string Required

            The separator character.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • json object
          Hide json attributes Show json attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • add_to_root boolean

            Flag that forces the parsed JSON to be added at the top level of the document. target_field must not be set when this option is chosen.

          • add_to_root_conflict_strategy string

            Values are replace or merge.

          • allow_duplicate_keys boolean

            When set to true, the JSON parser will not fail if the JSON contains duplicate keys. Instead, the last encountered value for any duplicate key wins.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • kv object
          Hide kv attributes Show kv attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • exclude_keys array[string]

            List of keys to exclude from document.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field_split string Required

            Regex pattern to use for splitting key-value pairs.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • include_keys array[string]

            List of keys to filter and insert into document. Defaults to including all keys.

          • prefix string

            Prefix to be added to extracted keys.

          • strip_brackets boolean

            If true. strip brackets (), <>, [] as well as quotes ' and " from extracted values.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • trim_key string

            String of characters to trim from extracted keys.

          • trim_value string

            String of characters to trim from extracted values.

          • value_split string Required

            Regex pattern to use for splitting the key from the value within a key-value pair.

        • lowercase object
          Hide lowercase attributes Show lowercase attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • network_direction object
          Hide network_direction attributes Show network_direction attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • source_ip string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • destination_ip string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • internal_networks array[string]

            List of internal networks. Supports IPv4 and IPv6 addresses and ranges in CIDR notation. Also supports the named ranges listed below. These may be constructed with template snippets. Must specify only one of internal_networks or internal_networks_field.

          • internal_networks_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and any required fields are missing, the processor quietly exits without modifying the document.

        • pipeline object
          Hide pipeline attributes Show pipeline attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • name string Required
          • ignore_missing_pipeline boolean

            Whether to ignore missing pipelines instead of failing.

        • redact object
          Hide redact attributes Show redact attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • patterns array[string] Required

            A list of grok expressions to match and redact named captures with

          • pattern_definitions object
          • prefix string

            Start a redacted section with this token

          • suffix string

            End a redacted section with this token

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • skip_if_unlicensed boolean

            If true and the current license does not support running redact processors, then the processor quietly exits without modifying the document

          • trace_redact boolean

            If true then ingest metadata _ingest._redact._is_redacted is set to true if the document has been redacted

        • registered_domain object
          Hide registered_domain attributes Show registered_domain attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and any required fields are missing, the processor quietly exits without modifying the document.

        • remove object
          Hide remove attributes Show remove attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string | array[string] Required
          • keep string | array[string]
          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • rename object
          Hide rename attributes Show rename attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • reroute object
          Hide reroute attributes Show reroute attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • destination string

            A static value for the target. Can’t be set when the dataset or namespace option is set.

          • dataset
          • namespace
        • script object
          Hide script attributes Show script attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • id string
          • lang
          • params object

            Object containing parameters for the script.

          • source
        • set object
          Hide set attributes Show set attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • copy_from string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_empty_value boolean

            If true and value is a template snippet that evaluates to null or the empty string, the processor quietly exits without modifying the document.

          • media_type string

            The media type for encoding value. Applies only when value is a template snippet. Must be one of application/json, text/plain, or application/x-www-form-urlencoded.

          • override boolean

            If true processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • value object

            The value to be set for the field. Supports template snippets. May specify only one of value or copy_from.

        • set_security_user object
          Hide set_security_user attributes Show set_security_user attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what user related properties are added to the field.

        • sort object
          Hide sort attributes Show sort attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • order string

            Values are asc or desc.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • split object
          Hide split attributes Show split attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • preserve_trailing boolean

            Preserves empty trailing fields, if any.

          • separator string Required

            A regex which matches the separator, for example, , or \s+.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • terminate object
          Hide terminate attributes Show terminate attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • trim object
          Hide trim attributes Show trim attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • uppercase object
          Hide uppercase attributes Show uppercase attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • urldecode object
          Hide urldecode attributes Show urldecode attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • uri_parts object
          Hide uri_parts attributes Show uri_parts attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • keep_original boolean

            If true, the processor copies the unparsed URI to <target_field>.original.

          • remove_if_successful boolean

            If true, the processor removes the field after parsing the URI string. If parsing fails, the processor does not remove the field.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • user_agent object
          Hide user_agent attributes Show user_agent attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • regex_file string

            The name of the file in the config/ingest-user-agent directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-user-agent will use the regexes.yaml from uap-core it ships with.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what properties are added to target_field.

            Values are name, os, device, original, or version.

          • extract_device_type boolean Beta

            Extracts device type from the user agent string on a best-effort basis.

      • processors array[object]

        Processors used to perform transformations on documents before indexing. Processors run sequentially in the order specified.

        Hide processors attributes Show processors attributes object
        • append object
          Hide append attributes Show append attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • value
          • allow_duplicates boolean

            If false, the processor does not append values already present in the field.

        • attachment object
          Hide attachment attributes Show attachment attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • indexed_chars number

            The number of chars being used for extraction to prevent huge fields. Use -1 for no limit.

          • indexed_chars_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Array of properties to select to be stored. Can be content, title, name, author, keywords, date, content_type, content_length, language.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • remove_binary boolean

            If true, the binary field will be removed from the document

          • resource_name string

            Field containing the name of the resource to decode. If specified, the processor passes this resource name to the underlying Tika library to enable Resource Name Based Detection.

        • bytes object
          Hide bytes attributes Show bytes attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • circle object
          Hide circle attributes Show circle attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • error_distance number Required

            The difference between the resulting inscribed distance from center to side and the circle’s radius (measured in meters for geo_shape, unit-less for shape).

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • shape_type string Required

            Values are geo_shape or shape.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • community_id object
          Hide community_id attributes Show community_id attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • source_ip string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • source_port string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • destination_ip string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • destination_port string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • iana_number string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • icmp_type string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • icmp_code string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • transport string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • seed number

            Seed for the community ID hash. Must be between 0 and 65535 (inclusive). The seed can prevent hash collisions between network domains, such as a staging and production network that use the same addressing scheme.

          • ignore_missing boolean

            If true and any required fields are missing, the processor quietly exits without modifying the document.

        • convert object
          Hide convert attributes Show convert attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • type string Required

            Values are integer, long, double, float, boolean, ip, string, or auto.

        • csv object
          Hide csv attributes Show csv attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • empty_value object

            Value used to fill empty fields. Empty fields are skipped if this is not provided. An empty field is one with no value (2 consecutive separators) or empty quotes ("").

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • quote string

            Quote used in CSV, has to be single character string.

          • separator string

            Separator used in CSV, has to be single character string.

          • target_fields string | array[string] Required
          • trim boolean

            Trim whitespaces in unquoted fields.

        • date object
          Hide date attributes Show date attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • formats array[string] Required

            An array of the expected date formats. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • locale string

            The locale to use when parsing the date, relevant when parsing month names or week days. Supports template snippets.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • timezone string

            The timezone to use when parsing the date. Supports template snippets.

          • output_format string

            The format to use when writing the date to target_field. Must be a valid java time pattern.

        • date_index_name object
          Hide date_index_name attributes Show date_index_name attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • date_formats array[string]

            An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • date_rounding string Required

            How to round the date when formatting the date into the index name. Valid values are: y (year), M (month), w (week), d (day), h (hour), m (minute) and s (second). Supports template snippets.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • index_name_format string

            The format to be used when printing the parsed date into the index name. A valid java time pattern is expected here. Supports template snippets.

          • index_name_prefix string

            A prefix of the index name to be prepended before the printed date. Supports template snippets.

          • locale string

            The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.

          • timezone string

            The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.

        • dissect object
          Hide dissect attributes Show dissect attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • append_separator string

            The character(s) that separate the appended fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to apply to the field.

        • dot_expander object
          Hide dot_expander attributes Show dot_expander attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • override boolean

            Controls the behavior when there is already an existing nested object that conflicts with the expanded field. When false, the processor will merge conflicts by combining the old and the new values into an array. When true, the value from the expanded field will overwrite the existing value.

          • path string

            The field that contains the field to expand. Only required if the field to expand is part another object field, because the field option can only understand leaf fields.

        • drop object
          Hide drop attributes Show drop attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • enrich object
          Hide enrich attributes Show enrich attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • max_matches number

            The maximum number of matched documents to include under the configured target field. The target_field will be turned into a json array if max_matches is higher than 1, otherwise target_field will become a json object. In order to avoid documents getting too large, the maximum allowed value is 128.

          • override boolean

            If processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • policy_name string Required

            The name of the enrich policy to use.

          • shape_relation string

            Values are intersects, disjoint, within, or contains.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • fail object
          Hide fail attributes Show fail attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • message string Required

            The error message thrown by the processor. Supports template snippets.

        • fingerprint object
          Hide fingerprint attributes Show fingerprint attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • fields string | array[string] Required
          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • salt string

            Salt value for the hash function.

          • method string

            Values are MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.

          • ignore_missing boolean

            If true, the processor ignores any missing fields. If all fields are missing, the processor silently exits without modifying the document.

        • foreach object
          Hide foreach attributes Show foreach attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true, the processor silently exits without changing the document if the field is null or missing.

          • processor object Required
        • ip_location object
          Hide ip_location attributes Show ip_location attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • database_file string

            The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found IP location data will be returned, even if the field contains an array.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the IP location lookup.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • download_database_on_pipeline_creation boolean

            If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • geo_grid object
          Hide geo_grid attributes Show geo_grid attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            The field to interpret as a geo-tile.= The field format is determined by the tile_type.

          • tile_type string Required

            Values are geotile, geohex, or geohash.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • parent_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • children_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • non_children_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • precision_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • target_format string

            Values are geojson or wkt.

        • geoip object
          Hide geoip attributes Show geoip attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • database_file string

            The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found geoip data will be returned, even if the field contains an array.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the geoip lookup.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • download_database_on_pipeline_creation boolean

            If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • grok object
          Hide grok attributes Show grok attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • ecs_compatibility string

            Must be disabled or v1. If v1, the processor uses patterns with Elastic Common Schema (ECS) field names.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern_definitions object

            A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition.

          • patterns array[string] Required

            An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches.

          • trace_match boolean

            When true, _ingest._grok_match_index will be inserted into your matched document’s metadata with the index into the pattern found in patterns that matched.

        • gsub object
          Hide gsub attributes Show gsub attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to be replaced.

          • replacement string Required

            The string to replace the matching patterns with.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • html_strip object
          Hide html_strip attributes Show html_strip attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document,

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • inference object
          Hide inference attributes Show inference attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • model_id string Required
          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field_map object

            Maps the document field names to the known field names of the model. This mapping takes precedence over any default mappings provided in the model configuration.

          • inference_config object
          • input_output
          • ignore_missing boolean

            If true and any of the input fields defined in input_ouput are missing then those missing fields are quietly ignored, otherwise a missing field causes a failure. Only applies when using input_output configurations to explicitly list the input fields.

        • join object
          Hide join attributes Show join attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • separator string Required

            The separator character.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • json object
          Hide json attributes Show json attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • add_to_root boolean

            Flag that forces the parsed JSON to be added at the top level of the document. target_field must not be set when this option is chosen.

          • add_to_root_conflict_strategy string

            Values are replace or merge.

          • allow_duplicate_keys boolean

            When set to true, the JSON parser will not fail if the JSON contains duplicate keys. Instead, the last encountered value for any duplicate key wins.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • kv object
          Hide kv attributes Show kv attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • exclude_keys array[string]

            List of keys to exclude from document.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field_split string Required

            Regex pattern to use for splitting key-value pairs.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • include_keys array[string]

            List of keys to filter and insert into document. Defaults to including all keys.

          • prefix string

            Prefix to be added to extracted keys.

          • strip_brackets boolean

            If true. strip brackets (), <>, [] as well as quotes ' and " from extracted values.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • trim_key string

            String of characters to trim from extracted keys.

          • trim_value string

            String of characters to trim from extracted values.

          • value_split string Required

            Regex pattern to use for splitting the key from the value within a key-value pair.

        • lowercase object
          Hide lowercase attributes Show lowercase attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • network_direction object
          Hide network_direction attributes Show network_direction attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • source_ip string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • destination_ip string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • internal_networks array[string]

            List of internal networks. Supports IPv4 and IPv6 addresses and ranges in CIDR notation. Also supports the named ranges listed below. These may be constructed with template snippets. Must specify only one of internal_networks or internal_networks_field.

          • internal_networks_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and any required fields are missing, the processor quietly exits without modifying the document.

        • pipeline object
          Hide pipeline attributes Show pipeline attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • name string Required
          • ignore_missing_pipeline boolean

            Whether to ignore missing pipelines instead of failing.

        • redact object
          Hide redact attributes Show redact attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • patterns array[string] Required

            A list of grok expressions to match and redact named captures with

          • pattern_definitions object
          • prefix string

            Start a redacted section with this token

          • suffix string

            End a redacted section with this token

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • skip_if_unlicensed boolean

            If true and the current license does not support running redact processors, then the processor quietly exits without modifying the document

          • trace_redact boolean

            If true then ingest metadata _ingest._redact._is_redacted is set to true if the document has been redacted

        • registered_domain object
          Hide registered_domain attributes Show registered_domain attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and any required fields are missing, the processor quietly exits without modifying the document.

        • remove object
          Hide remove attributes Show remove attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string | array[string] Required
          • keep string | array[string]
          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • rename object
          Hide rename attributes Show rename attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • reroute object
          Hide reroute attributes Show reroute attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • destination string

            A static value for the target. Can’t be set when the dataset or namespace option is set.

          • dataset
          • namespace
        • script object
          Hide script attributes Show script attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • id string
          • lang
          • params object

            Object containing parameters for the script.

          • source
        • set object
          Hide set attributes Show set attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • copy_from string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_empty_value boolean

            If true and value is a template snippet that evaluates to null or the empty string, the processor quietly exits without modifying the document.

          • media_type string

            The media type for encoding value. Applies only when value is a template snippet. Must be one of application/json, text/plain, or application/x-www-form-urlencoded.

          • override boolean

            If true processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • value object

            The value to be set for the field. Supports template snippets. May specify only one of value or copy_from.

        • set_security_user object
          Hide set_security_user attributes Show set_security_user attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what user related properties are added to the field.

        • sort object
          Hide sort attributes Show sort attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • order string

            Values are asc or desc.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • split object
          Hide split attributes Show split attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • preserve_trailing boolean

            Preserves empty trailing fields, if any.

          • separator string Required

            A regex which matches the separator, for example, , or \s+.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • terminate object
          Hide terminate attributes Show terminate attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • trim object
          Hide trim attributes Show trim attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • uppercase object
          Hide uppercase attributes Show uppercase attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • urldecode object
          Hide urldecode attributes Show urldecode attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • uri_parts object
          Hide uri_parts attributes Show uri_parts attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • keep_original boolean

            If true, the processor copies the unparsed URI to <target_field>.original.

          • remove_if_successful boolean

            If true, the processor removes the field after parsing the URI string. If parsing fails, the processor does not remove the field.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • user_agent object
          Hide user_agent attributes Show user_agent attributes object
          • description string

            Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • ignore_failure boolean

            Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • ignore_missing boolean

            If true and field does not exist, the processor quietly exits without modifying the document.

          • regex_file string

            The name of the file in the config/ingest-user-agent directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-user-agent will use the regexes.yaml from uap-core it ships with.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what properties are added to target_field.

            Values are name, os, device, original, or version.

          • extract_device_type boolean Beta

            Extracts device type from the user agent string on a best-effort basis.

      • version number
      • deprecated boolean

        Marks this ingest pipeline as deprecated. When a deprecated ingest pipeline is referenced as the default or final pipeline when creating or updating a non-deprecated index template, Elasticsearch will emit a deprecation warning.

      • _meta object
        Hide _meta attribute Show _meta attribute object
        • * object Additional properties

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • docs array[object] Required
      Hide docs attribute Show docs attribute object
      • doc object
        Hide doc attributes Show doc attributes object
        • _id string Required
        • _index string Required
        • _source object Required

          JSON body for the document.

          Hide _source attribute Show _source attribute object
          • * object Additional properties

        • _version number | string Required

          Some APIs will return values such as numbers also as a string (notably epoch timestamps). This behavior is used to capture this behavior while keeping the semantics of the field type.

          Depending on the target language, code generators can keep the union or remove it and leniently parse strings to the target type.

          One of:
          VersionNumber number StringifiedVersionNumber string
        • executed_pipelines array[string] Required

          A list of the names of the pipelines executed on this document.

        • ignored_fields array[object]

          A list of the fields that would be ignored at the indexing step. For example, a field whose value is larger than the allowed limit would make it through all of the pipelines, but would not be indexed into Elasticsearch.

          Hide ignored_fields attribute Show ignored_fields attribute object
          • * string Additional properties
        • error object
          Hide error attributes Show error attributes object
GET /_ingest/{index}/_simulate 
curl \
 --request GET 'http://api.example.com/_ingest/{index}/_simulate' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '"{\n  \"docs\": [\n    {\n      \"_id\": 123,\n      \"_index\": \"my-index\",\n      \"_source\": {\n        \"foo\": \"bar\"\n      }\n    },\n    {\n      \"_id\": 456,\n      \"_index\": \"my-index\",\n      \"_source\": {\n        \"foo\": \"rab\"\n      }\n    }\n  ]\n}"'
Request examples
In this example the index `my-index` has a default pipeline called `my-pipeline` and a final pipeline called `my-final-pipeline`. Since both documents are being ingested into `my-index`, both pipelines are run using the pipeline definitions that are already in the system. 
{
  "docs": [
    {
      "_id": 123,
      "_index": "my-index",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_id": 456,
      "_index": "my-index",
      "_source": {
        "foo": "rab"
      }
    }
  ]
}
In this example the index `my-index` has a default pipeline called `my-pipeline` and a final pipeline called `my-final-pipeline`. But a substitute definition of `my-pipeline` is provided in `pipeline_substitutions`. The substitute `my-pipeline` will be used in place of the `my-pipeline` that is in the system, and then the `my-final-pipeline` that is already defined in the system will run. 
{
  "docs": [
    {
      "_index": "my-index",
      "_id": 123,
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_index": "my-index",
      "_id": 456,
      "_source": {
        "foo": "rab"
      }
    }
  ],
  "pipeline_substitutions": {
    "my-pipeline": {
      "processors": [
        {
          "uppercase": {
            "field": "foo"
          }
        }
      ]
    }
  }
}
In this example, imagine that the index `my-index` has a strict mapping with only the `foo` keyword field defined. Say that field mapping came from a component template named `my-mappings-template`. You want to test adding a new field, `bar`. So a substitute definition of `my-mappings-template` is provided in `component_template_substitutions`. The substitute `my-mappings-template` will be used in place of the existing mapping for `my-index` and in place of the `my-mappings-template` that is in the system. 
{
  "docs": [
    {
      "_index": "my-index",
      "_id": "123",
      "_source": {
        "foo": "foo"
      }
    },
    {
      "_index": "my-index",
      "_id": "456",
      "_source": {
        "bar": "rab"
      }
    }
  ],
  "component_template_substitutions": {
    "my-mappings_template": {
      "template": {
        "mappings": {
          "dynamic": "strict",
          "properties": {
            "foo": {
              "type": "keyword"
            },
            "bar": {
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}
The pipeline, component template, and index template substitutions replace the existing pipeline details for the duration of this request. 
{
  "docs": [
    {
      "_id": "id",
      "_index": "my-index",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_id": "id",
      "_index": "my-index",
      "_source": {
        "foo": "rab"
      }
    }
  ],
  "pipeline_substitutions": {
    "my-pipeline": {
      "processors": [
        {
          "set": {
            "field": "field3",
            "value": "value3"
          }
        }
      ]
    }
  },
  "component_template_substitutions": {
    "my-component-template": {
      "template": {
        "mappings": {
          "dynamic": true,
          "properties": {
            "field3": {
              "type": "keyword"
            }
          }
        },
        "settings": {
          "index": {
            "default_pipeline": "my-pipeline"
          }
        }
      }
    }
  },
  "index_template_substitutions": {
    "my-index-template": {
      "index_patterns": [
        "my-index-*"
      ],
      "composed_of": [
        "component_template_1",
        "component_template_2"
      ]
    }
  },
  "mapping_addition": {
    "dynamic": "strict",
    "properties": {
      "foo": {
        "type": "keyword"
      }
    }
  }
}
Response examples (200)
A successful response when the simulation uses pipeline definitions that are already in the system. 
{
  "docs": [
    {
      "doc": null,
      "_id": 123,
      "_index": "my-index",
      "_version": -3,
      "_source": {
        "field1": "value1",
        "field2": "value2",
        "foo": "bar"
      },
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    },
    {
      "doc": null,
      "_id": 456,
      "_index": "my-index",
      "_version": "-3,",
      "_source": {
        "field1": "value1",
        "field2": "value2",
        "foo": "rab"
      },
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    }
  ]
}
A successful response when the simulation uses pipeline substitutions. 
{
  "docs": [
    {
      "doc": null,
      "_id": 123,
      "_index": "my-index",
      "_version": -3,
      "_source": {
        "field2": "value2",
        "foo": "BAR"
      },
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    },
    {
      "doc": null,
      "_id": 456,
      "_index": "my-index",
      "_version": -3,
      "_source": {
        "field2": "value2",
        "foo": "RAB"
      },
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    }
  ]
}
A successful response when the simulation uses pipeline substitutions. 
{
  "docs": [
    {
      "doc": {
        "_id": "123",
        "_index": "my-index",
        "_version": -3,
        "_source": {
          "foo": "foo"
        },
        "executed_pipelines": []
      }
    },
    {
      "doc": {
        "_id": "456",
        "_index": "my-index",
        "_version": -3,
        "_source": {
          "bar": "rab"
        },
      "executed_pipelines": []
      }
    }
  ]
}