Get snapshot repository information Added in 2.1.0

GET /_cat/repositories

Get a list of snapshot repositories for a cluster. IMPORTANT: cat APIs are only intended for human consumption using the command line or Kibana console. They are not intended for use by applications. For application consumption, use the get snapshot repository API.

Query parameters

  • h string | array[string]

    List of columns to appear in the response. Supports simple wildcards.

  • s string | array[string]

    List of columns that determine how the table should be sorted. Sorting defaults to ascending and can be changed by setting :asc or :desc as a suffix to the column name.

  • local boolean

    If true, the request computes the list of selected nodes from the local cluster state. If false the list of selected nodes are computed from the cluster state of the master node. In both cases the coordinating node will send requests for further information to each selected node.

  • Period to wait for a connection to the master node.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • id string

      The unique repository identifier.

    • type string

      The repository type.

GET /_cat/repositories
curl \
 --request GET 'http://api.example.com/_cat/repositories' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response from `GET /_cat/repositories?v=true&format=json`.
[
  {
    "id": "repo1",
    "type": "fs"
  },
  {
    "id": "repo2",
    "type": "s3"
  }
]





















































































































Get cluster statistics Added in 1.3.0

GET /_cluster/stats/nodes/{node_id}

Get basic index metrics (shard numbers, store size, memory usage) and information about the current nodes that form the cluster (number, roles, os, jvm versions, memory usage, cpu and installed plugins).

Path parameters

  • node_id string | array[string] Required

    Comma-separated list of node filters used to limit returned information. Defaults to all nodes in the cluster.

Query parameters

  • Include remote cluster data into the response

  • timeout string

    Period to wait for each node to respond. If a node does not respond before its timeout expires, the response does not include its stats. However, timed out nodes are included in the response’s _nodes.failed property. Defaults to no timeout.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • _nodes object
      Hide _nodes attributes Show _nodes attributes object
      • failures array[object]
        Hide failures attributes Show failures attributes object
      • total number Required

        Total number of nodes selected by the request.

      • successful number Required

        Number of nodes that responded successfully to the request.

      • failed number Required

        Number of nodes that rejected the request or failed to respond. If this value is not 0, a reason for the rejection or failure is included in the response.

    • cluster_name string Required
    • cluster_uuid string Required
    • indices object Required
      Hide indices attributes Show indices attributes object
      • analysis object Required
        Hide analysis attributes Show analysis attributes object
        • analyzer_types array[object] Required

          Contains statistics about analyzer types used in selected nodes.

          Hide analyzer_types attributes Show analyzer_types attributes object
          • name string Required
          • count number Required

            The number of occurrences of the field type in selected nodes.

          • index_count number Required

            The number of indices containing the field type in selected nodes.

          • For dense_vector field types, number of indexed vector types in selected nodes.

          • For dense_vector field types, the maximum dimension of all indexed vector types in selected nodes.

          • For dense_vector field types, the minimum dimension of all indexed vector types in selected nodes.

          • The number of fields that declare a script.

        • built_in_analyzers array[object] Required

          Contains statistics about built-in analyzers used in selected nodes.

          Hide built_in_analyzers attributes Show built_in_analyzers attributes object
          • name string Required
          • count number Required

            The number of occurrences of the field type in selected nodes.

          • index_count number Required

            The number of indices containing the field type in selected nodes.

          • For dense_vector field types, number of indexed vector types in selected nodes.

          • For dense_vector field types, the maximum dimension of all indexed vector types in selected nodes.

          • For dense_vector field types, the minimum dimension of all indexed vector types in selected nodes.

          • The number of fields that declare a script.

        • built_in_char_filters array[object] Required

          Contains statistics about built-in character filters used in selected nodes.

          Hide built_in_char_filters attributes Show built_in_char_filters attributes object
          • name string Required
          • count number Required

            The number of occurrences of the field type in selected nodes.

          • index_count number Required

            The number of indices containing the field type in selected nodes.

          • For dense_vector field types, number of indexed vector types in selected nodes.

          • For dense_vector field types, the maximum dimension of all indexed vector types in selected nodes.

          • For dense_vector field types, the minimum dimension of all indexed vector types in selected nodes.

          • The number of fields that declare a script.

        • built_in_filters array[object] Required

          Contains statistics about built-in token filters used in selected nodes.

          Hide built_in_filters attributes Show built_in_filters attributes object
          • name string Required
          • count number Required

            The number of occurrences of the field type in selected nodes.

          • index_count number Required

            The number of indices containing the field type in selected nodes.

          • For dense_vector field types, number of indexed vector types in selected nodes.

          • For dense_vector field types, the maximum dimension of all indexed vector types in selected nodes.

          • For dense_vector field types, the minimum dimension of all indexed vector types in selected nodes.

          • The number of fields that declare a script.

        • built_in_tokenizers array[object] Required

          Contains statistics about built-in tokenizers used in selected nodes.

          Hide built_in_tokenizers attributes Show built_in_tokenizers attributes object
          • name string Required
          • count number Required

            The number of occurrences of the field type in selected nodes.

          • index_count number Required

            The number of indices containing the field type in selected nodes.

          • For dense_vector field types, number of indexed vector types in selected nodes.

          • For dense_vector field types, the maximum dimension of all indexed vector types in selected nodes.

          • For dense_vector field types, the minimum dimension of all indexed vector types in selected nodes.

          • The number of fields that declare a script.

        • char_filter_types array[object] Required

          Contains statistics about character filter types used in selected nodes.

          Hide char_filter_types attributes Show char_filter_types attributes object
          • name string Required
          • count number Required

            The number of occurrences of the field type in selected nodes.

          • index_count number Required

            The number of indices containing the field type in selected nodes.

          • For dense_vector field types, number of indexed vector types in selected nodes.

          • For dense_vector field types, the maximum dimension of all indexed vector types in selected nodes.

          • For dense_vector field types, the minimum dimension of all indexed vector types in selected nodes.

          • The number of fields that declare a script.

        • filter_types array[object] Required

          Contains statistics about token filter types used in selected nodes.

          Hide filter_types attributes Show filter_types attributes object
          • name string Required
          • count number Required

            The number of occurrences of the field type in selected nodes.

          • index_count number Required

            The number of indices containing the field type in selected nodes.

          • For dense_vector field types, number of indexed vector types in selected nodes.

          • For dense_vector field types, the maximum dimension of all indexed vector types in selected nodes.

          • For dense_vector field types, the minimum dimension of all indexed vector types in selected nodes.

          • The number of fields that declare a script.

        • tokenizer_types array[object] Required

          Contains statistics about tokenizer types used in selected nodes.

          Hide tokenizer_types attributes Show tokenizer_types attributes object
          • name string Required
          • count number Required

            The number of occurrences of the field type in selected nodes.

          • index_count number Required

            The number of indices containing the field type in selected nodes.

          • For dense_vector field types, number of indexed vector types in selected nodes.

          • For dense_vector field types, the maximum dimension of all indexed vector types in selected nodes.

          • For dense_vector field types, the minimum dimension of all indexed vector types in selected nodes.

          • The number of fields that declare a script.

      • completion object Required
        Hide completion attributes Show completion attributes object
      • count number Required

        Total number of indices with shards assigned to selected nodes.

      • docs object Required
        Hide docs attributes Show docs attributes object
        • count number Required

          Total number of non-deleted documents across all primary shards assigned to selected nodes. This number is based on documents in Lucene segments and may include documents from nested fields.

        • deleted number

          Total number of deleted documents across all primary shards assigned to selected nodes. This number is based on documents in Lucene segments. Elasticsearch reclaims the disk space of deleted Lucene documents when a segment is merged.

      • fielddata object Required
        Hide fielddata attributes Show fielddata attributes object
      • query_cache object Required
        Hide query_cache attributes Show query_cache attributes object
        • cache_count number Required

          Total number of entries added to the query cache across all shards assigned to selected nodes. This number includes current and evicted entries.

        • cache_size number Required

          Total number of entries currently in the query cache across all shards assigned to selected nodes.

        • evictions number Required

          Total number of query cache evictions across all shards assigned to selected nodes.

        • hit_count number Required

          Total count of query cache hits across all shards assigned to selected nodes.

        • memory_size_in_bytes number Required

          Total amount, in bytes, of memory used for the query cache across all shards assigned to selected nodes.

        • miss_count number Required

          Total count of query cache misses across all shards assigned to selected nodes.

        • total_count number Required

          Total count of hits and misses in the query cache across all shards assigned to selected nodes.

      • segments object Required
        Hide segments attributes Show segments attributes object
      • shards object Required
        Hide shards attributes Show shards attributes object
        • index object
          Hide index attributes Show index attributes object
          • primaries object Required
            Hide primaries attributes Show primaries attributes object
            • avg number Required

              Mean number of shards in an index, counting only shards assigned to selected nodes.

            • max number Required

              Maximum number of shards in an index, counting only shards assigned to selected nodes.

            • min number Required

              Minimum number of shards in an index, counting only shards assigned to selected nodes.

          • replication object Required
            Hide replication attributes Show replication attributes object
            • avg number Required

              Mean number of shards in an index, counting only shards assigned to selected nodes.

            • max number Required

              Maximum number of shards in an index, counting only shards assigned to selected nodes.

            • min number Required

              Minimum number of shards in an index, counting only shards assigned to selected nodes.

          • shards object Required
            Hide shards attributes Show shards attributes object
            • avg number Required

              Mean number of shards in an index, counting only shards assigned to selected nodes.

            • max number Required

              Maximum number of shards in an index, counting only shards assigned to selected nodes.

            • min number Required

              Minimum number of shards in an index, counting only shards assigned to selected nodes.

        • Number of primary shards assigned to selected nodes.

        • Ratio of replica shards to primary shards across all selected nodes.

        • total number

          Total number of shards assigned to selected nodes.

      • store object Required
        Hide store attributes Show store attributes object
      • mappings object Required
        Hide mappings attributes Show mappings attributes object
        • field_types array[object] Required

          Contains statistics about field data types used in selected nodes.

          Hide field_types attributes Show field_types attributes object
          • name string Required
          • count number Required

            The number of occurrences of the field type in selected nodes.

          • index_count number Required

            The number of indices containing the field type in selected nodes.

          • For dense_vector field types, number of indexed vector types in selected nodes.

          • For dense_vector field types, the maximum dimension of all indexed vector types in selected nodes.

          • For dense_vector field types, the minimum dimension of all indexed vector types in selected nodes.

          • The number of fields that declare a script.

        • runtime_field_types array[object]

          Contains statistics about runtime field data types used in selected nodes.

          Hide runtime_field_types attributes Show runtime_field_types attributes object
          • chars_max number Required

            Maximum number of characters for a single runtime field script.

          • chars_total number Required

            Total number of characters for the scripts that define the current runtime field data type.

          • count number Required

            Number of runtime fields mapped to the field data type in selected nodes.

          • doc_max number Required

            Maximum number of accesses to doc_values for a single runtime field script

          • doc_total number Required

            Total number of accesses to doc_values for the scripts that define the current runtime field data type.

          • index_count number Required

            Number of indices containing a mapping of the runtime field data type in selected nodes.

          • lang array[string] Required

            Script languages used for the runtime fields scripts.

          • lines_max number Required

            Maximum number of lines for a single runtime field script.

          • lines_total number Required

            Total number of lines for the scripts that define the current runtime field data type.

          • name string Required
          • scriptless_count number Required

            Number of runtime fields that don’t declare a script.

          • shadowed_count number Required

            Number of runtime fields that shadow an indexed field.

          • source_max number Required

            Maximum number of accesses to _source for a single runtime field script.

          • source_total number Required

            Total number of accesses to _source for the scripts that define the current runtime field data type.

        • Total number of fields in all non-system indices.

        • Total number of fields in all non-system indices, accounting for mapping deduplication.

        • Total size of all mappings, in bytes, after deduplication and compression.

      • versions array[object]

        Contains statistics about analyzers and analyzer components used in selected nodes.

        Hide versions attributes Show versions attributes object
    • nodes object Required
      Hide nodes attributes Show nodes attributes object
      • count object Required
        Hide count attributes Show count attributes object
      • discovery_types object Required

        Contains statistics about the discovery types used by selected nodes.

        Hide discovery_types attribute Show discovery_types attribute object
        • * number Additional properties
      • fs object Required
        Hide fs attributes Show fs attributes object
        • available_in_bytes number Required

          Total number of bytes available to JVM in file stores across all selected nodes. Depending on operating system or process-level restrictions, this number may be less than nodes.fs.free_in_byes. This is the actual amount of free disk space the selected Elasticsearch nodes can use.

        • free_in_bytes number Required

          Total number of unallocated bytes in file stores across all selected nodes.

        • total_in_bytes number Required

          Total size, in bytes, of all file stores across all selected nodes.

      • indexing_pressure object Required
        Hide indexing_pressure attribute Show indexing_pressure attribute object
      • ingest object Required
        Hide ingest attributes Show ingest attributes object
        • number_of_pipelines number Required
        • processor_stats object Required
          Hide processor_stats attribute Show processor_stats attribute object
          • * object Additional properties
            Hide * attributes Show * attributes object
            • count number Required
            • current number Required
            • failed number Required
            • time string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • jvm object Required
        Hide jvm attributes Show jvm attributes object
        • Time unit for milliseconds

        • mem object Required
          Hide mem attributes Show mem attributes object
          • heap_max_in_bytes number Required

            Maximum amount of memory, in bytes, available for use by the heap across all selected nodes.

          • heap_used_in_bytes number Required

            Memory, in bytes, currently in use by the heap across all selected nodes.

        • threads number Required

          Number of active threads in use by JVM across all selected nodes.

        • versions array[object] Required

          Contains statistics about the JVM versions used by selected nodes.

          Hide versions attributes Show versions attributes object
      • network_types object Required
        Hide network_types attributes Show network_types attributes object
        • http_types object Required

          Contains statistics about the HTTP network types used by selected nodes.

          Hide http_types attribute Show http_types attribute object
          • * number Additional properties
        • transport_types object Required

          Contains statistics about the transport network types used by selected nodes.

          Hide transport_types attribute Show transport_types attribute object
          • * number Additional properties
      • os object Required
        Hide os attributes Show os attributes object
        • allocated_processors number Required

          Number of processors used to calculate thread pool size across all selected nodes. This number can be set with the processors setting of a node and defaults to the number of processors reported by the operating system. In both cases, this number will never be larger than 32.

        • architectures array[object]

          Contains statistics about processor architectures (for example, x86_64 or aarch64) used by selected nodes.

          Hide architectures attributes Show architectures attributes object
          • arch string Required

            Name of an architecture used by one or more selected nodes.

          • count number Required

            Number of selected nodes using the architecture.

        • available_processors number Required

          Number of processors available to JVM across all selected nodes.

        • mem object Required
          Hide mem attributes Show mem attributes object
          • Total amount, in bytes, of memory across all selected nodes, but using the value specified using the es.total_memory_bytes system property instead of measured total memory for those nodes where that system property was set.

          • free_in_bytes number Required

            Amount, in bytes, of free physical memory across all selected nodes.

          • free_percent number Required

            Percentage of free physical memory across all selected nodes.

          • total_in_bytes number Required

            Total amount, in bytes, of physical memory across all selected nodes.

          • used_in_bytes number Required

            Amount, in bytes, of physical memory in use across all selected nodes.

          • used_percent number Required

            Percentage of physical memory in use across all selected nodes.

        • names array[object] Required

          Contains statistics about operating systems used by selected nodes.

          Hide names attributes Show names attributes object
          • count number Required

            Number of selected nodes using the operating system.

          • name string Required
        • pretty_names array[object] Required

          Contains statistics about operating systems used by selected nodes.

          Hide pretty_names attributes Show pretty_names attributes object
          • count number Required

            Number of selected nodes using the operating system.

          • pretty_name string Required
      • packaging_types array[object] Required

        Contains statistics about Elasticsearch distributions installed on selected nodes.

        Hide packaging_types attributes Show packaging_types attributes object
        • count number Required

          Number of selected nodes using the distribution flavor and file type.

        • flavor string Required

          Type of Elasticsearch distribution. This is always default.

        • type string Required

          File type (such as tar or zip) used for the distribution package.

      • plugins array[object] Required

        Contains statistics about installed plugins and modules by selected nodes. If no plugins or modules are installed, this array is empty.

        Hide plugins attributes Show plugins attributes object
      • process object Required
        Hide process attributes Show process attributes object
        • cpu object Required
          Hide cpu attribute Show cpu attribute object
          • percent number Required

            Percentage of CPU used across all selected nodes. Returns -1 if not supported.

        • open_file_descriptors object Required
          Hide open_file_descriptors attributes Show open_file_descriptors attributes object
          • avg number Required

            Average number of concurrently open file descriptors. Returns -1 if not supported.

          • max number Required

            Maximum number of concurrently open file descriptors allowed across all selected nodes. Returns -1 if not supported.

          • min number Required

            Minimum number of concurrently open file descriptors across all selected nodes. Returns -1 if not supported.

      • versions array[string] Required

        Array of Elasticsearch versions used on selected nodes.

    • status string Required

      Values are green, GREEN, yellow, YELLOW, red, or RED.

    • timestamp number Required

      Unix timestamp, in milliseconds, for the last time the cluster statistics were refreshed.

    • ccs object Required
      Hide ccs attributes Show ccs attributes object
      • clusters object

        Contains remote cluster settings and metrics collected from them. The keys are cluster names, and the values are per-cluster data. Only present if include_remotes option is set to true.

        Hide clusters attribute Show clusters attribute object
        • * object Additional properties
          Hide * attributes Show * attributes object
          • cluster_uuid string Required

            The UUID of the remote cluster.

          • mode string Required

            The connection mode used to communicate with the remote cluster.

          • skip_unavailable boolean Required

            The skip_unavailable setting used for this remote cluster.

          • transport_compress string Required

            Transport compression setting used for this remote cluster.

          • status string Required

            Values are green, GREEN, yellow, YELLOW, red, or RED.

          • version array[string] Required

            The list of Elasticsearch versions used by the nodes on the remote cluster.

          • nodes_count number Required

            The total count of nodes in the remote cluster.

          • shards_count number Required

            The total number of shards in the remote cluster.

          • indices_count number Required

            The total number of indices in the remote cluster.

          • Total data set size, in bytes, of all shards assigned to selected nodes.

          • Total data set size of all shards assigned to selected nodes, as a human-readable string.

          • max_heap_in_bytes number Required

            Maximum amount of memory, in bytes, available for use by the heap across the nodes of the remote cluster.

          • max_heap string

            Maximum amount of memory available for use by the heap across the nodes of the remote cluster, as a human-readable string.

          • mem_total_in_bytes number Required

            Total amount, in bytes, of physical memory across the nodes of the remote cluster.

          • Total amount of physical memory across the nodes of the remote cluster, as a human-readable string.

      • _esql object
        Hide _esql attributes Show _esql attributes object
        • total number Required

          The total number of cross-cluster search requests that have been executed by the cluster.

        • success number Required

          The total number of cross-cluster search requests that have been successfully executed by the cluster.

        • skipped number Required

          The total number of cross-cluster search requests (successful or failed) that had at least one remote cluster skipped.

        • took object Required
          Hide took attributes Show took attributes object
          • Time unit for milliseconds

          • Time unit for milliseconds

          • Time unit for milliseconds

        • Hide took_mrt_true attributes Show took_mrt_true attributes object
          • Time unit for milliseconds

          • Time unit for milliseconds

          • Time unit for milliseconds

        • Hide took_mrt_false attributes Show took_mrt_false attributes object
          • Time unit for milliseconds

          • Time unit for milliseconds

          • Time unit for milliseconds

        • remotes_per_search_max number Required

          The maximum number of remote clusters that were queried in a single cross-cluster search request.

        • remotes_per_search_avg number Required

          The average number of remote clusters that were queried in a single cross-cluster search request.

        • failure_reasons object Required

          Statistics about the reasons for cross-cluster search request failures. The keys are the failure reason names and the values are the number of requests that failed for that reason.

          Hide failure_reasons attribute Show failure_reasons attribute object
          • * number Additional properties
        • features object Required

          The keys are the names of the search feature, and the values are the number of requests that used that feature. Single request can use more than one feature (e.g. both async and wildcard).

          Hide features attribute Show features attribute object
          • * number Additional properties
        • clients object Required

          Statistics about the clients that executed cross-cluster search requests. The keys are the names of the clients, and the values are the number of requests that were executed by that client. Only known clients (such as kibana or elasticsearch) are counted.

          Hide clients attribute Show clients attribute object
          • * number Additional properties
        • clusters object Required

          Statistics about the clusters that were queried in cross-cluster search requests. The keys are cluster names, and the values are per-cluster telemetry data. This also includes the local cluster itself, which uses the name (local).

          Hide clusters attribute Show clusters attribute object
          • * object Additional properties
            Hide * attributes Show * attributes object
            • total number Required

              The total number of successful (not skipped) cross-cluster search requests that were executed against this cluster. This may include requests where partial results were returned, but not requests in which the cluster has been skipped entirely.

            • skipped number Required

              The total number of cross-cluster search requests for which this cluster was skipped.

            • took object Required
GET /_cluster/stats/nodes/{node_id}
curl \
 --request GET 'http://api.example.com/_cluster/stats/nodes/{node_id}' \
 --header "Authorization: $API_KEY"






































































































Path parameters

  • connector_id string Required

    The unique identifier of the connector to be created or updated. ID is auto-generated if not provided.

application/json

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • result string Required

      Values are created, updated, deleted, not_found, or noop.

    • id string Required
PUT /_connector/{connector_id}
curl \
 --request PUT 'http://api.example.com/_connector/{connector_id}' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '"{\n  \"index_name\": \"search-google-drive\",\n  \"name\": \"My Connector\",\n  \"service_type\": \"google_drive\"\n}"'
Request examples
{
  "index_name": "search-google-drive",
  "name": "My Connector",
  "service_type": "google_drive"
}
{
  "index_name": "search-google-drive",
  "name": "My Connector",
  "description": "My Connector to sync data to Elastic index from Google Drive",
  "service_type": "google_drive",
  "language": "english"
}
Response examples (200)
{
  "result": "created",
  "id": "my-connector"
}















































































































































































































































Create a new document in the index Added in 5.0.0

PUT /{index}/_create/{id}

You can index a new JSON document with the /<target>/_doc/ or /<target>/_create/<_id> APIs Using _create guarantees that the document is indexed only if it does not already exist. It returns a 409 response when a document with a same ID already exists in the index. To update an existing document, you must use the /<target>/_doc/ API.

If the Elasticsearch security features are enabled, you must have the following index privileges for the target data stream, index, or index alias:

  • To add a document using the PUT /<target>/_create/<_id> or POST /<target>/_create/<_id> request formats, you must have the create_doc, create, index, or write index privilege.
  • To automatically create a data stream or index with this API request, you must have the auto_configure, create_index, or manage index privilege.

Automatic data stream creation requires a matching index template with data stream enabled.

Automatically create data streams and indices

If the request's target doesn't exist and matches an index template with a data_stream definition, the index operation automatically creates the data stream.

If the target doesn't exist and doesn't match a data stream template, the operation automatically creates the index and applies any matching index templates.

NOTE: Elasticsearch includes several built-in index templates. To avoid naming collisions with these templates, refer to index pattern documentation.

If no mapping exists, the index operation creates a dynamic mapping. By default, new fields and objects are automatically added to the mapping if needed.

Automatic index creation is controlled by the action.auto_create_index setting. If it is true, any index can be created automatically. You can modify this setting to explicitly allow or block automatic creation of indices that match specified patterns or set it to false to turn off automatic index creation entirely. Specify a comma-separated list of patterns you want to allow or prefix each pattern with + or - to indicate whether it should be allowed or blocked. When a list is specified, the default behaviour is to disallow.

NOTE: The action.auto_create_index setting affects the automatic creation of indices only. It does not affect the creation of data streams.

Routing

By default, shard placement — or routing — is controlled by using a hash of the document's ID value. For more explicit control, the value fed into the hash function used by the router can be directly specified on a per-operation basis using the routing parameter.

When setting up explicit mapping, you can also use the _routing field to direct the index operation to extract the routing value from the document itself. This does come at the (very minimal) cost of an additional document parsing pass. If the _routing mapping is defined and set to be required, the index operation will fail if no routing value is provided or extracted.

NOTE: Data streams do not support custom routing unless they were created with the allow_custom_routing setting enabled in the template.

Distributed

The index operation is directed to the primary shard based on its route and performed on the actual node containing this shard. After the primary shard completes the operation, if needed, the update is distributed to applicable replicas.

Active shards

To improve the resiliency of writes to the system, indexing operations can be configured to wait for a certain number of active shard copies before proceeding with the operation. If the requisite number of active shard copies are not available, then the write operation must wait and retry, until either the requisite shard copies have started or a timeout occurs. By default, write operations only wait for the primary shards to be active before proceeding (that is to say wait_for_active_shards is 1). This default can be overridden in the index settings dynamically by setting index.write.wait_for_active_shards. To alter this behavior per operation, use the wait_for_active_shards request parameter.

Valid values are all or any positive integer up to the total number of configured copies per shard in the index (which is number_of_replicas+1). Specifying a negative value or a number greater than the number of shard copies will throw an error.

For example, suppose you have a cluster of three nodes, A, B, and C and you create an index index with the number of replicas set to 3 (resulting in 4 shard copies, one more copy than there are nodes). If you attempt an indexing operation, by default the operation will only ensure the primary copy of each shard is available before proceeding. This means that even if B and C went down and A hosted the primary shard copies, the indexing operation would still proceed with only one copy of the data. If wait_for_active_shards is set on the request to 3 (and all three nodes are up), the indexing operation will require 3 active shard copies before proceeding. This requirement should be met because there are 3 active nodes in the cluster, each one holding a copy of the shard. However, if you set wait_for_active_shards to all (or to 4, which is the same in this situation), the indexing operation will not proceed as you do not have all 4 copies of each shard active in the index. The operation will timeout unless a new node is brought up in the cluster to host the fourth copy of the shard.

It is important to note that this setting greatly reduces the chances of the write operation not writing to the requisite number of shard copies, but it does not completely eliminate the possibility, because this check occurs before the write operation starts. After the write operation is underway, it is still possible for replication to fail on any number of shard copies but still succeed on the primary. The _shards section of the API response reveals the number of shard copies on which replication succeeded and failed.

External documentation

Path parameters

  • index string Required

    The name of the data stream or index to target. If the target doesn't exist and matches the name or wildcard (*) pattern of an index template with a data_stream definition, this request creates the data stream. If the target doesn't exist and doesn’t match a data stream template, this request creates the index.

  • id string Required

    A unique identifier for the document. To automatically generate a document ID, use the POST /<target>/_doc/ request format.

Query parameters

  • Only perform the operation if the document has this primary term.

  • Only perform the operation if the document has this sequence number.

  • True or false if to include the document source in the error message in case of parsing errors.

  • op_type string

    Set to create to only index the document if it does not already exist (put if absent). If a document with the specified _id already exists, the indexing operation will fail. The behavior is the same as using the <index>/_create endpoint. If a document ID is specified, this paramater defaults to index. Otherwise, it defaults to create. If the request targets a data stream, an op_type of create is required.

    Supported values include:

    • index: Overwrite any documents that already exist.
    • create: Only index documents that do not already exist.

    Values are index or create.

  • pipeline string

    The ID of the pipeline to use to preprocess incoming documents. If the index has a default ingest pipeline specified, setting the value to _none turns off the default ingest pipeline for this request. If a final pipeline is configured, it will always run regardless of the value of this parameter.

  • refresh string

    If true, Elasticsearch refreshes the affected shards to make this operation visible to search. If wait_for, it waits for a refresh to make this operation visible to search. If false, it does nothing with refreshes.

    Values are true, false, or wait_for.

  • If true, the destination must be an index alias.

  • If true, the request's actions must target a data stream (existing or to be created).

  • routing string

    A custom value that is used to route operations to a specific shard.

  • timeout string

    The period the request waits for the following operations: automatic index creation, dynamic mapping updates, waiting for active shards. Elasticsearch waits for at least the specified timeout period before failing. The actual wait time could be longer, particularly when multiple waits occur.

    This parameter is useful for situations where the primary shard assigned to perform the operation might not be available when the operation runs. Some reasons for this might be that the primary shard is currently recovering from a gateway or undergoing relocation. By default, the operation will wait on the primary shard to become available for at least 1 minute before failing and responding with an error. The actual wait time could be longer, particularly when multiple waits occur.

  • version number

    The explicit version number for concurrency control. It must be a non-negative long number.

  • The version type.

    Supported values include:

    • internal: Use internal versioning that starts at 1 and increments with each update or delete.
    • external: Only index the document if the specified version is strictly higher than the version of the stored document or if there is no existing document.
    • external_gte: Only index the document if the specified version is equal or higher than the version of the stored document or if there is no existing document. NOTE: The external_gte version type is meant for special use cases and should be used with care. If used incorrectly, it can result in loss of data.
    • force: This option is deprecated because it can cause primary and replica shards to diverge.

    Values are internal, external, external_gte, or force.

  • wait_for_active_shards number | string

    The number of shard copies that must be active before proceeding with the operation. You can set it to all or any positive integer up to the total number of shards in the index (number_of_replicas+1). The default value of 1 means it waits for each primary shard to be active.

application/json

Body Required

object object

Responses

PUT /{index}/_create/{id}
curl \
 --request PUT 'http://api.example.com/{index}/_create/{id}' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '"{\n  \"@timestamp\": \"2099-11-15T13:12:00\",\n  \"message\": \"GET /search HTTP/1.1 200 1070000\",\n  \"user\": {\n    \"id\": \"kimchy\"\n  }\n}"'
Request example
Run `PUT my-index-000001/_create/1` to index a document into the `my-index-000001` index if no document with that ID exists.
{
  "@timestamp": "2099-11-15T13:12:00",
  "message": "GET /search HTTP/1.1 200 1070000",
  "user": {
    "id": "kimchy"
  }
}





























































































































































































































Run a Fleet search Technical preview

POST /{index}/_fleet/_fleet_search

The purpose of the Fleet search API is to provide an API where the search will be run only after the provided checkpoint has been processed and is visible for searches inside of Elasticsearch.

Path parameters

  • index string Required

    A single target to search. If the target is an index alias, it must resolve to a single index.

Query parameters

  • analyzer string
  • Values are and, AND, or, or OR.

  • df string
  • docvalue_fields string | array[string]
  • expand_wildcards string | array[string]

    Supported values include:

    • all: Match any data stream or index, including hidden ones.
    • open: Match open, non-hidden indices. Also matches any non-hidden data stream.
    • closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.
    • hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.
    • none: Wildcard expressions are not accepted.
  • explain boolean
  • lenient boolean
  • routing string
  • scroll string

    A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

  • Supported values include:

    • query_then_fetch: Documents are scored using local term and document frequencies for the shard. This is usually faster but less accurate.
    • dfs_query_then_fetch: Documents are scored using global term and document frequencies across all shards. This is usually slower but more accurate.

    Values are query_then_fetch or dfs_query_then_fetch.

  • stats array[string]
  • stored_fields string | array[string]
  • Specifies which field to use for suggestions.

  • Supported values include:

    • missing: Only generate suggestions for terms that are not in the shard.
    • popular: Only suggest terms that occur in more docs on the shard than the original term.
    • always: Suggest any matching suggestions based on terms in the suggest text.

    Values are missing, popular, or always.

  • The source text for which the suggestions should be returned.

  • timeout string

    A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

  • track_total_hits boolean | number

    Number of hits matching the query to count accurately. If true, the exact number of hits is returned at the cost of some performance. If false, the response does not include the total number of hits matching the query. Defaults to 10,000 hits.

  • typed_keys boolean
  • version boolean
  • _source boolean | string | array[string]

    Defines how to fetch a source. Fetching can be disabled entirely, or the source can be filtered. Used as a query parameter along with the _source_includes and _source_excludes parameters.

  • _source_excludes string | array[string]
  • _source_includes string | array[string]
  • q string
  • size number
  • from number
  • sort string | array[string]
  • A comma separated list of checkpoints. When configured, the search API will only be executed on a shard after the relevant checkpoint has become visible for search. Defaults to an empty list which will cause Elasticsearch to immediately execute the search.

  • If true, returns partial results if there are shard request timeouts or shard failures. If false, returns an error with no partial results. Defaults to the configured cluster setting search.default_allow_partial_results, which is true by default.

application/json

Body

  • collapse object
    External documentation
  • explain boolean

    If true, returns detailed information about score computation as part of a hit.

  • ext object

    Configuration of search extensions defined by Elasticsearch plugins.

    Hide ext attribute Show ext attribute object
    • * object Additional properties
  • from number

    Starting document offset. By default, you cannot page through more than 10,000 hits using the from and size parameters. To page through more hits, use the search_after parameter.

  • Hide highlight attributes Show highlight attributes object
    • A string that contains each boundary character.

    • How far to scan for boundary characters.

    • Values are chars, sentence, or word.

    • Controls which locale is used to search for sentence and word boundaries. This parameter takes a form of a language tag, for example: "en-US", "fr-FR", "ja-JP".

    • force_source boolean Deprecated
    • Values are simple or span.

    • The size of the highlighted fragment in characters.

    • An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

      External documentation
    • If set to a non-negative value, highlighting stops at this defined maximum limit. The rest of the text is not processed, thus not highlighted and no error is returned The max_analyzed_offset query setting does not override the index.highlight.max_analyzed_offset setting, which prevails when it’s set to lower value than the query setting.

    • The amount of text you want to return from the beginning of the field if there are no matching fragments to highlight.

    • The maximum number of fragments to return. If the number of fragments is set to 0, no fragments are returned. Instead, the entire field contents are highlighted and returned. This can be handy when you need to highlight short texts such as a title or address, but fragmentation is not required. If number_of_fragments is 0, fragment_size is ignored.

    • options object
      Hide options attribute Show options attribute object
      • * object Additional properties
    • order string

      Value is score.

    • Controls the number of matching phrases in a document that are considered. Prevents the fvh highlighter from analyzing too many phrases and consuming too much memory. When using matched_fields, phrase_limit phrases per matched field are considered. Raising the limit increases query time and consumes more memory. Only supported by the fvh highlighter.

    • post_tags array[string]

      Use in conjunction with pre_tags to define the HTML tags to use for the highlighted text. By default, highlighted text is wrapped in <em> and </em> tags.

    • pre_tags array[string]

      Use in conjunction with post_tags to define the HTML tags to use for the highlighted text. By default, highlighted text is wrapped in <em> and </em> tags.

    • By default, only fields that contains a query match are highlighted. Set to false to highlight all fields.

    • Value is styled.

    • encoder string

      Values are default or html.

    • fields object Required
  • track_total_hits boolean | number

    Number of hits matching the query to count accurately. If true, the exact number of hits is returned at the cost of some performance. If false, the response does not include the total number of hits matching the query. Defaults to 10,000 hits.

  • indices_boost array[object]

    Boosts the _score of documents from specified indices.

    Hide indices_boost attribute Show indices_boost attribute object
    • * number Additional properties
  • docvalue_fields array[object]

    Array of wildcard (*) patterns. The request returns doc values for field names matching these patterns in the hits.fields property of the response.

    Hide docvalue_fields attributes Show docvalue_fields attributes object
    • field string Required

      Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • format string

      The format in which the values are returned.

  • Minimum _score for matching documents. Documents with a lower _score are not included in search results and results collected by aggregations.

  • An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

    External documentation
  • profile boolean
  • query object

    An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

    External documentation
  • rescore object | array[object]

    One of:
    Hide attributes Show attributes
    • query object
      Hide query attributes Show query attributes object
    • Hide learning_to_rank attributes Show learning_to_rank attributes object
      • model_id string Required

        The unique identifier of the trained model uploaded to Elasticsearch

      • params object

        Named parameters to be passed to the query templates used for feature

        Hide params attribute Show params attribute object
        • * object Additional properties
  • Retrieve a script evaluation (based on different fields) for each hit.

    Hide script_fields attribute Show script_fields attribute object
    • * object Additional properties
      Hide * attributes Show * attributes object
      • script object Required
        Hide script attributes Show script attributes object
        • source string | object

          One of:
        • id string
        • params object

          Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

          Hide params attribute Show params attribute object
          • * object Additional properties
        • lang string

          Any of:

          Values are painless, expression, mustache, or java.

        • options object
          Hide options attribute Show options attribute object
          • * string Additional properties
  • search_after array[number | string | boolean | null]

    A field value.

  • size number

    The number of hits to return. By default, you cannot page through more than 10,000 hits using the from and size parameters. To page through more hits, use the search_after parameter.

  • slice object
    Hide slice attributes Show slice attributes object
    • field string

      Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • id string Required
    • max number Required
  • sort string | object | array[string | object]

    One of:

    Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

  • _source boolean | object

    Defines how to fetch a source. Fetching can be disabled entirely, or the source can be filtered.

    One of:
  • fields array[object]

    Array of wildcard (*) patterns. The request returns values for field names matching these patterns in the hits.fields property of the response.

    Hide fields attributes Show fields attributes object
    • field string Required

      Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • format string

      The format in which the values are returned.

  • suggest object
    Hide suggest attribute Show suggest attribute object
    • text string

      Global suggest text, to avoid repetition when the same text is used in several suggesters

  • Maximum number of documents to collect for each shard. If a query reaches this limit, Elasticsearch terminates the query early. Elasticsearch collects documents before sorting. Defaults to 0, which does not terminate query execution early.

  • timeout string

    Specifies the period of time to wait for a response from each shard. If no response is received before the timeout expires, the request fails and returns an error. Defaults to no timeout.

  • If true, calculate and return document scores, even if the scores are not used for sorting.

  • version boolean

    If true, returns document version as part of a hit.

  • If true, returns sequence number and primary term of the last modification of each hit. See Optimistic concurrency control.

  • stored_fields string | array[string]
  • pit object
    Hide pit attributes Show pit attributes object
    • id string Required
    • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

  • Hide runtime_mappings attribute Show runtime_mappings attribute object
    • * object Additional properties
      Hide * attributes Show * attributes object
      • fields object

        For type composite

        Hide fields attribute Show fields attribute object
        • * object Additional properties
          Hide * attribute Show * attribute object
          • type string Required

            Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

      • fetch_fields array[object]

        For type lookup

        Hide fetch_fields attributes Show fetch_fields attributes object
        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • format string
      • format string

        A custom format for date type runtime fields.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • script object
        Hide script attributes Show script attributes object
        • source string | object

          One of:
        • id string
        • params object

          Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

          Hide params attribute Show params attribute object
          • * object Additional properties
        • lang string

          Any of:

          Values are painless, expression, mustache, or java.

        • options object
          Hide options attribute Show options attribute object
          • * string Additional properties
      • type string Required

        Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

  • stats array[string]

    Stats groups to associate with the search. Each group maintains a statistics aggregation for its associated searches. You can retrieve these stats using the indices stats API.

Responses

POST /{index}/_fleet/_fleet_search
curl \
 --request POST 'http://api.example.com/{index}/_fleet/_fleet_search' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '{"aggregations":{},"collapse":{},"explain":true,"ext":{"additionalProperty1":{},"additionalProperty2":{}},"from":42.0,"highlight":{"":"plain","boundary_chars":"string","boundary_max_scan":42.0,"boundary_scanner":"chars","boundary_scanner_locale":"string","force_source":true,"fragmenter":"simple","fragment_size":42.0,"highlight_filter":true,"highlight_query":{},"max_fragment_length":42.0,"max_analyzed_offset":42.0,"no_match_size":42.0,"number_of_fragments":42.0,"options":{"additionalProperty1":{},"additionalProperty2":{}},"order":"score","phrase_limit":42.0,"post_tags":["string"],"pre_tags":["string"],"require_field_match":true,"tags_schema":"styled","encoder":"default","fields":{}},"track_total_hits":true,"indices_boost":[{"additionalProperty1":42.0,"additionalProperty2":42.0}],"docvalue_fields":[{"field":"string","format":"string","include_unmapped":true}],"min_score":42.0,"post_filter":{},"profile":true,"query":{},"rescore":{"window_size":42.0,"query":{"rescore_query":{},"query_weight":42.0,"rescore_query_weight":42.0,"score_mode":"avg"},"learning_to_rank":{"model_id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}}}},"script_fields":{"additionalProperty1":{"script":{"":"painless","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true},"additionalProperty2":{"script":{"":"painless","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true}},"search_after":[42.0],"size":42.0,"slice":{"field":"string","id":"string","max":42.0},"":true,"fields":[{"field":"string","format":"string","include_unmapped":true}],"suggest":{"text":"string"},"terminate_after":42.0,"timeout":"string","track_scores":true,"version":true,"seq_no_primary_term":true,"stored_fields":"string","pit":{"id":"string","keep_alive":"string"},"runtime_mappings":{"additionalProperty1":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"":"painless","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"},"additionalProperty2":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"":"painless","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"}},"stats":["string"]}'






















































































































































































































































































Get index templates

GET /_template

Get information about one or more index templates.

IMPORTANT: This documentation is about legacy index templates, which are deprecated and will be replaced by the composable templates introduced in Elasticsearch 7.8.

External documentation

Query parameters

  • If true, returns settings in flat format.

  • local boolean

    If true, the request retrieves information from the local node only.

  • Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

Responses

GET /_template
curl \
 --request GET 'http://api.example.com/_template' \
 --header "Authorization: $API_KEY"




















































































































































































































































































































































































Get pipelines Added in 5.0.0

GET /_ingest/pipeline

Get information about one or more ingest pipelines. This API returns a local reference of the pipeline.

External documentation

Query parameters

  • Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

  • summary boolean

    Return pipelines without their definitions (default: false)

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • * object Additional properties
      Hide * attributes Show * attributes object
      • Description of the ingest pipeline.

      • on_failure array[object]

        Processors to run immediately after a processor failure.

        Hide on_failure attributes Show on_failure attributes object
        • append object
          Hide append attributes Show append attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If false, the processor does not append values already present in the field.

        • Hide attachment attributes Show attachment attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The number of chars being used for extraction to prevent huge fields. Use -1 for no limit.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Array of properties to select to be stored. Can be content, title, name, author, keywords, date, content_type, content_length, language.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true, the binary field will be removed from the document

          • Field containing the name of the resource to decode. If specified, the processor passes this resource name to the underlying Tika library to enable Resource Name Based Detection.

        • bytes object
          Hide bytes attributes Show bytes attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • circle object
          Hide circle attributes Show circle attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • error_distance number Required

            The difference between the resulting inscribed distance from center to side and the circle’s radius (measured in meters for geo_shape, unit-less for shape).

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • shape_type string Required

            Values are geo_shape or shape.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide community_id attributes Show community_id attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • seed number

            Seed for the community ID hash. Must be between 0 and 65535 (inclusive). The seed can prevent hash collisions between network domains, such as a staging and production network that use the same addressing scheme.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • convert object
          Hide convert attributes Show convert attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • type string Required

            Values are integer, long, double, float, boolean, ip, string, or auto.

        • csv object
          Hide csv attributes Show csv attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Value used to fill empty fields. Empty fields are skipped if this is not provided. An empty field is one with no value (2 consecutive separators) or empty quotes ("").

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • quote string

            Quote used in CSV, has to be single character string.

          • Separator used in CSV, has to be single character string.

          • target_fields string | array[string] Required
          • trim boolean

            Trim whitespaces in unquoted fields.

        • date object
          Hide date attributes Show date attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • formats array[string] Required

            An array of the expected date formats. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • locale string

            The locale to use when parsing the date, relevant when parsing month names or week days. Supports template snippets.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • timezone string

            The timezone to use when parsing the date. Supports template snippets.

          • The format to use when writing the date to target_field. Must be a valid java time pattern.

        • Hide date_index_name attributes Show date_index_name attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • date_formats array[string]

            An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • date_rounding string Required

            How to round the date when formatting the date into the index name. Valid values are: y (year), M (month), w (week), d (day), h (hour), m (minute) and s (second). Supports template snippets.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • The format to be used when printing the parsed date into the index name. A valid java time pattern is expected here. Supports template snippets.

          • A prefix of the index name to be prepended before the printed date. Supports template snippets.

          • locale string

            The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.

          • timezone string

            The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.

        • dissect object
          Hide dissect attributes Show dissect attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The character(s) that separate the appended fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to apply to the field.

        • Hide dot_expander attributes Show dot_expander attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • override boolean

            Controls the behavior when there is already an existing nested object that conflicts with the expanded field. When false, the processor will merge conflicts by combining the old and the new values into an array. When true, the value from the expanded field will overwrite the existing value.

          • path string

            The field that contains the field to expand. Only required if the field to expand is part another object field, because the field option can only understand leaf fields.

        • drop object
          Hide drop attributes Show drop attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • enrich object
          Hide enrich attributes Show enrich attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The maximum number of matched documents to include under the configured target field. The target_field will be turned into a json array if max_matches is higher than 1, otherwise target_field will become a json object. In order to avoid documents getting too large, the maximum allowed value is 128.

          • override boolean

            If processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • policy_name string Required

            The name of the enrich policy to use.

          • Values are intersects, disjoint, within, or contains.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • fail object
          Hide fail attributes Show fail attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • message string Required

            The error message thrown by the processor. Supports template snippets.

        • Hide fingerprint attributes Show fingerprint attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • fields string | array[string] Required
          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • salt string

            Salt value for the hash function.

          • method string

            Values are MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.

          • If true, the processor ignores any missing fields. If all fields are missing, the processor silently exits without modifying the document.

        • foreach object
          Hide foreach attributes Show foreach attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true, the processor silently exits without changing the document if the field is null or missing.

          • processor object Required
        • Hide ip_location attributes Show ip_location attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found IP location data will be returned, even if the field contains an array.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the IP location lookup.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • geo_grid object
          Hide geo_grid attributes Show geo_grid attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            The field to interpret as a geo-tile.= The field format is determined by the tile_type.

          • tile_type string Required

            Values are geotile, geohex, or geohash.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Values are geojson or wkt.

        • geoip object
          Hide geoip attributes Show geoip attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found geoip data will be returned, even if the field contains an array.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the geoip lookup.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • grok object
          Hide grok attributes Show grok attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Must be disabled or v1. If v1, the processor uses patterns with Elastic Common Schema (ECS) field names.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition.

          • patterns array[string] Required

            An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches.

          • When true, _ingest._grok_match_index will be inserted into your matched document’s metadata with the index into the pattern found in patterns that matched.

        • gsub object
          Hide gsub attributes Show gsub attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to be replaced.

          • replacement string Required

            The string to replace the matching patterns with.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide html_strip attributes Show html_strip attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document,

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide inference attributes Show inference attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • model_id string Required
          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Maps the document field names to the known field names of the model. This mapping takes precedence over any default mappings provided in the model configuration.

          • If true and any of the input fields defined in input_ouput are missing then those missing fields are quietly ignored, otherwise a missing field causes a failure. Only applies when using input_output configurations to explicitly list the input fields.

        • join object
          Hide join attributes Show join attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • separator string Required

            The separator character.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • json object
          Hide json attributes Show json attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Flag that forces the parsed JSON to be added at the top level of the document. target_field must not be set when this option is chosen.

          • Values are replace or merge.

          • When set to true, the JSON parser will not fail if the JSON contains duplicate keys. Instead, the last encountered value for any duplicate key wins.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • kv object
          Hide kv attributes Show kv attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • exclude_keys array[string]

            List of keys to exclude from document.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field_split string Required

            Regex pattern to use for splitting key-value pairs.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • include_keys array[string]

            List of keys to filter and insert into document. Defaults to including all keys.

          • prefix string

            Prefix to be added to extracted keys.

          • If true. strip brackets (), <>, [] as well as quotes ' and " from extracted values.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • trim_key string

            String of characters to trim from extracted keys.

          • String of characters to trim from extracted values.

          • value_split string Required

            Regex pattern to use for splitting the key from the value within a key-value pair.

        • Hide lowercase attributes Show lowercase attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide network_direction attributes Show network_direction attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • internal_networks array[string]

            List of internal networks. Supports IPv4 and IPv6 addresses and ranges in CIDR notation. Also supports the named ranges listed below. These may be constructed with template snippets. Must specify only one of internal_networks or internal_networks_field.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • pipeline object
          Hide pipeline attributes Show pipeline attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • name string Required
          • Whether to ignore missing pipelines instead of failing.

        • redact object
          Hide redact attributes Show redact attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • patterns array[string] Required

            A list of grok expressions to match and redact named captures with

          • prefix string

            Start a redacted section with this token

          • suffix string

            End a redacted section with this token

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • If true and the current license does not support running redact processors, then the processor quietly exits without modifying the document

          • If true then ingest metadata _ingest._redact._is_redacted is set to true if the document has been redacted

        • Hide registered_domain attributes Show registered_domain attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • remove object
          Hide remove attributes Show remove attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string | array[string] Required
          • keep string | array[string]
          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • rename object
          Hide rename attributes Show rename attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • reroute object
          Hide reroute attributes Show reroute attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • A static value for the target. Can’t be set when the dataset or namespace option is set.

        • script object
          Hide script attributes Show script attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • id string
          • params object

            Object containing parameters for the script.

        • set object
          Hide set attributes Show set attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and value is a template snippet that evaluates to null or the empty string, the processor quietly exits without modifying the document.

          • The media type for encoding value. Applies only when value is a template snippet. Must be one of application/json, text/plain, or application/x-www-form-urlencoded.

          • override boolean

            If true processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • value object

            The value to be set for the field. Supports template snippets. May specify only one of value or copy_from.

        • Hide set_security_user attributes Show set_security_user attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what user related properties are added to the field.

        • sort object
          Hide sort attributes Show sort attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • order string

            Values are asc or desc.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • split object
          Hide split attributes Show split attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Preserves empty trailing fields, if any.

          • separator string Required

            A regex which matches the separator, for example, , or \s+.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide terminate attributes Show terminate attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • trim object
          Hide trim attributes Show trim attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide uppercase attributes Show uppercase attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide urldecode attributes Show urldecode attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide uri_parts attributes Show uri_parts attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • If true, the processor copies the unparsed URI to <target_field>.original.

          • If true, the processor removes the field after parsing the URI string. If parsing fails, the processor does not remove the field.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide user_agent attributes Show user_agent attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The name of the file in the config/ingest-user-agent directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-user-agent will use the regexes.yaml from uap-core it ships with.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what properties are added to target_field.

            Values are name, os, device, original, or version.

          • Extracts device type from the user agent string on a best-effort basis.

      • processors array[object]

        Processors used to perform transformations on documents before indexing. Processors run sequentially in the order specified.

        Hide processors attributes Show processors attributes object
        • append object
          Hide append attributes Show append attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If false, the processor does not append values already present in the field.

        • Hide attachment attributes Show attachment attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The number of chars being used for extraction to prevent huge fields. Use -1 for no limit.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Array of properties to select to be stored. Can be content, title, name, author, keywords, date, content_type, content_length, language.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true, the binary field will be removed from the document

          • Field containing the name of the resource to decode. If specified, the processor passes this resource name to the underlying Tika library to enable Resource Name Based Detection.

        • bytes object
          Hide bytes attributes Show bytes attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • circle object
          Hide circle attributes Show circle attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • error_distance number Required

            The difference between the resulting inscribed distance from center to side and the circle’s radius (measured in meters for geo_shape, unit-less for shape).

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • shape_type string Required

            Values are geo_shape or shape.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide community_id attributes Show community_id attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • seed number

            Seed for the community ID hash. Must be between 0 and 65535 (inclusive). The seed can prevent hash collisions between network domains, such as a staging and production network that use the same addressing scheme.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • convert object
          Hide convert attributes Show convert attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • type string Required

            Values are integer, long, double, float, boolean, ip, string, or auto.

        • csv object
          Hide csv attributes Show csv attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Value used to fill empty fields. Empty fields are skipped if this is not provided. An empty field is one with no value (2 consecutive separators) or empty quotes ("").

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • quote string

            Quote used in CSV, has to be single character string.

          • Separator used in CSV, has to be single character string.

          • target_fields string | array[string] Required
          • trim boolean

            Trim whitespaces in unquoted fields.

        • date object
          Hide date attributes Show date attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • formats array[string] Required

            An array of the expected date formats. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • locale string

            The locale to use when parsing the date, relevant when parsing month names or week days. Supports template snippets.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • timezone string

            The timezone to use when parsing the date. Supports template snippets.

          • The format to use when writing the date to target_field. Must be a valid java time pattern.

        • Hide date_index_name attributes Show date_index_name attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • date_formats array[string]

            An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • date_rounding string Required

            How to round the date when formatting the date into the index name. Valid values are: y (year), M (month), w (week), d (day), h (hour), m (minute) and s (second). Supports template snippets.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • The format to be used when printing the parsed date into the index name. A valid java time pattern is expected here. Supports template snippets.

          • A prefix of the index name to be prepended before the printed date. Supports template snippets.

          • locale string

            The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.

          • timezone string

            The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.

        • dissect object
          Hide dissect attributes Show dissect attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The character(s) that separate the appended fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to apply to the field.

        • Hide dot_expander attributes Show dot_expander attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • override boolean

            Controls the behavior when there is already an existing nested object that conflicts with the expanded field. When false, the processor will merge conflicts by combining the old and the new values into an array. When true, the value from the expanded field will overwrite the existing value.

          • path string

            The field that contains the field to expand. Only required if the field to expand is part another object field, because the field option can only understand leaf fields.

        • drop object
          Hide drop attributes Show drop attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • enrich object
          Hide enrich attributes Show enrich attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The maximum number of matched documents to include under the configured target field. The target_field will be turned into a json array if max_matches is higher than 1, otherwise target_field will become a json object. In order to avoid documents getting too large, the maximum allowed value is 128.

          • override boolean

            If processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • policy_name string Required

            The name of the enrich policy to use.

          • Values are intersects, disjoint, within, or contains.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • fail object
          Hide fail attributes Show fail attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • message string Required

            The error message thrown by the processor. Supports template snippets.

        • Hide fingerprint attributes Show fingerprint attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • fields string | array[string] Required
          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • salt string

            Salt value for the hash function.

          • method string

            Values are MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.

          • If true, the processor ignores any missing fields. If all fields are missing, the processor silently exits without modifying the document.

        • foreach object
          Hide foreach attributes Show foreach attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true, the processor silently exits without changing the document if the field is null or missing.

          • processor object Required
        • Hide ip_location attributes Show ip_location attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found IP location data will be returned, even if the field contains an array.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the IP location lookup.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • geo_grid object
          Hide geo_grid attributes Show geo_grid attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            The field to interpret as a geo-tile.= The field format is determined by the tile_type.

          • tile_type string Required

            Values are geotile, geohex, or geohash.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Values are geojson or wkt.

        • geoip object
          Hide geoip attributes Show geoip attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found geoip data will be returned, even if the field contains an array.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the geoip lookup.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • grok object
          Hide grok attributes Show grok attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Must be disabled or v1. If v1, the processor uses patterns with Elastic Common Schema (ECS) field names.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition.

          • patterns array[string] Required

            An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches.

          • When true, _ingest._grok_match_index will be inserted into your matched document’s metadata with the index into the pattern found in patterns that matched.

        • gsub object
          Hide gsub attributes Show gsub attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to be replaced.

          • replacement string Required

            The string to replace the matching patterns with.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide html_strip attributes Show html_strip attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document,

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide inference attributes Show inference attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • model_id string Required
          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Maps the document field names to the known field names of the model. This mapping takes precedence over any default mappings provided in the model configuration.

          • If true and any of the input fields defined in input_ouput are missing then those missing fields are quietly ignored, otherwise a missing field causes a failure. Only applies when using input_output configurations to explicitly list the input fields.

        • join object
          Hide join attributes Show join attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • separator string Required

            The separator character.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • json object
          Hide json attributes Show json attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Flag that forces the parsed JSON to be added at the top level of the document. target_field must not be set when this option is chosen.

          • Values are replace or merge.

          • When set to true, the JSON parser will not fail if the JSON contains duplicate keys. Instead, the last encountered value for any duplicate key wins.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • kv object
          Hide kv attributes Show kv attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • exclude_keys array[string]

            List of keys to exclude from document.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field_split string Required

            Regex pattern to use for splitting key-value pairs.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • include_keys array[string]

            List of keys to filter and insert into document. Defaults to including all keys.

          • prefix string

            Prefix to be added to extracted keys.

          • If true. strip brackets (), <>, [] as well as quotes ' and " from extracted values.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • trim_key string

            String of characters to trim from extracted keys.

          • String of characters to trim from extracted values.

          • value_split string Required

            Regex pattern to use for splitting the key from the value within a key-value pair.

        • Hide lowercase attributes Show lowercase attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide network_direction attributes Show network_direction attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • internal_networks array[string]

            List of internal networks. Supports IPv4 and IPv6 addresses and ranges in CIDR notation. Also supports the named ranges listed below. These may be constructed with template snippets. Must specify only one of internal_networks or internal_networks_field.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • pipeline object
          Hide pipeline attributes Show pipeline attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • name string Required
          • Whether to ignore missing pipelines instead of failing.

        • redact object
          Hide redact attributes Show redact attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • patterns array[string] Required

            A list of grok expressions to match and redact named captures with

          • prefix string

            Start a redacted section with this token

          • suffix string

            End a redacted section with this token

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • If true and the current license does not support running redact processors, then the processor quietly exits without modifying the document

          • If true then ingest metadata _ingest._redact._is_redacted is set to true if the document has been redacted

        • Hide registered_domain attributes Show registered_domain attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • remove object
          Hide remove attributes Show remove attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string | array[string] Required
          • keep string | array[string]
          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • rename object
          Hide rename attributes Show rename attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • reroute object
          Hide reroute attributes Show reroute attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • A static value for the target. Can’t be set when the dataset or namespace option is set.

        • script object
          Hide script attributes Show script attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • id string
          • params object

            Object containing parameters for the script.

        • set object
          Hide set attributes Show set attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and value is a template snippet that evaluates to null or the empty string, the processor quietly exits without modifying the document.

          • The media type for encoding value. Applies only when value is a template snippet. Must be one of application/json, text/plain, or application/x-www-form-urlencoded.

          • override boolean

            If true processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • value object

            The value to be set for the field. Supports template snippets. May specify only one of value or copy_from.

        • Hide set_security_user attributes Show set_security_user attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what user related properties are added to the field.

        • sort object
          Hide sort attributes Show sort attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • order string

            Values are asc or desc.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • split object
          Hide split attributes Show split attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Preserves empty trailing fields, if any.

          • separator string Required

            A regex which matches the separator, for example, , or \s+.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide terminate attributes Show terminate attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • trim object
          Hide trim attributes Show trim attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide uppercase attributes Show uppercase attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide urldecode attributes Show urldecode attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide uri_parts attributes Show uri_parts attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • If true, the processor copies the unparsed URI to <target_field>.original.

          • If true, the processor removes the field after parsing the URI string. If parsing fails, the processor does not remove the field.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide user_agent attributes Show user_agent attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The name of the file in the config/ingest-user-agent directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-user-agent will use the regexes.yaml from uap-core it ships with.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what properties are added to target_field.

            Values are name, os, device, original, or version.

          • Extracts device type from the user agent string on a best-effort basis.

      • version number
      • deprecated boolean

        Marks this ingest pipeline as deprecated. When a deprecated ingest pipeline is referenced as the default or final pipeline when creating or updating a non-deprecated index template, Elasticsearch will emit a deprecation warning.

      • _meta object
        Hide _meta attribute Show _meta attribute object
        • * object Additional properties
GET /_ingest/pipeline
curl \
 --request GET 'http://api.example.com/_ingest/pipeline' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response for retrieving information about an ingest pipeline.
{
  "my-pipeline-id" : {
    "description" : "describe pipeline",
    "version" : 123,
    "processors" : [
      {
        "set" : {
          "field" : "foo",
          "value" : "bar"
        }
      }
    ]
  }
}




















Simulate data ingestion Technical preview

GET /_ingest/_simulate

Run ingest pipelines against a set of provided documents, optionally with substitute pipeline definitions, to simulate ingesting data into an index.

This API is meant to be used for troubleshooting or pipeline development, as it does not actually index any data into Elasticsearch.

The API runs the default and final pipeline for that index against a set of documents provided in the body of the request. If a pipeline contains a reroute processor, it follows that reroute processor to the new index, running that index's pipelines as well the same way that a non-simulated ingest would. No data is indexed into Elasticsearch. Instead, the transformed document is returned, along with the list of pipelines that have been run and the name of the index where the document would have been indexed if this were not a simulation. The transformed document is validated against the mappings that would apply to this index, and any validation error is reported in the result.

This API differs from the simulate pipeline API in that you specify a single pipeline for that API, and it runs only that one pipeline. The simulate pipeline API is more useful for developing a single pipeline, while the simulate ingest API is more useful for troubleshooting the interaction of the various pipelines that get applied when ingesting into an index.

By default, the pipeline definitions that are currently in the system are used. However, you can supply substitute pipeline definitions in the body of the request. These will be used in place of the pipeline definitions that are already in the system. This can be used to replace existing pipeline definitions or to create new ones. The pipeline substitutions are used only within this request.

Query parameters

  • pipeline string

    The pipeline to use as the default pipeline. This value can be used to override the default pipeline of the index.

application/json

Body Required

  • docs array[object] Required

    Sample documents to test in the pipeline.

    Hide docs attributes Show docs attributes object
  • A map of component template names to substitute component template definition objects.

    Hide component_template_substitutions attribute Show component_template_substitutions attribute object
  • A map of index template names to substitute index template definition objects.

    Hide index_template_substitutions attribute Show index_template_substitutions attribute object
  • Hide mapping_addition attributes Show mapping_addition attributes object
    • Hide all_field attributes Show all_field attributes object
    • dynamic string

      Values are strict, runtime, true, or false.

    • dynamic_templates array[object]
    • Hide _field_names attribute Show _field_names attribute object
    • Hide index_field attribute Show index_field attribute object
    • _meta object
      Hide _meta attribute Show _meta attribute object
      • * object Additional properties
    • _routing object
      Hide _routing attribute Show _routing attribute object
    • _size object
      Hide _size attribute Show _size attribute object
    • _source object
      Hide _source attributes Show _source attributes object
    • runtime object
      Hide runtime attribute Show runtime attribute object
      • * object Additional properties
        Hide * attributes Show * attributes object
        • fields object

          For type composite

          Hide fields attribute Show fields attribute object
          • * object Additional properties
            Hide * attribute Show * attribute object
            • type string Required

              Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

        • fetch_fields array[object]

          For type lookup

          Hide fetch_fields attributes Show fetch_fields attributes object
          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • format string
        • format string

          A custom format for date type runtime fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • script object
          Hide script attributes Show script attributes object
          • source string | object

            One of:
          • id string
          • params object

            Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

            Hide params attribute Show params attribute object
            • * object Additional properties
          • lang string

            Any of:

            Values are painless, expression, mustache, or java.

          • options object
            Hide options attribute Show options attribute object
            • * string Additional properties
        • type string Required

          Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

    • enabled boolean
    • Values are true or false.

    • Hide _data_stream_timestamp attribute Show _data_stream_timestamp attribute object
  • Pipelines to test. If you don’t specify the pipeline request path parameter, this parameter is required. If you specify both this and the request path parameter, the API only uses the request path parameter.

    Hide pipeline_substitutions attribute Show pipeline_substitutions attribute object
    • * object Additional properties
      Hide * attributes Show * attributes object
      • Description of the ingest pipeline.

      • on_failure array[object]

        Processors to run immediately after a processor failure.

        Hide on_failure attributes Show on_failure attributes object
        • append object
          Hide append attributes Show append attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If false, the processor does not append values already present in the field.

        • Hide attachment attributes Show attachment attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The number of chars being used for extraction to prevent huge fields. Use -1 for no limit.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Array of properties to select to be stored. Can be content, title, name, author, keywords, date, content_type, content_length, language.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true, the binary field will be removed from the document

          • Field containing the name of the resource to decode. If specified, the processor passes this resource name to the underlying Tika library to enable Resource Name Based Detection.

        • bytes object
          Hide bytes attributes Show bytes attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • circle object
          Hide circle attributes Show circle attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • error_distance number Required

            The difference between the resulting inscribed distance from center to side and the circle’s radius (measured in meters for geo_shape, unit-less for shape).

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • shape_type string Required

            Values are geo_shape or shape.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide community_id attributes Show community_id attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • seed number

            Seed for the community ID hash. Must be between 0 and 65535 (inclusive). The seed can prevent hash collisions between network domains, such as a staging and production network that use the same addressing scheme.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • convert object
          Hide convert attributes Show convert attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • type string Required

            Values are integer, long, double, float, boolean, ip, string, or auto.

        • csv object
          Hide csv attributes Show csv attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Value used to fill empty fields. Empty fields are skipped if this is not provided. An empty field is one with no value (2 consecutive separators) or empty quotes ("").

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • quote string

            Quote used in CSV, has to be single character string.

          • Separator used in CSV, has to be single character string.

          • target_fields string | array[string] Required
          • trim boolean

            Trim whitespaces in unquoted fields.

        • date object
          Hide date attributes Show date attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • formats array[string] Required

            An array of the expected date formats. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • locale string

            The locale to use when parsing the date, relevant when parsing month names or week days. Supports template snippets.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • timezone string

            The timezone to use when parsing the date. Supports template snippets.

          • The format to use when writing the date to target_field. Must be a valid java time pattern.

        • Hide date_index_name attributes Show date_index_name attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • date_formats array[string]

            An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • date_rounding string Required

            How to round the date when formatting the date into the index name. Valid values are: y (year), M (month), w (week), d (day), h (hour), m (minute) and s (second). Supports template snippets.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • The format to be used when printing the parsed date into the index name. A valid java time pattern is expected here. Supports template snippets.

          • A prefix of the index name to be prepended before the printed date. Supports template snippets.

          • locale string

            The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.

          • timezone string

            The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.

        • dissect object
          Hide dissect attributes Show dissect attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The character(s) that separate the appended fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to apply to the field.

        • Hide dot_expander attributes Show dot_expander attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • override boolean

            Controls the behavior when there is already an existing nested object that conflicts with the expanded field. When false, the processor will merge conflicts by combining the old and the new values into an array. When true, the value from the expanded field will overwrite the existing value.

          • path string

            The field that contains the field to expand. Only required if the field to expand is part another object field, because the field option can only understand leaf fields.

        • drop object
          Hide drop attributes Show drop attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • enrich object
          Hide enrich attributes Show enrich attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The maximum number of matched documents to include under the configured target field. The target_field will be turned into a json array if max_matches is higher than 1, otherwise target_field will become a json object. In order to avoid documents getting too large, the maximum allowed value is 128.

          • override boolean

            If processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • policy_name string Required

            The name of the enrich policy to use.

          • Values are intersects, disjoint, within, or contains.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • fail object
          Hide fail attributes Show fail attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • message string Required

            The error message thrown by the processor. Supports template snippets.

        • Hide fingerprint attributes Show fingerprint attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • fields string | array[string] Required
          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • salt string

            Salt value for the hash function.

          • method string

            Values are MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.

          • If true, the processor ignores any missing fields. If all fields are missing, the processor silently exits without modifying the document.

        • foreach object
          Hide foreach attributes Show foreach attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true, the processor silently exits without changing the document if the field is null or missing.

          • processor object Required
        • Hide ip_location attributes Show ip_location attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found IP location data will be returned, even if the field contains an array.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the IP location lookup.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • geo_grid object
          Hide geo_grid attributes Show geo_grid attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            The field to interpret as a geo-tile.= The field format is determined by the tile_type.

          • tile_type string Required

            Values are geotile, geohex, or geohash.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Values are geojson or wkt.

        • geoip object
          Hide geoip attributes Show geoip attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found geoip data will be returned, even if the field contains an array.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the geoip lookup.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • grok object
          Hide grok attributes Show grok attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Must be disabled or v1. If v1, the processor uses patterns with Elastic Common Schema (ECS) field names.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition.

          • patterns array[string] Required

            An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches.

          • When true, _ingest._grok_match_index will be inserted into your matched document’s metadata with the index into the pattern found in patterns that matched.

        • gsub object
          Hide gsub attributes Show gsub attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to be replaced.

          • replacement string Required

            The string to replace the matching patterns with.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide html_strip attributes Show html_strip attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document,

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide inference attributes Show inference attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • model_id string Required
          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Maps the document field names to the known field names of the model. This mapping takes precedence over any default mappings provided in the model configuration.

          • If true and any of the input fields defined in input_ouput are missing then those missing fields are quietly ignored, otherwise a missing field causes a failure. Only applies when using input_output configurations to explicitly list the input fields.

        • join object
          Hide join attributes Show join attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • separator string Required

            The separator character.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • json object
          Hide json attributes Show json attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Flag that forces the parsed JSON to be added at the top level of the document. target_field must not be set when this option is chosen.

          • Values are replace or merge.

          • When set to true, the JSON parser will not fail if the JSON contains duplicate keys. Instead, the last encountered value for any duplicate key wins.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • kv object
          Hide kv attributes Show kv attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • exclude_keys array[string]

            List of keys to exclude from document.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field_split string Required

            Regex pattern to use for splitting key-value pairs.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • include_keys array[string]

            List of keys to filter and insert into document. Defaults to including all keys.

          • prefix string

            Prefix to be added to extracted keys.

          • If true. strip brackets (), <>, [] as well as quotes ' and " from extracted values.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • trim_key string

            String of characters to trim from extracted keys.

          • String of characters to trim from extracted values.

          • value_split string Required

            Regex pattern to use for splitting the key from the value within a key-value pair.

        • Hide lowercase attributes Show lowercase attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide network_direction attributes Show network_direction attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • internal_networks array[string]

            List of internal networks. Supports IPv4 and IPv6 addresses and ranges in CIDR notation. Also supports the named ranges listed below. These may be constructed with template snippets. Must specify only one of internal_networks or internal_networks_field.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • pipeline object
          Hide pipeline attributes Show pipeline attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • name string Required
          • Whether to ignore missing pipelines instead of failing.

        • redact object
          Hide redact attributes Show redact attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • patterns array[string] Required

            A list of grok expressions to match and redact named captures with

          • prefix string

            Start a redacted section with this token

          • suffix string

            End a redacted section with this token

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • If true and the current license does not support running redact processors, then the processor quietly exits without modifying the document

          • If true then ingest metadata _ingest._redact._is_redacted is set to true if the document has been redacted

        • Hide registered_domain attributes Show registered_domain attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • remove object
          Hide remove attributes Show remove attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string | array[string] Required
          • keep string | array[string]
          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • rename object
          Hide rename attributes Show rename attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • reroute object
          Hide reroute attributes Show reroute attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • A static value for the target. Can’t be set when the dataset or namespace option is set.

        • script object
          Hide script attributes Show script attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • id string
          • params object

            Object containing parameters for the script.

        • set object
          Hide set attributes Show set attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and value is a template snippet that evaluates to null or the empty string, the processor quietly exits without modifying the document.

          • The media type for encoding value. Applies only when value is a template snippet. Must be one of application/json, text/plain, or application/x-www-form-urlencoded.

          • override boolean

            If true processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • value object

            The value to be set for the field. Supports template snippets. May specify only one of value or copy_from.

        • Hide set_security_user attributes Show set_security_user attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what user related properties are added to the field.

        • sort object
          Hide sort attributes Show sort attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • order string

            Values are asc or desc.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • split object
          Hide split attributes Show split attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Preserves empty trailing fields, if any.

          • separator string Required

            A regex which matches the separator, for example, , or \s+.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide terminate attributes Show terminate attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • trim object
          Hide trim attributes Show trim attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide uppercase attributes Show uppercase attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide urldecode attributes Show urldecode attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide uri_parts attributes Show uri_parts attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • If true, the processor copies the unparsed URI to <target_field>.original.

          • If true, the processor removes the field after parsing the URI string. If parsing fails, the processor does not remove the field.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide user_agent attributes Show user_agent attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The name of the file in the config/ingest-user-agent directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-user-agent will use the regexes.yaml from uap-core it ships with.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what properties are added to target_field.

            Values are name, os, device, original, or version.

          • Extracts device type from the user agent string on a best-effort basis.

      • processors array[object]

        Processors used to perform transformations on documents before indexing. Processors run sequentially in the order specified.

        Hide processors attributes Show processors attributes object
        • append object
          Hide append attributes Show append attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If false, the processor does not append values already present in the field.

        • Hide attachment attributes Show attachment attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The number of chars being used for extraction to prevent huge fields. Use -1 for no limit.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Array of properties to select to be stored. Can be content, title, name, author, keywords, date, content_type, content_length, language.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true, the binary field will be removed from the document

          • Field containing the name of the resource to decode. If specified, the processor passes this resource name to the underlying Tika library to enable Resource Name Based Detection.

        • bytes object
          Hide bytes attributes Show bytes attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • circle object
          Hide circle attributes Show circle attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • error_distance number Required

            The difference between the resulting inscribed distance from center to side and the circle’s radius (measured in meters for geo_shape, unit-less for shape).

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • shape_type string Required

            Values are geo_shape or shape.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide community_id attributes Show community_id attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • seed number

            Seed for the community ID hash. Must be between 0 and 65535 (inclusive). The seed can prevent hash collisions between network domains, such as a staging and production network that use the same addressing scheme.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • convert object
          Hide convert attributes Show convert attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • type string Required

            Values are integer, long, double, float, boolean, ip, string, or auto.

        • csv object
          Hide csv attributes Show csv attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Value used to fill empty fields. Empty fields are skipped if this is not provided. An empty field is one with no value (2 consecutive separators) or empty quotes ("").

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • quote string

            Quote used in CSV, has to be single character string.

          • Separator used in CSV, has to be single character string.

          • target_fields string | array[string] Required
          • trim boolean

            Trim whitespaces in unquoted fields.

        • date object
          Hide date attributes Show date attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • formats array[string] Required

            An array of the expected date formats. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • locale string

            The locale to use when parsing the date, relevant when parsing month names or week days. Supports template snippets.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • timezone string

            The timezone to use when parsing the date. Supports template snippets.

          • The format to use when writing the date to target_field. Must be a valid java time pattern.

        • Hide date_index_name attributes Show date_index_name attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • date_formats array[string]

            An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • date_rounding string Required

            How to round the date when formatting the date into the index name. Valid values are: y (year), M (month), w (week), d (day), h (hour), m (minute) and s (second). Supports template snippets.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • The format to be used when printing the parsed date into the index name. A valid java time pattern is expected here. Supports template snippets.

          • A prefix of the index name to be prepended before the printed date. Supports template snippets.

          • locale string

            The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.

          • timezone string

            The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.

        • dissect object
          Hide dissect attributes Show dissect attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The character(s) that separate the appended fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to apply to the field.

        • Hide dot_expander attributes Show dot_expander attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • override boolean

            Controls the behavior when there is already an existing nested object that conflicts with the expanded field. When false, the processor will merge conflicts by combining the old and the new values into an array. When true, the value from the expanded field will overwrite the existing value.

          • path string

            The field that contains the field to expand. Only required if the field to expand is part another object field, because the field option can only understand leaf fields.

        • drop object
          Hide drop attributes Show drop attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • enrich object
          Hide enrich attributes Show enrich attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The maximum number of matched documents to include under the configured target field. The target_field will be turned into a json array if max_matches is higher than 1, otherwise target_field will become a json object. In order to avoid documents getting too large, the maximum allowed value is 128.

          • override boolean

            If processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • policy_name string Required

            The name of the enrich policy to use.

          • Values are intersects, disjoint, within, or contains.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • fail object
          Hide fail attributes Show fail attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • message string Required

            The error message thrown by the processor. Supports template snippets.

        • Hide fingerprint attributes Show fingerprint attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • fields string | array[string] Required
          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • salt string

            Salt value for the hash function.

          • method string

            Values are MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.

          • If true, the processor ignores any missing fields. If all fields are missing, the processor silently exits without modifying the document.

        • foreach object
          Hide foreach attributes Show foreach attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true, the processor silently exits without changing the document if the field is null or missing.

          • processor object Required
        • Hide ip_location attributes Show ip_location attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found IP location data will be returned, even if the field contains an array.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the IP location lookup.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • geo_grid object
          Hide geo_grid attributes Show geo_grid attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            The field to interpret as a geo-tile.= The field format is determined by the tile_type.

          • tile_type string Required

            Values are geotile, geohex, or geohash.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Values are geojson or wkt.

        • geoip object
          Hide geoip attributes Show geoip attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found geoip data will be returned, even if the field contains an array.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the geoip lookup.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • grok object
          Hide grok attributes Show grok attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Must be disabled or v1. If v1, the processor uses patterns with Elastic Common Schema (ECS) field names.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition.

          • patterns array[string] Required

            An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches.

          • When true, _ingest._grok_match_index will be inserted into your matched document’s metadata with the index into the pattern found in patterns that matched.

        • gsub object
          Hide gsub attributes Show gsub attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to be replaced.

          • replacement string Required

            The string to replace the matching patterns with.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide html_strip attributes Show html_strip attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document,

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide inference attributes Show inference attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • model_id string Required
          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Maps the document field names to the known field names of the model. This mapping takes precedence over any default mappings provided in the model configuration.

          • If true and any of the input fields defined in input_ouput are missing then those missing fields are quietly ignored, otherwise a missing field causes a failure. Only applies when using input_output configurations to explicitly list the input fields.

        • join object
          Hide join attributes Show join attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • separator string Required

            The separator character.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • json object
          Hide json attributes Show json attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Flag that forces the parsed JSON to be added at the top level of the document. target_field must not be set when this option is chosen.

          • Values are replace or merge.

          • When set to true, the JSON parser will not fail if the JSON contains duplicate keys. Instead, the last encountered value for any duplicate key wins.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • kv object
          Hide kv attributes Show kv attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • exclude_keys array[string]

            List of keys to exclude from document.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field_split string Required

            Regex pattern to use for splitting key-value pairs.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • include_keys array[string]

            List of keys to filter and insert into document. Defaults to including all keys.

          • prefix string

            Prefix to be added to extracted keys.

          • If true. strip brackets (), <>, [] as well as quotes ' and " from extracted values.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • trim_key string

            String of characters to trim from extracted keys.

          • String of characters to trim from extracted values.

          • value_split string Required

            Regex pattern to use for splitting the key from the value within a key-value pair.

        • Hide lowercase attributes Show lowercase attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide network_direction attributes Show network_direction attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • internal_networks array[string]

            List of internal networks. Supports IPv4 and IPv6 addresses and ranges in CIDR notation. Also supports the named ranges listed below. These may be constructed with template snippets. Must specify only one of internal_networks or internal_networks_field.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • pipeline object
          Hide pipeline attributes Show pipeline attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • name string Required
          • Whether to ignore missing pipelines instead of failing.

        • redact object
          Hide redact attributes Show redact attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • patterns array[string] Required

            A list of grok expressions to match and redact named captures with

          • prefix string

            Start a redacted section with this token

          • suffix string

            End a redacted section with this token

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • If true and the current license does not support running redact processors, then the processor quietly exits without modifying the document

          • If true then ingest metadata _ingest._redact._is_redacted is set to true if the document has been redacted

        • Hide registered_domain attributes Show registered_domain attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • remove object
          Hide remove attributes Show remove attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string | array[string] Required
          • keep string | array[string]
          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • rename object
          Hide rename attributes Show rename attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • reroute object
          Hide reroute attributes Show reroute attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • A static value for the target. Can’t be set when the dataset or namespace option is set.

        • script object
          Hide script attributes Show script attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • id string
          • params object

            Object containing parameters for the script.

        • set object
          Hide set attributes Show set attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and value is a template snippet that evaluates to null or the empty string, the processor quietly exits without modifying the document.

          • The media type for encoding value. Applies only when value is a template snippet. Must be one of application/json, text/plain, or application/x-www-form-urlencoded.

          • override boolean

            If true processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • value object

            The value to be set for the field. Supports template snippets. May specify only one of value or copy_from.

        • Hide set_security_user attributes Show set_security_user attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what user related properties are added to the field.

        • sort object
          Hide sort attributes Show sort attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • order string

            Values are asc or desc.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • split object
          Hide split attributes Show split attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Preserves empty trailing fields, if any.

          • separator string Required

            A regex which matches the separator, for example, , or \s+.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide terminate attributes Show terminate attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • trim object
          Hide trim attributes Show trim attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide uppercase attributes Show uppercase attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide urldecode attributes Show urldecode attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide uri_parts attributes Show uri_parts attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • If true, the processor copies the unparsed URI to <target_field>.original.

          • If true, the processor removes the field after parsing the URI string. If parsing fails, the processor does not remove the field.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Hide user_agent attributes Show user_agent attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object
          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The name of the file in the config/ingest-user-agent directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-user-agent will use the regexes.yaml from uap-core it ships with.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what properties are added to target_field.

            Values are name, os, device, original, or version.

          • Extracts device type from the user agent string on a best-effort basis.

      • version number
      • deprecated boolean

        Marks this ingest pipeline as deprecated. When a deprecated ingest pipeline is referenced as the default or final pipeline when creating or updating a non-deprecated index template, Elasticsearch will emit a deprecation warning.

      • _meta object
        Hide _meta attribute Show _meta attribute object
        • * object Additional properties

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • docs array[object] Required
      Hide docs attribute Show docs attribute object
      • doc object
        Hide doc attributes Show doc attributes object
        • _id string Required
        • _index string Required
        • _source object Required

          JSON body for the document.

          Hide _source attribute Show _source attribute object
          • * object Additional properties
        • _version number | string Required

          Some APIs will return values such as numbers also as a string (notably epoch timestamps). This behavior is used to capture this behavior while keeping the semantics of the field type.

          Depending on the target language, code generators can keep the union or remove it and leniently parse strings to the target type.

        • executed_pipelines array[string] Required

          A list of the names of the pipelines executed on this document.

        • ignored_fields array[object]

          A list of the fields that would be ignored at the indexing step. For example, a field whose value is larger than the allowed limit would make it through all of the pipelines, but would not be indexed into Elasticsearch.

          Hide ignored_fields attribute Show ignored_fields attribute object
          • * string Additional properties
        • error object
          Hide error attributes Show error attributes object
GET /_ingest/_simulate
curl \
 --request GET 'http://api.example.com/_ingest/_simulate' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '"{\n  \"docs\": [\n    {\n      \"_id\": 123,\n      \"_index\": \"my-index\",\n      \"_source\": {\n        \"foo\": \"bar\"\n      }\n    },\n    {\n      \"_id\": 456,\n      \"_index\": \"my-index\",\n      \"_source\": {\n        \"foo\": \"rab\"\n      }\n    }\n  ]\n}"'
In this example the index `my-index` has a default pipeline called `my-pipeline` and a final pipeline called `my-final-pipeline`. Since both documents are being ingested into `my-index`, both pipelines are run using the pipeline definitions that are already in the system.
{
  "docs": [
    {
      "_id": 123,
      "_index": "my-index",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_id": 456,
      "_index": "my-index",
      "_source": {
        "foo": "rab"
      }
    }
  ]
}
In this example the index `my-index` has a default pipeline called `my-pipeline` and a final pipeline called `my-final-pipeline`. But a substitute definition of `my-pipeline` is provided in `pipeline_substitutions`. The substitute `my-pipeline` will be used in place of the `my-pipeline` that is in the system, and then the `my-final-pipeline` that is already defined in the system will run.
{
  "docs": [
    {
      "_index": "my-index",
      "_id": 123,
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_index": "my-index",
      "_id": 456,
      "_source": {
        "foo": "rab"
      }
    }
  ],
  "pipeline_substitutions": {
    "my-pipeline": {
      "processors": [
        {
          "uppercase": {
            "field": "foo"
          }
        }
      ]
    }
  }
}
In this example, imagine that the index `my-index` has a strict mapping with only the `foo` keyword field defined. Say that field mapping came from a component template named `my-mappings-template`. You want to test adding a new field, `bar`. So a substitute definition of `my-mappings-template` is provided in `component_template_substitutions`. The substitute `my-mappings-template` will be used in place of the existing mapping for `my-index` and in place of the `my-mappings-template` that is in the system.
{
  "docs": [
    {
      "_index": "my-index",
      "_id": "123",
      "_source": {
        "foo": "foo"
      }
    },
    {
      "_index": "my-index",
      "_id": "456",
      "_source": {
        "bar": "rab"
      }
    }
  ],
  "component_template_substitutions": {
    "my-mappings_template": {
      "template": {
        "mappings": {
          "dynamic": "strict",
          "properties": {
            "foo": {
              "type": "keyword"
            },
            "bar": {
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}
The pipeline, component template, and index template substitutions replace the existing pipeline details for the duration of this request.
{
  "docs": [
    {
      "_id": "id",
      "_index": "my-index",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_id": "id",
      "_index": "my-index",
      "_source": {
        "foo": "rab"
      }
    }
  ],
  "pipeline_substitutions": {
    "my-pipeline": {
      "processors": [
        {
          "set": {
            "field": "field3",
            "value": "value3"
          }
        }
      ]
    }
  },
  "component_template_substitutions": {
    "my-component-template": {
      "template": {
        "mappings": {
          "dynamic": true,
          "properties": {
            "field3": {
              "type": "keyword"
            }
          }
        },
        "settings": {
          "index": {
            "default_pipeline": "my-pipeline"
          }
        }
      }
    }
  },
  "index_template_substitutions": {
    "my-index-template": {
      "index_patterns": [
        "my-index-*"
      ],
      "composed_of": [
        "component_template_1",
        "component_template_2"
      ]
    }
  },
  "mapping_addition": {
    "dynamic": "strict",
    "properties": {
      "foo": {
        "type": "keyword"
      }
    }
  }
}
A successful response when the simulation uses pipeline definitions that are already in the system.
{
  "docs": [
    {
      "doc": null,
      "_id": 123,
      "_index": "my-index",
      "_version": -3,
      "_source": {
        "field1": "value1",
        "field2": "value2",
        "foo": "bar"
      },
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    },
    {
      "doc": null,
      "_id": 456,
      "_index": "my-index",
      "_version": "-3,",
      "_source": {
        "field1": "value1",
        "field2": "value2",
        "foo": "rab"
      },
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    }
  ]
}
A successful response when the simulation uses pipeline substitutions.
{
  "docs": [
    {
      "doc": null,
      "_id": 123,
      "_index": "my-index",
      "_version": -3,
      "_source": {
        "field2": "value2",
        "foo": "BAR"
      },
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    },
    {
      "doc": null,
      "_id": 456,
      "_index": "my-index",
      "_version": -3,
      "_source": {
        "field2": "value2",
        "foo": "RAB"
      },
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    }
  ]
}
A successful response when the simulation uses pipeline substitutions.
{
  "docs": [
    {
      "doc": {
        "_id": "123",
        "_index": "my-index",
        "_version": -3,
        "_source": {
          "foo": "foo"
        },
        "executed_pipelines": []
      }
    },
    {
      "doc": {
        "_id": "456",
        "_index": "my-index",
        "_version": -3,
        "_source": {
          "bar": "rab"
        },
      "executed_pipelines": []
      }
    }
  ]
}
























































































































Delete a datafeed Added in 5.4.0

DELETE /_ml/datafeeds/{datafeed_id}

Path parameters

  • datafeed_id string Required

    A numerical character string that uniquely identifies the datafeed. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters.

Query parameters

  • force boolean

    Use to forcefully delete a started datafeed; this method is quicker than stopping and deleting the datafeed.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • acknowledged boolean Required

      For a successful response, this value is always true. On failure, an exception is returned instead.

DELETE /_ml/datafeeds/{datafeed_id}
curl \
 --request DELETE 'http://api.example.com/_ml/datafeeds/{datafeed_id}' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response when deleting a datafeed.
{
  "acknowledged": true
}
































































































































































































Preview a datafeed Added in 5.4.0

GET /_ml/datafeeds/_preview

This API returns the first "page" of search results from a datafeed. You can preview an existing datafeed or provide configuration details for a datafeed and anomaly detection job in the API. The preview shows the structure of the data that will be passed to the anomaly detection engine. IMPORTANT: When Elasticsearch security features are enabled, the preview uses the credentials of the user that called the API. However, when the datafeed starts it uses the roles of the last user that created or updated the datafeed. To get a preview that accurately reflects the behavior of the datafeed, use the appropriate credentials. You can also use secondary authorization headers to supply the credentials.

Query parameters

  • start string | number

    The start time from where the datafeed preview should begin

  • end string | number

    The end time when the datafeed preview should stop

application/json

Body

  • Hide datafeed_config attributes Show datafeed_config attributes object
    • If set, the datafeed performs aggregation searches. Support for aggregations is limited and should be used only with low cardinality data.

    • Hide chunking_config attributes Show chunking_config attributes object
      • mode string Required

        Values are auto, manual, or off.

      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • Hide delayed_data_check_config attributes Show delayed_data_check_config attributes object
      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • enabled boolean Required

        Specifies whether the datafeed periodically checks for delayed data.

    • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • indices string | array[string]
    • Hide indices_options attributes Show indices_options attributes object
      • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.

      • expand_wildcards string | array[string]
      • If true, missing or closed indices are not included in the response.

      • If true, concrete, expanded or aliased indices are ignored when frozen.

    • job_id string
    • If a real-time datafeed has never seen any data (including during any initial training period) then it will automatically stop itself and close its associated job after this many real-time searches that return no documents. In other words, it will stop after frequency times max_empty_searches of real-time operation. If not set then a datafeed with no end time that sees no data will remain started until it is explicitly stopped.

    • query object

      An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

      External documentation
    • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • Hide runtime_mappings attribute Show runtime_mappings attribute object
      • * object Additional properties
        Hide * attributes Show * attributes object
        • fields object

          For type composite

          Hide fields attribute Show fields attribute object
          • * object Additional properties
            Hide * attribute Show * attribute object
            • type string Required

              Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

        • fetch_fields array[object]

          For type lookup

          Hide fetch_fields attributes Show fetch_fields attributes object
          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • format string
        • format string

          A custom format for date type runtime fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • script object
          Hide script attributes Show script attributes object
          • source string | object

            One of:
          • id string
          • params object

            Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

            Hide params attribute Show params attribute object
            • * object Additional properties
          • lang string

            Any of:

            Values are painless, expression, mustache, or java.

          • options object
            Hide options attribute Show options attribute object
            • * string Additional properties
        • type string Required

          Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

    • Specifies scripts that evaluate custom expressions and returns script fields to the datafeed. The detector configuration objects in a job can contain functions that use these script fields.

      Hide script_fields attribute Show script_fields attribute object
      • * object Additional properties
        Hide * attributes Show * attributes object
        • script object Required
          Hide script attributes Show script attributes object
          • source string | object

            One of:
          • id string
          • params object

            Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

            Hide params attribute Show params attribute object
            • * object Additional properties
          • lang string

            Any of:

            Values are painless, expression, mustache, or java.

          • options object
            Hide options attribute Show options attribute object
            • * string Additional properties
    • The size parameter that is used in Elasticsearch searches when the datafeed does not use aggregations. The maximum value is the value of index.max_result_window, which is 10,000 by default.

  • Hide job_config attributes Show job_config attributes object
    • Advanced configuration option. Specifies whether this job can open when there is insufficient machine learning node capacity for it to be immediately assigned to a node.

    • analysis_config object Required
      Hide analysis_config attributes Show analysis_config attributes object
      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • categorization_analyzer string | object

        One of:
      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • If categorization_field_name is specified, you can also define optional filters. This property expects an array of regular expressions. The expressions are used to filter out matching sequences from the categorization field values. You can use this functionality to fine tune the categorization by excluding sequences from consideration when categories are defined. For example, you can exclude SQL statements that appear in your log files. This property cannot be used at the same time as categorization_analyzer. If you only want to define simple regular expression filters that are applied prior to tokenization, setting this property is the easiest method. If you also want to customize the tokenizer or post-tokenization filtering, use the categorization_analyzer property instead and include the filters as pattern_replace character filters. The effect is exactly the same.

      • detectors array[object] Required

        Detector configuration objects specify which data fields a job analyzes. They also specify which analytical functions are used. You can specify multiple detectors for a job. If the detectors array does not contain at least one detector, no analysis can occur and an error is returned.

        Hide detectors attributes Show detectors attributes object
        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • custom_rules array[object]

          Custom rules enable you to customize the way detectors operate. For example, a rule may dictate conditions under which results should be skipped. Kibana refers to custom rules as job rules.

          Hide custom_rules attributes Show custom_rules attributes object
          • actions array[string]

            The set of actions to be triggered when the rule applies. If more than one action is specified the effects of all actions are combined.

            Supported values include:

            • skip_result: The result will not be created. Unless you also specify skip_model_update, the model will be updated as usual with the corresponding series value.
            • skip_model_update: The value for that series will not be used to update the model. Unless you also specify skip_result, the results will be created as usual. This action is suitable when certain values are expected to be consistently anomalous and they affect the model in a way that negatively impacts the rest of the results.

            Values are skip_result or skip_model_update.

          • conditions array[object]

            An array of numeric conditions when the rule applies. A rule must either have a non-empty scope or at least one condition. Multiple conditions are combined together with a logical AND.

          • scope object

            A scope of series where the rule applies. A rule must either have a non-empty scope or at least one condition. By default, the scope includes all series. Scoping is allowed for any of the fields that are also specified in by_field_name, over_field_name, or partition_field_name.

        • A description of the detector.

        • A unique identifier for the detector. This identifier is based on the order of the detectors in the analysis_config, starting at zero. If you specify a value for this property, it is ignored.

        • Values are all, none, by, or over.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • function string

          The analysis function that is used. For example, count, rare, mean, min, max, or sum.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • use_null boolean

          Defines whether a new series is used as the null series when there is no value for the by or partition fields.

      • influencers array[string]

        A comma separated list of influencer field names. Typically these can be the by, over, or partition fields that are used in the detector configuration. You might also want to use a field name that is not specifically named in a detector, but is available as part of the input data. When you use multiple detectors, the use of influencers is recommended as it aggregates results for each influencer entity.

      • latency string

        A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • This functionality is reserved for internal use. It is not supported for use in customer environments and is not subject to the support SLA of official GA features. If set to true, the analysis will automatically find correlations between metrics for a given by field value and report anomalies when those correlations cease to hold. For example, suppose CPU and memory usage on host A is usually highly correlated with the same metrics on host B. Perhaps this correlation occurs because they are running a load-balanced application. If you enable this property, anomalies will be reported when, for example, CPU usage on host A is high and the value of CPU usage on host B is low. That is to say, you’ll see an anomaly when the CPU of host A is unusual given the CPU of host B. To use the multivariate_by_fields property, you must also specify by_field_name in your detector.

      • Hide per_partition_categorization attributes Show per_partition_categorization attributes object
        • enabled boolean

          To enable this setting, you must also set the partition_field_name property to the same value in every detector that uses the keyword mlcategory. Otherwise, job creation fails.

        • This setting can be set to true only if per-partition categorization is enabled. If true, both categorization and subsequent anomaly detection stops for partitions where the categorization status changes to warn. This setting makes it viable to have a job where it is expected that categorization works well for some partitions but not others; you do not pay the cost of bad categorization forever in the partitions where it works badly.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • Hide analysis_limits attributes Show analysis_limits attributes object
    • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • Custom metadata about the job

    • Advanced configuration option, which affects the automatic removal of old model snapshots for this job. It specifies a period of time (in days) after which only the first snapshot per day is retained. This period is relative to the timestamp of the most recent snapshot for this job.

    • data_description object Required
      Hide data_description attributes Show data_description attributes object
      • format string

        Only JSON format is supported at this time.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • The time format, which can be epoch, epoch_ms, or a custom pattern. The value epoch refers to UNIX or Epoch time (the number of seconds since 1 Jan 1970). The value epoch_ms indicates that time is measured in milliseconds since the epoch. The epoch and epoch_ms time formats accept either integer or real values. Custom patterns must conform to the Java DateTimeFormatter class. When you use date-time formatting patterns, it is recommended that you provide the full date, time and time zone. For example: yyyy-MM-dd'T'HH:mm:ssX. If the pattern that you specify is not sufficient to produce a complete timestamp, job creation fails.

    • Hide datafeed_config attributes Show datafeed_config attributes object
      • If set, the datafeed performs aggregation searches. Support for aggregations is limited and should be used only with low cardinality data.

      • Hide chunking_config attributes Show chunking_config attributes object
        • mode string Required

          Values are auto, manual, or off.

        • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • Hide delayed_data_check_config attributes Show delayed_data_check_config attributes object
        • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • enabled boolean Required

          Specifies whether the datafeed periodically checks for delayed data.

      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • indices string | array[string]
      • Hide indices_options attributes Show indices_options attributes object
        • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.

        • expand_wildcards string | array[string]
        • If true, missing or closed indices are not included in the response.

        • If true, concrete, expanded or aliased indices are ignored when frozen.

      • job_id string
      • If a real-time datafeed has never seen any data (including during any initial training period) then it will automatically stop itself and close its associated job after this many real-time searches that return no documents. In other words, it will stop after frequency times max_empty_searches of real-time operation. If not set then a datafeed with no end time that sees no data will remain started until it is explicitly stopped.

      • query object

        An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

        External documentation
      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • Hide runtime_mappings attribute Show runtime_mappings attribute object
        • * object Additional properties
          Hide * attributes Show * attributes object
          • fields object

            For type composite

            Hide fields attribute Show fields attribute object
            • * object Additional properties
              Hide * attribute Show * attribute object
              • type string Required

                Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

          • fetch_fields array[object]

            For type lookup

            Hide fetch_fields attributes Show fetch_fields attributes object
            • field string Required

              Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

            • format string
          • format string

            A custom format for date type runtime fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • script object
            Hide script attributes Show script attributes object
          • type string Required

            Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

      • Specifies scripts that evaluate custom expressions and returns script fields to the datafeed. The detector configuration objects in a job can contain functions that use these script fields.

        Hide script_fields attribute Show script_fields attribute object
      • The size parameter that is used in Elasticsearch searches when the datafeed does not use aggregations. The maximum value is the value of index.max_result_window, which is 10,000 by default.

    • A description of the job.

    • groups array[string]

      A list of job groups. A job can belong to no groups or many.

    • job_id string
    • job_type string

      Reserved for future use, currently set to anomaly_detector.

    • Hide model_plot_config attributes Show model_plot_config attributes object
      • If true, enables calculation and storage of the model change annotations for each entity that is being analyzed.

      • enabled boolean

        If true, enables calculation and storage of the model bounds for each entity that is being analyzed.

      • terms string

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • Advanced configuration option, which affects the automatic removal of old model snapshots for this job. It specifies the maximum period of time (in days) that snapshots are retained. This period is relative to the timestamp of the most recent snapshot for this job. The default value is 10, which means snapshots ten days older than the newest snapshot are deleted.

    • Advanced configuration option. The period over which adjustments to the score are applied, as new data is seen. The default value is the longer of 30 days or 100 bucket_spans.

    • Advanced configuration option. The period of time (in days) that results are retained. Age is calculated relative to the timestamp of the latest bucket result. If this property has a non-null value, once per day at 00:30 (server time), results that are the specified number of days older than the latest bucket result are deleted from Elasticsearch. The default value is null, which means all results are retained. Annotations generated by the system also count as results for retention purposes; they are deleted after the same number of days as results. Annotations added by users are retained forever.

Responses

GET /_ml/datafeeds/_preview
curl \
 --request GET 'http://api.example.com/_ml/datafeeds/_preview' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '{"datafeed_config":{"aggregations":{},"chunking_config":{"mode":"auto","time_span":"string"},"datafeed_id":"string","delayed_data_check_config":{"check_window":"string","enabled":true},"frequency":"string","indices":"string","indices_options":{"allow_no_indices":true,"expand_wildcards":"string","ignore_unavailable":true,"ignore_throttled":true},"job_id":"string","max_empty_searches":42.0,"query":{},"query_delay":"string","runtime_mappings":{"additionalProperty1":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"":"painless","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"},"additionalProperty2":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"":"painless","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"}},"script_fields":{"additionalProperty1":{"script":{"":"painless","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true},"additionalProperty2":{"script":{"":"painless","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true}},"scroll_size":42.0},"job_config":{"allow_lazy_open":true,"analysis_config":{"bucket_span":"string","":"string","categorization_field_name":"string","categorization_filters":["string"],"detectors":[{"by_field_name":"string","custom_rules":[{"actions":["skip_result"],"conditions":[{}],"scope":{}}],"detector_description":"string","detector_index":42.0,"exclude_frequent":"all","field_name":"string","function":"string","over_field_name":"string","partition_field_name":"string","use_null":true}],"influencers":["string"],"latency":"string","model_prune_window":"string","multivariate_by_fields":true,"per_partition_categorization":{"enabled":true,"stop_on_warn":true},"summary_count_field_name":"string"},"analysis_limits":{"categorization_examples_limit":42.0,"":42.0},"background_persist_interval":"string","custom_settings":{},"daily_model_snapshot_retention_after_days":42.0,"data_description":{"format":"string","time_field":"string","time_format":"string","field_delimiter":"string"},"datafeed_config":{"aggregations":{},"chunking_config":{"mode":"auto","time_span":"string"},"datafeed_id":"string","delayed_data_check_config":{"check_window":"string","enabled":true},"frequency":"string","indices":"string","indices_options":{"allow_no_indices":true,"expand_wildcards":"string","ignore_unavailable":true,"ignore_throttled":true},"job_id":"string","max_empty_searches":42.0,"query":{},"query_delay":"string","runtime_mappings":{"additionalProperty1":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"":"painless","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"},"additionalProperty2":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"":"painless","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"}},"script_fields":{"additionalProperty1":{"script":{"":"painless","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true},"additionalProperty2":{"script":{"":"painless","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true}},"scroll_size":42.0},"description":"string","groups":["string"],"job_id":"string","job_type":"string","model_plot_config":{"annotations_enabled":true,"enabled":true,"terms":"string"},"model_snapshot_retention_days":42.0,"renormalization_window_days":42.0,"results_index_name":"string","results_retention_days":42.0}}'


















































































































































































































































































































Get a script or search template

GET /_scripts/{id}

Retrieves a stored script or search template.

Path parameters

  • id string Required

    The identifier for the stored script or search template.

Query parameters

  • The period to wait for the master node. If the master node is not available before the timeout expires, the request fails and returns an error. It can also be set to -1 to indicate that the request should never timeout.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • _id string Required
    • found boolean Required
    • script object
      Hide script attributes Show script attributes object
      • lang string Required

        Any of:

        Values are painless, expression, mustache, or java.

      • options object
        Hide options attribute Show options attribute object
        • * string Additional properties
      • source string | object Required

        One of:
GET /_scripts/{id}
curl \
 --request GET 'http://api.example.com/_scripts/{id}' \
 --header "Authorization: $API_KEY"

























































Run a scrolling search

GET /_search/scroll

IMPORTANT: The scroll API is no longer recommend for deep pagination. If you need to preserve the index state while paging through more than 10,000 hits, use the search_after parameter with a point in time (PIT).

The scroll API gets large sets of results from a single scrolling search request. To get the necessary scroll ID, submit a search API request that includes an argument for the scroll query parameter. The scroll parameter indicates how long Elasticsearch should retain the search context for the request. The search response returns a scroll ID in the _scroll_id response body parameter. You can then use the scroll ID with the scroll API to retrieve the next batch of results for the request. If the Elasticsearch security features are enabled, the access to the results of a specific scroll ID is restricted to the user or API key that submitted the search.

You can also use the scroll API to specify a new scroll parameter that extends or shortens the retention period for the search context.

IMPORTANT: Results from a scrolling search reflect the state of the index at the time of the initial search request. Subsequent indexing or document changes only affect later search and scroll requests.

External documentation

Query parameters

  • scroll string

    The period to retain the search context for scrolling.

  • scroll_id string Deprecated

    The scroll ID for scrolled search

  • If true, the API response’s hit.total property is returned as an integer. If false, the API response’s hit.total property is returned as an object.

application/json

Body

  • scroll string

    A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

  • scroll_id string Required

Responses

GET /_search/scroll
curl \
 --request GET 'http://api.example.com/_search/scroll' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '"{\n  \"scroll_id\" : \"DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ==\"\n}"'
Request example
Run `GET /_search/scroll` to get the next batch of results for a scrolling search.
{
  "scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ=="
}




































Count search results

POST /{index}/_count

Get the number of documents matching a query.

The query can be provided either by using a simple query string as a parameter, or by defining Query DSL within the request body. The query is optional. When no query is provided, the API uses match_all to count all the documents.

The count API supports multi-target syntax. You can run a single count API search across multiple data streams and indices.

The operation is broadcast across all shards. For each shard ID group, a replica is chosen and the search is run against it. This means that replicas increase the scalability of the count.

Path parameters

  • index string | array[string] Required

    A comma-separated list of data streams, indices, and aliases to search. It supports wildcards (*). To search all data streams and indices, omit this parameter or use * or _all.

Query parameters

  • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.

  • analyzer string

    The analyzer to use for the query string. This parameter can be used only when the q query string parameter is specified.

  • If true, wildcard and prefix queries are analyzed. This parameter can be used only when the q query string parameter is specified.

  • The default operator for query string query: AND or OR. This parameter can be used only when the q query string parameter is specified.

    Values are and, AND, or, or OR.

  • df string

    The field to use as a default when no field prefix is given in the query string. This parameter can be used only when the q query string parameter is specified.

  • expand_wildcards string | array[string]

    The type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. It supports comma-separated values, such as open,hidden.

    Supported values include:

    • all: Match any data stream or index, including hidden ones.
    • open: Match open, non-hidden indices. Also matches any non-hidden data stream.
    • closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.
    • hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.
    • none: Wildcard expressions are not accepted.
  • ignore_throttled boolean Deprecated

    If true, concrete, expanded, or aliased indices are ignored when frozen.

  • If false, the request returns an error if it targets a missing or closed index.

  • lenient boolean

    If true, format-based query failures (such as providing text to a numeric field) in the query string will be ignored. This parameter can be used only when the q query string parameter is specified.

  • The minimum _score value that documents must have to be included in the result.

  • The node or shard the operation should be performed on. By default, it is random.

  • routing string

    A custom value used to route operations to a specific shard.

  • The maximum number of documents to collect for each shard. If a query reaches this limit, Elasticsearch terminates the query early. Elasticsearch collects documents before sorting.

    IMPORTANT: Use with caution. Elasticsearch applies this parameter to each shard handling the request. When possible, let Elasticsearch perform early termination automatically. Avoid specifying this parameter for requests that target data streams with backing indices across multiple data tiers.

  • q string

    The query in Lucene query string syntax. This parameter cannot be used with a request body.

application/json

Body

Responses

POST /{index}/_count
curl \
 --request POST 'http://api.example.com/{index}/_count' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '"{\n  \"query\" : {\n    \"term\" : { \"user.id\" : \"kimchy\" }\n  }\n}"'
Request example
Run `GET /my-index-000001/_count?q=user:kimchy`. Alternatively, run `GET /my-index-000001/_count` with the same query in the request body. Both requests count the number of documents in `my-index-000001` with a `user.id` of `kimchy`.
{
  "query" : {
    "term" : { "user.id" : "kimchy" }
  }
}
Response examples (200)
A successful response from `GET /my-index-000001/_count?q=user:kimchy`.
{
  "count": 1,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  }
}



















































































































































































































































































Get API key information Added in 6.7.0

GET /_security/api_key

Retrieves information for one or more API keys. NOTE: If you have only the manage_own_api_key privilege, this API returns only the API keys that you own. If you have read_security, manage_api_key or greater privileges (including manage_security), this API returns all API keys regardless of ownership.

Query parameters

  • id string

    An API key id. This parameter cannot be used with any of name, realm_name or username.

  • name string

    An API key name. This parameter cannot be used with any of id, realm_name or username. It supports prefix search with wildcard.

  • owner boolean

    A boolean flag that can be used to query API keys owned by the currently authenticated user. The realm_name or username parameters cannot be specified when this parameter is set to true as they are assumed to be the currently authenticated ones.

  • The name of an authentication realm. This parameter cannot be used with either id or name or when owner flag is set to true.

  • username string

    The username of a user. This parameter cannot be used with either id or name or when owner flag is set to true.

  • Return the snapshot of the owner user's role descriptors associated with the API key. An API key's actual permission is the intersection of its assigned role descriptors and the owner user's role descriptors.

  • A boolean flag that can be used to query API keys that are currently active. An API key is considered active if it is neither invalidated, nor expired at query time. You can specify this together with other parameters such as owner or name. If active_only is false, the response will include both active and inactive (expired or invalidated) keys.

  • Determines whether to also retrieve the profile uid, for the API key owner principal, if it exists.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • api_keys array[object] Required
      Hide api_keys attributes Show api_keys attributes object
      • id string Required
      • name string Required
      • type string Required

        Values are rest or cross_cluster.

      • creation number

        Time unit for milliseconds

      • Time unit for milliseconds

      • invalidated boolean Required

        Invalidation status for the API key. If the key has been invalidated, it has a value of true. Otherwise, it is false.

      • Time unit for milliseconds

      • username string Required
      • realm string Required

        Realm name of the principal for which this API key was created.

      • Realm type of the principal for which this API key was created

      • metadata object Required
        Hide metadata attribute Show metadata attribute object
        • * object Additional properties
      • The role descriptors assigned to this API key when it was created or last updated. An empty role descriptor means the API key inherits the owner user’s permissions.

        Hide role_descriptors attribute Show role_descriptors attribute object
        • * object Additional properties
          Hide * attributes Show * attributes object
          • cluster array[string]

            A list of cluster privileges. These privileges define the cluster level actions that API keys are able to execute.

          • indices array[object]

            A list of indices permissions entries.

          • remote_indices array[object]

            A list of indices permissions for remote clusters.

          • remote_cluster array[object]

            A list of cluster permissions for remote clusters. NOTE: This is limited a subset of the cluster permissions.

          • global array[object] | object

            An object defining global privileges. A global privilege is a form of cluster privilege that is request-aware. Support for global privileges is currently limited to the management of application privileges.

          • applications array[object]

            A list of application privilege entries

          • metadata object
            Hide metadata attribute Show metadata attribute object
            • * object Additional properties
          • run_as array[string]

            A list of users that the API keys can impersonate. NOTE: In Elastic Cloud Serverless, the run-as feature is disabled. For API compatibility, you can still specify an empty run_as field, but a non-empty list will be rejected.

          • Optional description of the role descriptor

          • Hide restriction attribute Show restriction attribute object
            • workflows array[string] Required

              A list of workflows to which the API key is restricted. NOTE: In order to use a role restriction, an API key must be created with a single role descriptor.

          • Hide transient_metadata attribute Show transient_metadata attribute object
            • * object Additional properties
      • limited_by array[object]

        The owner user’s permissions associated with the API key. It is a point-in-time snapshot captured at creation and subsequent updates. An API key’s effective permissions are an intersection of its assigned privileges and the owner user’s permissions.

        Hide limited_by attribute Show limited_by attribute object
        • * object Additional properties
          Hide * attributes Show * attributes object
          • cluster array[string]

            A list of cluster privileges. These privileges define the cluster level actions that API keys are able to execute.

          • indices array[object]

            A list of indices permissions entries.

          • remote_indices array[object]

            A list of indices permissions for remote clusters.

          • remote_cluster array[object]

            A list of cluster permissions for remote clusters. NOTE: This is limited a subset of the cluster permissions.

          • applications array[object]

            A list of application privilege entries

          • metadata object
          • run_as array[string]

            A list of users that the API keys can impersonate. NOTE: In Elastic Cloud Serverless, the run-as feature is disabled. For API compatibility, you can still specify an empty run_as field, but a non-empty list will be rejected.

          • Optional description of the role descriptor

      • access object
        Hide access attributes Show access attributes object
        • replication array[object]

          A list of indices permission entries for cross-cluster replication.

          Hide replication attributes Show replication attributes object
      • The profile uid for the API key owner principal, if requested and if it exists

      • _sort array[number | string | boolean | null]

        A field value.

GET /_security/api_key
curl \
 --request GET 'http://api.example.com/_security/api_key' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response from `GET /_security/api_key?id=VuaCfGcBCdbkQm-e5aOx&with_limited_by=true`.
{
  "api_keys": [ 
    {
      "id": "VuaCfGcBCdbkQm-e5aOx", 
      "name": "my-api-key", 
      "creation": 1548550550158, 
      "expiration": 1548551550158, 
      "invalidated": false, 
      "username": "myuser", 
      "realm": "native1", 
      "realm_type": "native",
      "metadata": { 
        "application": "myapp"
      },
      "role_descriptors": { }, 
      "limited_by": [  
        {
          "role-power-user": {
            "cluster": [
              "monitor"
            ],
            "indices": [
              {
                "names": [
                  "*"
                ],
                "privileges": [
                  "read"
                ],
                "allow_restricted_indices": false
              }
            ],
            "applications": [ ],
            "run_as": [ ],
            "metadata": { },
            "transient_metadata": {
              "enabled": true
            }
          }
        }
      ]
    }
  ]
}
A successful response from `GET /_security/api_key?username=myuser&realm_name=native1`. The response contains all API keys for the user `myuser` in the `native1` realm.
{
  "api_keys": [
    {
      "id": "0GF5GXsBCXxz2eDxWwFN",
      "name": "hadoop_myuser_key",
      "creation": 1548550550158,
      "expiration": 1548551550158,
      "invalidated": false,
      "username": "myuser",
      "realm": "native1",
      "realm_type": "native",
      "metadata": {
        "application": "myapp"
      },
      "role_descriptors": {
        "role-a": {
          "cluster": [
            "monitor"
          ],
          "indices": [
            {
              "names": [
                "index-a"
              ],
              "privileges": [
                "read"
              ],
              "allow_restricted_indices": false
            }
          ],
          "applications": [ ],
          "run_as": [ ],
          "metadata": { },
          "transient_metadata": {
            "enabled": true
          }
        }
      }
    },
    {
      "id": "6wHJmcQpReKBa42EHV5SBw",
      "name": "api-key-name-2",
      "creation": 1548550550158,
      "invalidated": false,
      "username": "user-y",
      "realm": "realm-2",
      "metadata": {},
      "role_descriptors": { }
    }
  ]
}












































































Get users

GET /_security/user/{username}

Get information about users in the native realm and built-in users.

Path parameters

  • username string | array[string] Required

    An identifier for the user. You can specify multiple usernames as a comma-separated list. If you omit this parameter, the API retrieves information about all users.

Query parameters

  • Determines whether to retrieve the user profile UID, if it exists, for the users.

Responses

GET /_security/user/{username}
curl \
 --request GET 'http://api.example.com/_security/user/{username}' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response from `GET /_security/user/jacknich?with_profile_uid=true`. It includes the user `profile_uid` as part of the response.
{
  "jacknich": {
    "username": "jacknich",
    "roles": [
      "admin", "other_role1"
    ],
    "full_name": "Jack Nicholson",
    "email": "jacknich@example.com",
    "metadata": { "intelligence" : 7 },
    "enabled": true,
    "profile_uid": "u_79HkWkwmnBH5gqFKwoxggWPjEBOur1zLPXQPEl1VBW0_0"
  }
}