Get behavioral analytics collections Deprecated Technical preview

GET /_application/analytics
Api key auth Basic auth Bearer auth

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • * object Additional properties
      Hide * attribute Show * attribute object
      • event_data_stream object Required
        Hide event_data_stream attribute Show event_data_stream attribute object
GET /_application/analytics 
curl \
 --request GET 'http://api.example.com/_application/analytics' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response from `GET _application/analytics/my*` 
{
  "my_analytics_collection": {
      "event_data_stream": {
          "name": "behavioral_analytics-events-my_analytics_collection"
      }
  },
  "my_analytics_collection2": {
      "event_data_stream": {
          "name": "behavioral_analytics-events-my_analytics_collection2"
      }
  }
}

Get transform information Added in 7.7.0

GET /_cat/transforms/{transform_id}
Api key auth Basic auth Bearer auth

Get configuration and usage information about transforms.

CAT APIs are only intended for human consumption using the Kibana console or command line. They are not intended for use by applications. For application consumption, use the get transform statistics API.

Path parameters

  • transform_id string Required

    A transform identifier or a wildcard expression. If you do not specify one of these options, the API returns information for all transforms.

Query parameters

  • allow_no_match boolean

    Specifies what to do when the request: contains wildcard expressions and there are no transforms that match; contains the _all string or no identifiers and there are no matches; contains wildcard expressions and there are only partial matches. If true, it returns an empty transforms array when there are no matches and the subset of results when there are partial matches. If false, the request returns a 404 status code when there are no matches or only partial matches.

  • from number

    Skips the specified number of transforms.

  • h string | array[string]

    Comma-separated list of column names to display.

    Supported values include:

    • changes_last_detection_time (or cldt): The timestamp when changes were last detected in the source indices.
    • checkpoint (or cp): The sequence number for the checkpoint.
    • checkpoint_duration_time_exp_avg (or cdtea, checkpointTimeExpAvg): Exponential moving average of the duration of the checkpoint, in milliseconds.
    • checkpoint_progress (or c, checkpointProgress): The progress of the next checkpoint that is currently in progress.
    • create_time (or ct, createTime): The time the transform was created.
    • delete_time (or dtime): The amount of time spent deleting, in milliseconds.
    • description (or d): The description of the transform.
    • dest_index (or di, destIndex): The destination index for the transform. The mappings of the destination index are deduced based on the source fields when possible. If alternate mappings are required, use the Create index API prior to starting the transform.
    • documents_deleted (or docd): The number of documents that have been deleted from the destination index due to the retention policy for this transform.
    • documents_indexed (or doci): The number of documents that have been indexed into the destination index for the transform.
    • docs_per_second (or dps): Specifies a limit on the number of input documents per second. This setting throttles the transform by adding a wait time between search requests. The default value is null, which disables throttling.
    • documents_processed (or docp): The number of documents that have been processed from the source index of the transform.
    • frequency (or f): The interval between checks for changes in the source indices when the transform is running continuously. Also determines the retry interval in the event of transient failures while the transform is searching or indexing. The minimum value is 1s and the maximum is 1h. The default value is 1m.
    • id: Identifier for the transform.
    • index_failure (or if): The number of indexing failures.
    • index_time (or itime): The amount of time spent indexing, in milliseconds.
    • index_total (or it): The number of index operations.
    • indexed_documents_exp_avg (or idea): Exponential moving average of the number of new documents that have been indexed.
    • last_search_time (or lst, lastSearchTime): The timestamp of the last search in the source indices. This field is only shown if the transform is running.
    • max_page_search_size (or mpsz): Defines the initial page size to use for the composite aggregation for each checkpoint. If circuit breaker exceptions occur, the page size is dynamically adjusted to a lower value. The minimum value is 10 and the maximum is 65,536. The default value is 500.
    • pages_processed (or pp): The number of search or bulk index operations processed. Documents are processed in batches instead of individually.
    • pipeline (or p): The unique identifier for an ingest pipeline.
    • processed_documents_exp_avg (or pdea): Exponential moving average of the number of documents that have been processed.
    • processing_time (or pt): The amount of time spent processing results, in milliseconds.
    • reason (or r): If a transform has a failed state, this property provides details about the reason for the failure.
    • search_failure (or sf): The number of search failures.
    • search_time (or stime): The amount of time spent searching, in milliseconds.
    • search_total (or st): The number of search operations on the source index for the transform.
    • source_index (or si, sourceIndex): The source indices for the transform. It can be a single index, an index pattern (for example, "my-index-*"), an array of indices (for example, ["my-index-000001", "my-index-000002"]), or an array of index patterns (for example, ["my-index-*", "my-other-index-*"]. For remote indices use the syntax "remote_name:index_name". If any indices are in remote clusters then the master node and at least one transform node must have the remote_cluster_client node role.

    • state (or s): The status of the transform, which can be one of the following values:

      • aborting: The transform is aborting.
      • failed: The transform failed. For more information about the failure, check the reason field.
      • indexing: The transform is actively processing data and creating new documents.
      • started: The transform is running but not actively indexing data.
      • stopped: The transform is stopped.
      • stopping: The transform is stopping.

    • transform_type (or tt): Indicates the type of transform: batch or continuous.

    • trigger_count (or tc): The number of times the transform has been triggered by the scheduler. For example, the scheduler triggers the transform indexer to check for updates or ingest new data at an interval specified in the frequency property.

    • version (or v): The version of Elasticsearch that existed on the node when the transform was created.

  • s string | array[string]

    Comma-separated list of column names or column aliases used to sort the response.

    Supported values include:

    • changes_last_detection_time (or cldt): The timestamp when changes were last detected in the source indices.
    • checkpoint (or cp): The sequence number for the checkpoint.
    • checkpoint_duration_time_exp_avg (or cdtea, checkpointTimeExpAvg): Exponential moving average of the duration of the checkpoint, in milliseconds.
    • checkpoint_progress (or c, checkpointProgress): The progress of the next checkpoint that is currently in progress.
    • create_time (or ct, createTime): The time the transform was created.
    • delete_time (or dtime): The amount of time spent deleting, in milliseconds.
    • description (or d): The description of the transform.
    • dest_index (or di, destIndex): The destination index for the transform. The mappings of the destination index are deduced based on the source fields when possible. If alternate mappings are required, use the Create index API prior to starting the transform.
    • documents_deleted (or docd): The number of documents that have been deleted from the destination index due to the retention policy for this transform.
    • documents_indexed (or doci): The number of documents that have been indexed into the destination index for the transform.
    • docs_per_second (or dps): Specifies a limit on the number of input documents per second. This setting throttles the transform by adding a wait time between search requests. The default value is null, which disables throttling.
    • documents_processed (or docp): The number of documents that have been processed from the source index of the transform.
    • frequency (or f): The interval between checks for changes in the source indices when the transform is running continuously. Also determines the retry interval in the event of transient failures while the transform is searching or indexing. The minimum value is 1s and the maximum is 1h. The default value is 1m.
    • id: Identifier for the transform.
    • index_failure (or if): The number of indexing failures.
    • index_time (or itime): The amount of time spent indexing, in milliseconds.
    • index_total (or it): The number of index operations.
    • indexed_documents_exp_avg (or idea): Exponential moving average of the number of new documents that have been indexed.
    • last_search_time (or lst, lastSearchTime): The timestamp of the last search in the source indices. This field is only shown if the transform is running.
    • max_page_search_size (or mpsz): Defines the initial page size to use for the composite aggregation for each checkpoint. If circuit breaker exceptions occur, the page size is dynamically adjusted to a lower value. The minimum value is 10 and the maximum is 65,536. The default value is 500.
    • pages_processed (or pp): The number of search or bulk index operations processed. Documents are processed in batches instead of individually.
    • pipeline (or p): The unique identifier for an ingest pipeline.
    • processed_documents_exp_avg (or pdea): Exponential moving average of the number of documents that have been processed.
    • processing_time (or pt): The amount of time spent processing results, in milliseconds.
    • reason (or r): If a transform has a failed state, this property provides details about the reason for the failure.
    • search_failure (or sf): The number of search failures.
    • search_time (or stime): The amount of time spent searching, in milliseconds.
    • search_total (or st): The number of search operations on the source index for the transform.
    • source_index (or si, sourceIndex): The source indices for the transform. It can be a single index, an index pattern (for example, "my-index-*"), an array of indices (for example, ["my-index-000001", "my-index-000002"]), or an array of index patterns (for example, ["my-index-*", "my-other-index-*"]. For remote indices use the syntax "remote_name:index_name". If any indices are in remote clusters then the master node and at least one transform node must have the remote_cluster_client node role.

    • state (or s): The status of the transform, which can be one of the following values:

      • aborting: The transform is aborting.
      • failed: The transform failed. For more information about the failure, check the reason field.
      • indexing: The transform is actively processing data and creating new documents.
      • started: The transform is running but not actively indexing data.
      • stopped: The transform is stopped.
      • stopping: The transform is stopping.

    • transform_type (or tt): Indicates the type of transform: batch or continuous.

    • trigger_count (or tc): The number of times the transform has been triggered by the scheduler. For example, the scheduler triggers the transform indexer to check for updates or ingest new data at an interval specified in the frequency property.

    • version (or v): The version of Elasticsearch that existed on the node when the transform was created.

  • time string

    The unit used to display time values.

    Values are nanos, micros, ms, s, m, h, or d.

  • size number

    The maximum number of transforms to obtain.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • id string
    • state string

      The status of the transform. Returned values include: aborting: The transform is aborting. failed: The transform failed. For more information about the failure, check thereasonfield. indexing: The transform is actively processing data and creating new documents. started: The transform is running but not actively indexing data. stopped: The transform is stopped. stopping`: The transform is stopping.

    • checkpoint string

      The sequence number for the checkpoint.

    • documents_processed string

      The number of documents that have been processed from the source index of the transform.

    • checkpoint_progress string | null

      The progress of the next checkpoint that is currently in progress.

      One of:
      string-1 string string-2 string | null

    • last_search_time string | null

      The timestamp of the last search in the source indices. This field is shown only if the transform is running.

      One of:
      string-1 string string-2 string | null

    • changes_last_detection_time string | null

      The timestamp when changes were last detected in the source indices.

      One of:
      string-1 string string-2 string | null
    • create_time string

      The time the transform was created.

    • version string
    • source_index string

      The source indices for the transform.

    • dest_index string

      The destination index for the transform.

    • pipeline string

      The unique identifier for the ingest pipeline.

    • description string

      The description of the transform.

    • transform_type string

      The type of transform: batch or continuous.

    • frequency string

      The interval between checks for changes in the source indices when the transform is running continuously.

    • max_page_search_size string

      The initial page size that is used for the composite aggregation for each checkpoint.

    • docs_per_second string

      The number of input documents per second.

    • reason string

      If a transform has a failed state, these details describe the reason for failure.

    • search_total string

      The total number of search operations on the source index for the transform.

    • search_failure string

      The total number of search failures.

    • search_time string

      The total amount of search time, in milliseconds.

    • index_total string

      The total number of index operations done by the transform.

    • index_failure string

      The total number of indexing failures.

    • index_time string

      The total time spent indexing documents, in milliseconds.

    • documents_indexed string

      The number of documents that have been indexed into the destination index for the transform.

    • delete_time string

      The total time spent deleting documents, in milliseconds.

    • documents_deleted string

      The number of documents deleted from the destination index due to the retention policy for the transform.

    • trigger_count string

      The number of times the transform has been triggered by the scheduler. For example, the scheduler triggers the transform indexer to check for updates or ingest new data at an interval specified in the frequency property.

    • pages_processed string

      The number of search or bulk index operations processed. Documents are processed in batches instead of individually.

    • processing_time string

      The total time spent processing results, in milliseconds.

    • checkpoint_duration_time_exp_avg string

      The exponential moving average of the duration of the checkpoint, in milliseconds.

    • indexed_documents_exp_avg string

      The exponential moving average of the number of new documents that have been indexed.

    • processed_documents_exp_avg string

      The exponential moving average of the number of documents that have been processed.
GET /_cat/transforms/{transform_id} 
curl \
 --request GET 'http://api.example.com/_cat/transforms/{transform_id}' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response from `GET /_cat/transforms?v=true&format=json`. 
[
  {
    "id" : "ecommerce_transform",
    "state" : "started",
    "checkpoint" : "1",
    "documents_processed" : "705",
    "checkpoint_progress" : "100.00",
    "changes_last_detection_time" : null
  }
]

Get remote cluster information Added in 6.1.0

GET /_remote/info
Api key auth Basic auth Bearer auth

Get information about configured remote clusters. The API returns connection and endpoint information keyed by the configured remote cluster alias.


This API returns information that reflects current state on the local cluster. The connected field does not necessarily reflect whether a remote cluster is down or unavailable, only whether there is currently an open connection to it. Elasticsearch does not spontaneously try to reconnect to a disconnected remote cluster. To trigger a reconnection, attempt a cross-cluster search, ES|QL cross-cluster search, or try the resolve cluster endpoint.

External documentation

Responses

GET /_remote/info 
curl \
 --request GET 'http://api.example.com/_remote/info' \
 --header "Authorization: $API_KEY"

Get node statistics

GET /_nodes/stats/{metric}/{index_metric}
Api key auth Basic auth Bearer auth

Get statistics for nodes in a cluster. By default, all stats are returned. You can limit the returned information by using metrics.

Path parameters

  • metric string | array[string] Required

    Limit the information returned to the specified metrics

  • index_metric string | array[string] Required

    Limit the information returned for indices metric to the specific index metrics. It can be used only if indices (or all) metric is specified.

Query parameters

  • completion_fields string | array[string]

    Comma-separated list or wildcard expressions of fields to include in fielddata and suggest statistics.

  • fielddata_fields string | array[string]

    Comma-separated list or wildcard expressions of fields to include in fielddata statistics.

  • fields string | array[string]

    Comma-separated list or wildcard expressions of fields to include in the statistics.

  • groups boolean

    Comma-separated list of search groups to include in the search statistics.

  • include_segment_file_sizes boolean

    If true, the call reports the aggregated disk usage of each one of the Lucene index files (only applies if segment stats are requested).

  • level string

    Indicates whether statistics are aggregated at the cluster, index, or shard level.

    Values are cluster, indices, or shards.

  • timeout string

    Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

  • types array[string]

    A comma-separated list of document types for the indexing index metric.

  • include_unloaded_segments boolean

    If true, the response includes information from segments that are not loaded into memory.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • _nodes object
      Hide _nodes attributes Show _nodes attributes object
      • failures array[object]
        Hide failures attributes Show failures attributes object
      • total number Required

        Total number of nodes selected by the request.

      • successful number Required

        Number of nodes that responded successfully to the request.

      • failed number Required

        Number of nodes that rejected the request or failed to respond. If this value is not 0, a reason for the rejection or failure is included in the response.

    • cluster_name string
    • nodes object Required
      Hide nodes attribute Show nodes attribute object
      • * object Additional properties
        Hide * attributes Show * attributes object
        • adaptive_selection object

          Statistics about adaptive replica selection.

          Hide adaptive_selection attribute Show adaptive_selection attribute object
          • * object Additional properties
            Hide * attributes Show * attributes object
            • avg_queue_size number

              The exponentially weighted moving average queue size of search requests on the keyed node.

            • avg_response_time string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • avg_response_time_ns number

              The exponentially weighted moving average response time, in nanoseconds, of search requests on the keyed node.

            • avg_service_time string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • avg_service_time_ns number

              The exponentially weighted moving average service time, in nanoseconds, of search requests on the keyed node.

            • outgoing_searches number

              The number of outstanding search requests to the keyed node from the node these stats are for.

            • rank string

              The rank of this node; used for shard selection when routing search requests.

        • breakers object

          Statistics about the field data circuit breaker.

          Hide breakers attribute Show breakers attribute object
          • * object Additional properties
            Hide * attributes Show * attributes object
            • estimated_size string

              Estimated memory used for the operation.

            • estimated_size_in_bytes number

              Estimated memory used, in bytes, for the operation.

            • limit_size string

              Memory limit for the circuit breaker.

            • limit_size_in_bytes number

              Memory limit, in bytes, for the circuit breaker.

            • overhead number

              A constant that all estimates for the circuit breaker are multiplied with to calculate a final estimate.

            • tripped number

              Total number of times the circuit breaker has been triggered and prevented an out of memory error.

        • fs object
          Hide fs attributes Show fs attributes object
          • data array[object]

            List of all file stores.

          • timestamp number

            Last time the file stores statistics were refreshed. Recorded in milliseconds since the Unix Epoch.

          • total object
            Hide total attributes Show total attributes object
            • available string

              Total disk space available to this Java virtual machine on all file stores. Depending on OS or process level restrictions, this might appear less than free. This is the actual amount of free disk space the Elasticsearch node can utilise.

            • available_in_bytes number

              Total number of bytes available to this Java virtual machine on all file stores. Depending on OS or process level restrictions, this might appear less than free_in_bytes. This is the actual amount of free disk space the Elasticsearch node can utilise.

            • free string

              Total unallocated disk space in all file stores.

            • free_in_bytes number

              Total number of unallocated bytes in all file stores.

            • total string

              Total size of all file stores.

            • total_in_bytes number

              Total size of all file stores in bytes.

          • io_stats object
            Hide io_stats attributes Show io_stats attributes object
            • devices array[object]

              Array of disk metrics for each device that is backing an Elasticsearch data path. These disk metrics are probed periodically and averages between the last probe and the current probe are computed.

            • total object
        • host string
        • http object
          Hide http attributes Show http attributes object
          • current_open number

            Current number of open HTTP connections for the node.

          • total_opened number

            Total number of HTTP connections opened for the node.

          • clients array[object]

            Information on current and recently-closed HTTP client connections. Clients that have been closed longer than the http.client_stats.closed_channels.max_age setting will not be represented here.

          • routes object Required Added in 8.12.0

            Detailed HTTP stats broken down by route

            Hide routes attribute Show routes attribute object
            • * object Additional properties
        • ingest object
          Hide ingest attributes Show ingest attributes object
          • pipelines object

            Contains statistics about ingest pipelines for the node.

            Hide pipelines attribute Show pipelines attribute object
            • * object Additional properties
          • total object
            Hide total attributes Show total attributes object
            • count number Required

              Total number of documents ingested during the lifetime of this node.

            • current number Required

              Total number of documents currently being ingested.

            • failed number Required

              Total number of failed ingest operations during the lifetime of this node.

        • ip string | array[string]

          IP address and port for the node.

          One of:
          Ip string array-2 array[string]
        • jvm object
          Hide jvm attributes Show jvm attributes object
          • buffer_pools object

            Contains statistics about JVM buffer pools for the node.

            Hide buffer_pools attribute Show buffer_pools attribute object
            • * object Additional properties
          • classes object
            Hide classes attributes Show classes attributes object
          • gc object
            Hide gc attribute Show gc attribute object
            • collectors object

              Contains statistics about JVM garbage collectors for the node.

          • mem object
            Hide mem attributes Show mem attributes object
          • threads object
            Hide threads attributes Show threads attributes object
            • count number

              Number of active threads in use by JVM.

            • peak_count number

              Highest number of threads used by JVM.

          • timestamp number

            Last time JVM statistics were refreshed.

          • uptime string

            Human-readable JVM uptime. Only returned if the human query parameter is true.

          • uptime_in_millis number

            JVM uptime in milliseconds.

        • name string
        • os object
          Hide os attributes Show os attributes object
          • cpu object
            Hide cpu attributes Show cpu attributes object
            • percent number
            • sys string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • total string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • user string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • load_average object
          • swap object
            Hide swap attributes Show swap attributes object
          • cgroup object
            Hide cgroup attributes Show cgroup attributes object
          • timestamp number
        • process object
          Hide process attributes Show process attributes object
          • cpu object
            Hide cpu attributes Show cpu attributes object
            • percent number
            • sys string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • total string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • user string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • load_average object
          • mem object
            Hide mem attributes Show mem attributes object
          • open_file_descriptors number

            Number of opened file descriptors associated with the current or -1 if not supported.

          • max_file_descriptors number

            Maximum number of file descriptors allowed on the system, or -1 if not supported.

          • timestamp number

            Last time the statistics were refreshed. Recorded in milliseconds since the Unix Epoch.

        • roles array[string]
          • @doc_id node-roles

          Values are master, data, data_cold, data_content, data_frozen, data_hot, data_warm, client, ingest, ml, voting_only, transform, remote_cluster_client, or coordinating_only.

        • script object
          Hide script attributes Show script attributes object
          • cache_evictions number

            Total number of times the script cache has evicted old data.

          • compilations number

            Total number of inline script compilations performed by the node.

          • compilations_history object

            Contains this recent history of script compilations.

            Hide compilations_history attribute Show compilations_history attribute object
            • * number Additional properties
          • compilation_limit_triggered number

            Total number of times the script compilation circuit breaker has limited inline script compilations.

          • contexts array[object]
        • script_cache object
        • thread_pool object

          Statistics about each thread pool, including current size, queue and rejected tasks.

          Hide thread_pool attribute Show thread_pool attribute object
          • * object Additional properties
            Hide * attributes Show * attributes object
            • active number

              Number of active threads in the thread pool.

            • completed number

              Number of tasks completed by the thread pool executor.

            • largest number

              Highest number of active threads in the thread pool.

            • queue number

              Number of tasks in queue for the thread pool.

            • rejected number

              Number of tasks rejected by the thread pool executor.

            • threads number

              Number of threads in the thread pool.

        • timestamp number
        • transport object
          Hide transport attributes Show transport attributes object
          • inbound_handling_time_histogram array[object]

            The distribution of the time spent handling each inbound message on a transport thread, represented as a histogram.

          • outbound_handling_time_histogram array[object]

            The distribution of the time spent sending each outbound transport message on a transport thread, represented as a histogram.

          • rx_count number

            Total number of RX (receive) packets received by the node during internal cluster communication.

          • rx_size string

            Size of RX packets received by the node during internal cluster communication.

          • rx_size_in_bytes number

            Size, in bytes, of RX packets received by the node during internal cluster communication.

          • server_open number

            Current number of inbound TCP connections used for internal communication between nodes.

          • tx_count number

            Total number of TX (transmit) packets sent by the node during internal cluster communication.

          • tx_size string

            Size of TX packets sent by the node during internal cluster communication.

          • tx_size_in_bytes number

            Size, in bytes, of TX packets sent by the node during internal cluster communication.

          • total_outbound_connections number

            The cumulative number of outbound transport connections that this node has opened since it started. Each transport connection may comprise multiple TCP connections but is only counted once in this statistic. Transport connections are typically long-lived so this statistic should remain constant in a stable cluster.

        • transport_address string
        • attributes object

          Contains a list of attributes for the node.

          Hide attributes attribute Show attributes attribute object
          • * string Additional properties
        • discovery object
          Hide discovery attributes Show discovery attributes object
          • cluster_state_queue object
            Hide cluster_state_queue attributes Show cluster_state_queue attributes object
            • total number

              Total number of cluster states in queue.

            • pending number

              Number of pending cluster states in queue.

            • committed number

              Number of committed cluster states in queue.

          • published_cluster_states object
            Hide published_cluster_states attributes Show published_cluster_states attributes object
          • cluster_state_update object

            Contains low-level statistics about how long various activities took during cluster state updates while the node was the elected master. Omitted if the node is not master-eligible. Every field whose name ends in _time within this object is also represented as a raw number of milliseconds in a field whose name ends in _time_millis. The human-readable fields with a _time suffix are only returned if requested with the ?human=true query parameter.

            Hide cluster_state_update attribute Show cluster_state_update attribute object
            • * object Additional properties
          • serialized_cluster_states object
            Hide serialized_cluster_states attributes Show serialized_cluster_states attributes object
          • cluster_applier_stats object
            Hide cluster_applier_stats attribute Show cluster_applier_stats attribute object
        • indexing_pressure object
          Hide indexing_pressure attribute Show indexing_pressure attribute object
          • memory object
            Hide memory attributes Show memory attributes object
        • indices object
          Hide indices attributes Show indices attributes object
GET /_nodes/stats/{metric}/{index_metric} 
curl \
 --request GET 'http://api.example.com/_nodes/stats/{metric}/{index_metric}' \
 --header "Authorization: $API_KEY"

Get node statistics

GET /_nodes/{node_id}/stats/{metric}/{index_metric}
Api key auth Basic auth Bearer auth

Get statistics for nodes in a cluster. By default, all stats are returned. You can limit the returned information by using metrics.

Path parameters

  • node_id string | array[string] Required

    Comma-separated list of node IDs or names used to limit returned information.

  • metric string | array[string] Required

    Limit the information returned to the specified metrics

  • index_metric string | array[string] Required

    Limit the information returned for indices metric to the specific index metrics. It can be used only if indices (or all) metric is specified.

Query parameters

  • completion_fields string | array[string]

    Comma-separated list or wildcard expressions of fields to include in fielddata and suggest statistics.

  • fielddata_fields string | array[string]

    Comma-separated list or wildcard expressions of fields to include in fielddata statistics.

  • fields string | array[string]

    Comma-separated list or wildcard expressions of fields to include in the statistics.

  • groups boolean

    Comma-separated list of search groups to include in the search statistics.

  • include_segment_file_sizes boolean

    If true, the call reports the aggregated disk usage of each one of the Lucene index files (only applies if segment stats are requested).

  • level string

    Indicates whether statistics are aggregated at the cluster, index, or shard level.

    Values are cluster, indices, or shards.

  • timeout string

    Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

  • types array[string]

    A comma-separated list of document types for the indexing index metric.

  • include_unloaded_segments boolean

    If true, the response includes information from segments that are not loaded into memory.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • _nodes object
      Hide _nodes attributes Show _nodes attributes object
      • failures array[object]
        Hide failures attributes Show failures attributes object
      • total number Required

        Total number of nodes selected by the request.

      • successful number Required

        Number of nodes that responded successfully to the request.

      • failed number Required

        Number of nodes that rejected the request or failed to respond. If this value is not 0, a reason for the rejection or failure is included in the response.

    • cluster_name string
    • nodes object Required
      Hide nodes attribute Show nodes attribute object
      • * object Additional properties
        Hide * attributes Show * attributes object
        • adaptive_selection object

          Statistics about adaptive replica selection.

          Hide adaptive_selection attribute Show adaptive_selection attribute object
          • * object Additional properties
            Hide * attributes Show * attributes object
            • avg_queue_size number

              The exponentially weighted moving average queue size of search requests on the keyed node.

            • avg_response_time string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • avg_response_time_ns number

              The exponentially weighted moving average response time, in nanoseconds, of search requests on the keyed node.

            • avg_service_time string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • avg_service_time_ns number

              The exponentially weighted moving average service time, in nanoseconds, of search requests on the keyed node.

            • outgoing_searches number

              The number of outstanding search requests to the keyed node from the node these stats are for.

            • rank string

              The rank of this node; used for shard selection when routing search requests.

        • breakers object

          Statistics about the field data circuit breaker.

          Hide breakers attribute Show breakers attribute object
          • * object Additional properties
            Hide * attributes Show * attributes object
            • estimated_size string

              Estimated memory used for the operation.

            • estimated_size_in_bytes number

              Estimated memory used, in bytes, for the operation.

            • limit_size string

              Memory limit for the circuit breaker.

            • limit_size_in_bytes number

              Memory limit, in bytes, for the circuit breaker.

            • overhead number

              A constant that all estimates for the circuit breaker are multiplied with to calculate a final estimate.

            • tripped number

              Total number of times the circuit breaker has been triggered and prevented an out of memory error.

        • fs object
          Hide fs attributes Show fs attributes object
          • data array[object]

            List of all file stores.

          • timestamp number

            Last time the file stores statistics were refreshed. Recorded in milliseconds since the Unix Epoch.

          • total object
            Hide total attributes Show total attributes object
            • available string

              Total disk space available to this Java virtual machine on all file stores. Depending on OS or process level restrictions, this might appear less than free. This is the actual amount of free disk space the Elasticsearch node can utilise.

            • available_in_bytes number

              Total number of bytes available to this Java virtual machine on all file stores. Depending on OS or process level restrictions, this might appear less than free_in_bytes. This is the actual amount of free disk space the Elasticsearch node can utilise.

            • free string

              Total unallocated disk space in all file stores.

            • free_in_bytes number

              Total number of unallocated bytes in all file stores.

            • total string

              Total size of all file stores.

            • total_in_bytes number

              Total size of all file stores in bytes.

          • io_stats object
            Hide io_stats attributes Show io_stats attributes object
            • devices array[object]

              Array of disk metrics for each device that is backing an Elasticsearch data path. These disk metrics are probed periodically and averages between the last probe and the current probe are computed.

            • total object
        • host string
        • http object
          Hide http attributes Show http attributes object
          • current_open number

            Current number of open HTTP connections for the node.

          • total_opened number

            Total number of HTTP connections opened for the node.

          • clients array[object]

            Information on current and recently-closed HTTP client connections. Clients that have been closed longer than the http.client_stats.closed_channels.max_age setting will not be represented here.

          • routes object Required Added in 8.12.0

            Detailed HTTP stats broken down by route

            Hide routes attribute Show routes attribute object
            • * object Additional properties
        • ingest object
          Hide ingest attributes Show ingest attributes object
          • pipelines object

            Contains statistics about ingest pipelines for the node.

            Hide pipelines attribute Show pipelines attribute object
            • * object Additional properties
          • total object
            Hide total attributes Show total attributes object
            • count number Required

              Total number of documents ingested during the lifetime of this node.

            • current number Required

              Total number of documents currently being ingested.

            • failed number Required

              Total number of failed ingest operations during the lifetime of this node.

        • ip string | array[string]

          IP address and port for the node.

          One of:
          Ip string array-2 array[string]
        • jvm object
          Hide jvm attributes Show jvm attributes object
          • buffer_pools object

            Contains statistics about JVM buffer pools for the node.

            Hide buffer_pools attribute Show buffer_pools attribute object
            • * object Additional properties
          • classes object
            Hide classes attributes Show classes attributes object
          • gc object
            Hide gc attribute Show gc attribute object
            • collectors object

              Contains statistics about JVM garbage collectors for the node.

          • mem object
            Hide mem attributes Show mem attributes object
          • threads object
            Hide threads attributes Show threads attributes object
            • count number

              Number of active threads in use by JVM.

            • peak_count number

              Highest number of threads used by JVM.

          • timestamp number

            Last time JVM statistics were refreshed.

          • uptime string

            Human-readable JVM uptime. Only returned if the human query parameter is true.

          • uptime_in_millis number

            JVM uptime in milliseconds.

        • name string
        • os object
          Hide os attributes Show os attributes object
          • cpu object
            Hide cpu attributes Show cpu attributes object
            • percent number
            • sys string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • total string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • user string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • load_average object
          • swap object
            Hide swap attributes Show swap attributes object
          • cgroup object
            Hide cgroup attributes Show cgroup attributes object
          • timestamp number
        • process object
          Hide process attributes Show process attributes object
          • cpu object
            Hide cpu attributes Show cpu attributes object
            • percent number
            • sys string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • total string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • user string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • load_average object
          • mem object
            Hide mem attributes Show mem attributes object
          • open_file_descriptors number

            Number of opened file descriptors associated with the current or -1 if not supported.

          • max_file_descriptors number

            Maximum number of file descriptors allowed on the system, or -1 if not supported.

          • timestamp number

            Last time the statistics were refreshed. Recorded in milliseconds since the Unix Epoch.

        • roles array[string]
          • @doc_id node-roles

          Values are master, data, data_cold, data_content, data_frozen, data_hot, data_warm, client, ingest, ml, voting_only, transform, remote_cluster_client, or coordinating_only.

        • script object
          Hide script attributes Show script attributes object
          • cache_evictions number

            Total number of times the script cache has evicted old data.

          • compilations number

            Total number of inline script compilations performed by the node.

          • compilations_history object

            Contains this recent history of script compilations.

            Hide compilations_history attribute Show compilations_history attribute object
            • * number Additional properties
          • compilation_limit_triggered number

            Total number of times the script compilation circuit breaker has limited inline script compilations.

          • contexts array[object]
        • script_cache object
        • thread_pool object

          Statistics about each thread pool, including current size, queue and rejected tasks.

          Hide thread_pool attribute Show thread_pool attribute object
          • * object Additional properties
            Hide * attributes Show * attributes object
            • active number

              Number of active threads in the thread pool.

            • completed number

              Number of tasks completed by the thread pool executor.

            • largest number

              Highest number of active threads in the thread pool.

            • queue number

              Number of tasks in queue for the thread pool.

            • rejected number

              Number of tasks rejected by the thread pool executor.

            • threads number

              Number of threads in the thread pool.

        • timestamp number
        • transport object
          Hide transport attributes Show transport attributes object
          • inbound_handling_time_histogram array[object]

            The distribution of the time spent handling each inbound message on a transport thread, represented as a histogram.

          • outbound_handling_time_histogram array[object]

            The distribution of the time spent sending each outbound transport message on a transport thread, represented as a histogram.

          • rx_count number

            Total number of RX (receive) packets received by the node during internal cluster communication.

          • rx_size string

            Size of RX packets received by the node during internal cluster communication.

          • rx_size_in_bytes number

            Size, in bytes, of RX packets received by the node during internal cluster communication.

          • server_open number

            Current number of inbound TCP connections used for internal communication between nodes.

          • tx_count number

            Total number of TX (transmit) packets sent by the node during internal cluster communication.

          • tx_size string

            Size of TX packets sent by the node during internal cluster communication.

          • tx_size_in_bytes number

            Size, in bytes, of TX packets sent by the node during internal cluster communication.

          • total_outbound_connections number

            The cumulative number of outbound transport connections that this node has opened since it started. Each transport connection may comprise multiple TCP connections but is only counted once in this statistic. Transport connections are typically long-lived so this statistic should remain constant in a stable cluster.

        • transport_address string
        • attributes object

          Contains a list of attributes for the node.

          Hide attributes attribute Show attributes attribute object
          • * string Additional properties
        • discovery object
          Hide discovery attributes Show discovery attributes object
          • cluster_state_queue object
            Hide cluster_state_queue attributes Show cluster_state_queue attributes object
            • total number

              Total number of cluster states in queue.

            • pending number

              Number of pending cluster states in queue.

            • committed number

              Number of committed cluster states in queue.

          • published_cluster_states object
            Hide published_cluster_states attributes Show published_cluster_states attributes object
          • cluster_state_update object

            Contains low-level statistics about how long various activities took during cluster state updates while the node was the elected master. Omitted if the node is not master-eligible. Every field whose name ends in _time within this object is also represented as a raw number of milliseconds in a field whose name ends in _time_millis. The human-readable fields with a _time suffix are only returned if requested with the ?human=true query parameter.

            Hide cluster_state_update attribute Show cluster_state_update attribute object
            • * object Additional properties
          • serialized_cluster_states object
            Hide serialized_cluster_states attributes Show serialized_cluster_states attributes object
          • cluster_applier_stats object
            Hide cluster_applier_stats attribute Show cluster_applier_stats attribute object
        • indexing_pressure object
          Hide indexing_pressure attribute Show indexing_pressure attribute object
          • memory object
            Hide memory attributes Show memory attributes object
        • indices object
          Hide indices attributes Show indices attributes object
GET /_nodes/{node_id}/stats/{metric}/{index_metric} 
curl \
 --request GET 'http://api.example.com/_nodes/{node_id}/stats/{metric}/{index_metric}' \
 --header "Authorization: $API_KEY"

Update the connector configuration Beta

PUT /_connector/{connector_id}/_configuration
Api key auth Basic auth Bearer auth

Update the configuration field in the connector document.

Path parameters

  • connector_id string Required

    The unique identifier of the connector to be updated

application/json

Body Required

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • result string Required

      Values are created, updated, deleted, not_found, or noop.
PUT /_connector/{connector_id}/_configuration 
curl \
 --request PUT 'http://api.example.com/_connector/{connector_id}/_configuration' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '"{\n    \"values\": {\n        \"tenant_id\": \"my-tenant-id\",\n        \"tenant_name\": \"my-sharepoint-site\",\n        \"client_id\": \"foo\",\n        \"secret_value\": \"bar\",\n        \"site_collections\": \"*\"\n    }\n}"'
Request examples 
{
    "values": {
        "tenant_id": "my-tenant-id",
        "tenant_name": "my-sharepoint-site",
        "client_id": "foo",
        "secret_value": "bar",
        "site_collections": "*"
    }
}
{
    "values": {
        "secret_value": "foo-bar"
    }
}
Response examples (200) 
{
  "result": "updated"
}

Update the connector name and description Beta

PUT /_connector/{connector_id}/_name
Api key auth Basic auth Bearer auth

Path parameters

  • connector_id string Required

    The unique identifier of the connector to be updated

application/json

Body Required

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • result string Required

      Values are created, updated, deleted, not_found, or noop.
PUT /_connector/{connector_id}/_name 
curl \
 --request PUT 'http://api.example.com/_connector/{connector_id}/_name' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '"{\n    \"name\": \"Custom connector\",\n    \"description\": \"This is my customized connector\"\n}"'
Request example 
{
    "name": "Custom connector",
    "description": "This is my customized connector"
}
Response examples (200) 
{
  "result": "updated"
}

Get data stream lifecycle stats Added in 8.12.0

GET /_lifecycle/stats
Api key auth Basic auth Bearer auth

Get statistics about the data streams that are managed by a data stream lifecycle.

Responses

GET /_lifecycle/stats 
curl \
 --request GET 'http://api.example.com/_lifecycle/stats' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response for `GET _lifecycle/stats?human&pretty` 
{
  "last_run_duration_in_millis": 2,
  "last_run_duration": "2ms",
  "time_between_starts_in_millis": 9998,
  "time_between_starts": "9.99s",
  "data_streams_count": 2,
  "data_streams": [
    {
      "name": "my-data-stream",
      "backing_indices_in_total": 2,
      "backing_indices_in_error": 0
    },
    {
      "name": "my-other-stream",
      "backing_indices_in_total": 2,
      "backing_indices_in_error": 1
    }
  ]
}

Graph explore

The graph explore API enables you to extract and summarize information about the documents and terms in an Elasticsearch data stream or index.

Get started with Graph

Get mapping definitions

GET /{index}/_mapping
Api key auth Basic auth Bearer auth

For data streams, the API retrieves mappings for the stream’s backing indices.

Path parameters

  • index string | array[string] Required

    Comma-separated list of data streams, indices, and aliases used to limit the request. Supports wildcards (*). To target all data streams and indices, omit this parameter or use * or _all.

Query parameters

  • allow_no_indices boolean

    If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices.

  • expand_wildcards string | array[string]

    Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as open,hidden. Valid values are: all, open, closed, hidden, none.

    Supported values include:

    • all: Match any data stream or index, including hidden ones.
    • open: Match open, non-hidden indices. Also matches any non-hidden data stream.
    • closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.
    • hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.
    • none: Wildcard expressions are not accepted.
  • ignore_unavailable boolean

    If false, the request returns an error if it targets a missing or closed index.

  • local boolean Deprecated

    If true, the request retrieves information from the local node only.

  • master_timeout string

    Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

Responses

GET /{index}/_mapping 
curl \
 --request GET 'http://api.example.com/{index}/_mapping' \
 --header "Authorization: $API_KEY"

Get index segments

GET /{index}/_segments
Api key auth Basic auth Bearer auth

Get low-level information about the Lucene segments in index shards. For data streams, the API returns information about the stream's backing indices.

Path parameters

  • index string | array[string] Required

    Comma-separated list of data streams, indices, and aliases used to limit the request. Supports wildcards (*). To target all data streams and indices, omit this parameter or use * or _all.

Query parameters

  • allow_no_indices boolean

    If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices.

  • expand_wildcards string | array[string]

    Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as open,hidden. Valid values are: all, open, closed, hidden, none.

    Supported values include:

    • all: Match any data stream or index, including hidden ones.
    • open: Match open, non-hidden indices. Also matches any non-hidden data stream.
    • closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.
    • hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.
    • none: Wildcard expressions are not accepted.
  • ignore_unavailable boolean

    If false, the request returns an error if it targets a missing or closed index.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
GET /{index}/_segments 
curl \
 --request GET 'http://api.example.com/{index}/_segments' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response for creating a new index for a data stream. 
{
  "acknowledged": true,
  "shards_acknowledged": true,
  "old_index": ".ds-my-data-stream-2099.05.06-000001",
  "new_index": ".ds-my-data-stream-2099.05.07-000002",
  "rolled_over": true,
  "dry_run": false,
  "lazy": false,
  "conditions": {
    "[max_age: 7d]": false,
    "[max_docs: 1000]": true,
    "[max_primary_shard_size: 50gb]": false,
    "[max_primary_shard_docs: 2000]": false
  }
}

Delete pipelines Added in 5.0.0

DELETE /_ingest/pipeline/{id}
Api key auth Basic auth Bearer auth

Delete one or more ingest pipelines.

External documentation

Path parameters

  • id string Required

    Pipeline ID or wildcard expression of pipeline IDs used to limit the request. To delete all ingest pipelines in a cluster, use a value of *.

Query parameters

  • master_timeout string

    Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

  • timeout string

    Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • acknowledged boolean Required

      For a successful response, this value is always true. On failure, an exception is returned instead.
DELETE /_ingest/pipeline/{id} 
curl \
 --request DELETE 'http://api.example.com/_ingest/pipeline/{id}' \
 --header "Authorization: $API_KEY"

Update the license

PUT /_license
Api key auth Basic auth Bearer auth

You can update your license at runtime without shutting down your nodes. License updates take effect immediately. If the license you are installing does not support all of the features that were available with your previous license, however, you are notified in the response. You must then re-submit the API request with the acknowledge parameter set to true.

NOTE: If Elasticsearch security features are enabled and you are installing a gold or higher license, you must enable TLS on the transport networking layer before you install the license. If the operator privileges feature is enabled, only operator users can use this API.

Query parameters

  • acknowledge boolean

    Specifies whether you acknowledge the license changes.

  • master_timeout string

    The period to wait for a connection to the master node.

  • timeout string

    The period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

application/json

Body

Responses

PUT /_license 
curl \
 --request PUT 'http://api.example.com/_license' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '"{\n  \"licenses\": [\n    {\n      \"uid\":\"893361dc-9749-4997-93cb-802e3d7fa4xx\",\n      \"type\":\"basic\",\n      \"issue_date_in_millis\":1411948800000,\n      \"expiry_date_in_millis\":1914278399999,\n      \"max_nodes\":1,\n      \"issued_to\":\"issuedTo\",\n      \"issuer\":\"issuer\",\n      \"signature\":\"xx\"\n    }\n    ]\n}"'
Request example
Run `PUT _license` to update to a basic license. NOTE: These values are invalid; you must substitute the appropriate contents from your license file. 
{
  "licenses": [
    {
      "uid":"893361dc-9749-4997-93cb-802e3d7fa4xx",
      "type":"basic",
      "issue_date_in_millis":1411948800000,
      "expiry_date_in_millis":1914278399999,
      "max_nodes":1,
      "issued_to":"issuedTo",
      "issuer":"issuer",
      "signature":"xx"
    }
    ]
}
Response examples (200)
If you update to a basic license and you previously had a license with more features, you receive this type of response. You must re-submit the API request and set the `acknowledge` parameter to `true`. 
{
  "acknowledged": false,
  "license_status": "valid",
  "acknowledge": {
    "message": "\"\"\"This license update requires acknowledgement. To acknowledge the license, please read the following messages and update the license again, this time with the \"acknowledge=true\" parameter:\"\"\"",
    "watcher": [
      "Watcher will be disabled"
    ],
    "logstash": [
      "Logstash will no longer poll for centrally-managed pipelines"
    ],
    "security": [
      "The following X-Pack security functionality will be disabled ..."
    ]
  }
}

Machine learning

Create a datafeed Added in 5.4.0

PUT /_ml/datafeeds/{datafeed_id}
Api key auth Basic auth Bearer auth

Datafeeds retrieve data from Elasticsearch for analysis by an anomaly detection job. You can associate only one datafeed with each anomaly detection job. The datafeed contains a query that runs at a defined interval (frequency). If you are concerned about delayed data, you can add a delay (query_delay') at each interval. By default, the datafeed uses the following query:{"match_all": {"boost": 1}}`.

When Elasticsearch security features are enabled, your datafeed remembers which roles the user who created it had at the time of creation and runs the query using those same roles. If you provide secondary authorization headers, those credentials are used instead. You must use Kibana, this API, or the create anomaly detection jobs API to create a datafeed. Do not add a datafeed directly to the .ml-config index. Do not give users write privileges on the .ml-config index.

Path parameters

  • datafeed_id string Required

    A numerical character string that uniquely identifies the datafeed. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters.

Query parameters

  • allow_no_indices boolean

    If true, wildcard indices expressions that resolve into no concrete indices are ignored. This includes the _all string or when no indices are specified.

  • expand_wildcards string | array[string]

    Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values.

    Supported values include:

    • all: Match any data stream or index, including hidden ones.
    • open: Match open, non-hidden indices. Also matches any non-hidden data stream.
    • closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.
    • hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.
    • none: Wildcard expressions are not accepted.
  • ignore_throttled boolean Deprecated

    If true, concrete, expanded, or aliased indices are ignored when frozen.

  • ignore_unavailable boolean

    If true, unavailable indices (missing or closed) are ignored.

application/json

Body Required

  • aggregations object

    If set, the datafeed performs aggregation searches. Support for aggregations is limited and should be used only with low cardinality data.

  • chunking_config object
    Hide chunking_config attributes Show chunking_config attributes object
    • mode string Required

      Values are auto, manual, or off.

    • time_span string

      A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

  • delayed_data_check_config object
    Hide delayed_data_check_config attributes Show delayed_data_check_config attributes object
    • check_window string

      A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • enabled boolean Required

      Specifies whether the datafeed periodically checks for delayed data.

  • frequency string

    A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

  • indices string | array[string]
  • indices_options object
    Hide indices_options attributes Show indices_options attributes object
    • allow_no_indices boolean

      If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.

    • expand_wildcards string | array[string]
    • ignore_unavailable boolean

      If true, missing or closed indices are not included in the response.

    • ignore_throttled boolean

      If true, concrete, expanded or aliased indices are ignored when frozen.

  • job_id string
  • max_empty_searches number

    If a real-time datafeed has never seen any data (including during any initial training period), it automatically stops and closes the associated job after this many real-time searches return no documents. In other words, it stops after frequency times max_empty_searches of real-time operation. If not set, a datafeed with no end time that sees no data remains started until it is explicitly stopped. By default, it is not set.

  • query object

    An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

    External documentation
  • query_delay string

    A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

  • runtime_mappings object
    Hide runtime_mappings attribute Show runtime_mappings attribute object
    • * object Additional properties
      Hide * attributes Show * attributes object
      • fields object

        For type composite

        Hide fields attribute Show fields attribute object
        • * object Additional properties
          Hide * attribute Show * attribute object
          • type string Required

            Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

      • fetch_fields array[object]

        For type lookup

        Hide fetch_fields attributes Show fetch_fields attributes object
        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • format string
      • format string

        A custom format for date type runtime fields.

      • input_field string

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • target_field string

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • target_index string
      • script object
        Hide script attributes Show script attributes object

        • source string | object

          One of:
          ScriptSource string SearchRequestBody object
        • id string
        • params object

          Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

          Hide params attribute Show params attribute object
          • * object Additional properties

        • lang string

          Any of:
          ScriptLanguage string ScriptLanguage string

          Values are painless, expression, mustache, or java.

        • options object
          Hide options attribute Show options attribute object
          • * string Additional properties
      • type string Required

        Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

  • script_fields object

    Specifies scripts that evaluate custom expressions and returns script fields to the datafeed. The detector configuration objects in a job can contain functions that use these script fields.

    Hide script_fields attribute Show script_fields attribute object
    • * object Additional properties
      Hide * attributes Show * attributes object
      • script object Required
        Hide script attributes Show script attributes object

        • source string | object

          One of:
          ScriptSource string SearchRequestBody object
        • id string
        • params object

          Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

          Hide params attribute Show params attribute object
          • * object Additional properties

        • lang string

          Any of:
          ScriptLanguage string ScriptLanguage string

          Values are painless, expression, mustache, or java.

        • options object
          Hide options attribute Show options attribute object
          • * string Additional properties
      • ignore_failure boolean
  • scroll_size number

    The size parameter that is used in Elasticsearch searches when the datafeed does not use aggregations. The maximum value is the value of index.max_result_window, which is 10,000 by default.

  • headers object

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • aggregations object
    • authorization object
      Hide authorization attributes Show authorization attributes object
      • api_key object
        Hide api_key attributes Show api_key attributes object
        • id string Required

          The identifier for the API key.

        • name string Required

          The name of the API key.

      • roles array[string]

        If a user ID was used for the most recent update to the datafeed, its roles at the time of the update are listed in the response.

      • service_account string

        If a service account was used for the most recent update to the datafeed, the account name is listed in the response.

    • chunking_config object Required
      Hide chunking_config attributes Show chunking_config attributes object
      • mode string Required

        Values are auto, manual, or off.

      • time_span string

        A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • delayed_data_check_config object
      Hide delayed_data_check_config attributes Show delayed_data_check_config attributes object
      • check_window string

        A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • enabled boolean Required

        Specifies whether the datafeed periodically checks for delayed data.

    • datafeed_id string Required
    • frequency string

      A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • indices array[string] Required
    • job_id string Required
    • indices_options object
      Hide indices_options attributes Show indices_options attributes object
      • allow_no_indices boolean

        If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.

      • expand_wildcards string | array[string]
      • ignore_unavailable boolean

        If true, missing or closed indices are not included in the response.

      • ignore_throttled boolean

        If true, concrete, expanded or aliased indices are ignored when frozen.

    • max_empty_searches number
    • query object Required

      An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

      External documentation
    • query_delay string Required

      A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • runtime_mappings object
      Hide runtime_mappings attribute Show runtime_mappings attribute object
      • * object Additional properties
        Hide * attributes Show * attributes object
        • fields object

          For type composite

          Hide fields attribute Show fields attribute object
          • * object Additional properties
            Hide * attribute Show * attribute object
            • type string Required

              Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

        • fetch_fields array[object]

          For type lookup

          Hide fetch_fields attributes Show fetch_fields attributes object
          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • format string
        • format string

          A custom format for date type runtime fields.

        • input_field string

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • target_field string

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • target_index string
        • script object
          Hide script attributes Show script attributes object

          • source string | object

            One of:
            ScriptSource string SearchRequestBody object
          • id string
          • params object

            Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

            Hide params attribute Show params attribute object
            • * object Additional properties

          • lang string

            Any of:
            ScriptLanguage string ScriptLanguage string

            Values are painless, expression, mustache, or java.

          • options object
            Hide options attribute Show options attribute object
            • * string Additional properties
        • type string Required

          Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

    • script_fields object
      Hide script_fields attribute Show script_fields attribute object
      • * object Additional properties
        Hide * attributes Show * attributes object
        • script object Required
          Hide script attributes Show script attributes object

          • source string | object

            One of:
            ScriptSource string SearchRequestBody object
          • id string
          • params object

            Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

            Hide params attribute Show params attribute object
            • * object Additional properties

          • lang string

            Any of:
            ScriptLanguage string ScriptLanguage string

            Values are painless, expression, mustache, or java.

          • options object
            Hide options attribute Show options attribute object
            • * string Additional properties
        • ignore_failure boolean
    • scroll_size number Required
PUT /_ml/datafeeds/{datafeed_id} 
curl \
 --request PUT 'http://api.example.com/_ml/datafeeds/{datafeed_id}' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '{"aggregations":{},"chunking_config":{"mode":"auto","time_span":"string"},"delayed_data_check_config":{"check_window":"string","enabled":true},"frequency":"string","indices":"string","indices_options":{"allow_no_indices":true,"expand_wildcards":"string","ignore_unavailable":true,"ignore_throttled":true},"job_id":"string","max_empty_searches":42.0,"query":{},"query_delay":"string","runtime_mappings":{"additionalProperty1":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"":"painless","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"},"additionalProperty2":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"":"painless","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"}},"script_fields":{"additionalProperty1":{"script":{"":"painless","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true},"additionalProperty2":{"script":{"":"painless","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true}},"scroll_size":42.0,"headers":{}}'

Get anomaly detection job results for categories Added in 5.4.0

POST /_ml/anomaly_detectors/{job_id}/results/categories/{category_id}
Api key auth Basic auth Bearer auth

Path parameters

  • job_id string Required

    Identifier for the anomaly detection job.

  • category_id string Required

    Identifier for the category, which is unique in the job. If you specify neither the category ID nor the partition_field_value, the API returns information about all categories. If you specify only the partition_field_value, it returns information about all categories for the specified partition.

Query parameters

  • from number

    Skips the specified number of categories.

  • partition_field_value string

    Only return categories for the specified partition.

  • size number

    Specifies the maximum number of categories to obtain.

application/json

Body

  • page object
    Hide page attributes Show page attributes object
    • from number

      Skips the specified number of items.

    • size number

      Specifies the maximum number of items to obtain.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • categories array[object] Required
      Hide categories attributes Show categories attributes object
      • category_id number Required
      • examples array[string] Required

        A list of examples of actual values that matched the category.

      • grok_pattern string
      • job_id string Required
      • max_matching_length number Required
      • partition_field_name string

        If per-partition categorization is enabled, this property identifies the field used to segment the categorization. It is not present when per-partition categorization is disabled.

      • partition_field_value string

        If per-partition categorization is enabled, this property identifies the value of the partition_field_name for the category. It is not present when per-partition categorization is disabled.

      • regex string Required

        A regular expression that is used to search for values that match the category.

      • terms string Required

        A space separated list of the common tokens that are matched in values of the category.

      • num_matches number

        The number of messages that have been matched by this category. This is only guaranteed to have the latest accurate count after a job _flush or _close

      • preferred_to_categories array[string]

        A list of category_id entries that this current category encompasses. Any new message that is processed by the categorizer will match against this category and not any of the categories in this list. This is only guaranteed to have the latest accurate list of categories after a job _flush or _close

      • p string
      • result_type string Required
      • mlcategory string Required
    • count number Required
POST /_ml/anomaly_detectors/{job_id}/results/categories/{category_id} 
curl \
 --request POST 'http://api.example.com/_ml/anomaly_detectors/{job_id}/results/categories/{category_id}' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '{"page":{"from":42.0,"size":42.0}}'

Upgrade a snapshot Added in 5.4.0

POST /_ml/anomaly_detectors/{job_id}/model_snapshots/{snapshot_id}/_upgrade
Api key auth Basic auth Bearer auth

Upgrade an anomaly detection model snapshot to the latest major version. Over time, older snapshot formats are deprecated and removed. Anomaly detection jobs support only snapshots that are from the current or previous major version. This API provides a means to upgrade a snapshot to the current major version. This aids in preparing the cluster for an upgrade to the next major version. Only one snapshot per anomaly detection job can be upgraded at a time and the upgraded snapshot cannot be the current snapshot of the anomaly detection job.

Path parameters

  • job_id string Required

    Identifier for the anomaly detection job.

  • snapshot_id string Required

    A numerical character string that uniquely identifies the model snapshot.

Query parameters

  • wait_for_completion boolean

    When true, the API won’t respond until the upgrade is complete. Otherwise, it responds as soon as the upgrade task is assigned to a node.

  • timeout string

    Controls the time to wait for the request to complete.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • node string Required
    • completed boolean Required

      When true, this means the task is complete. When false, it is still running.
POST /_ml/anomaly_detectors/{job_id}/model_snapshots/{snapshot_id}/_upgrade 
curl \
 --request POST 'http://api.example.com/_ml/anomaly_detectors/{job_id}/model_snapshots/{snapshot_id}/_upgrade' \
 --header "Authorization: $API_KEY"

Get data frame analytics job stats Added in 7.3.0

GET /_ml/data_frame/analytics/{id}/_stats
Api key auth Basic auth Bearer auth

Path parameters

  • id string Required

    Identifier for the data frame analytics job. If you do not specify this option, the API returns information for the first hundred data frame analytics jobs.

Query parameters

  • allow_no_match boolean

    Specifies what to do when the request:

    1. Contains wildcard expressions and there are no data frame analytics jobs that match.
    2. Contains the _all string or no identifiers and there are no matches.
    3. Contains wildcard expressions and there are only partial matches.

    The default value returns an empty data_frame_analytics array when there are no matches and the subset of results when there are partial matches. If this parameter is false, the request returns a 404 status code when there are no matches or only partial matches.

  • from number

    Skips the specified number of data frame analytics jobs.

  • size number

    Specifies the maximum number of data frame analytics jobs to obtain.

  • verbose boolean

    Defines whether the stats response should be verbose.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • count number Required
    • data_frame_analytics array[object] Required

      An array of objects that contain usage information for data frame analytics jobs, which are sorted by the id value in ascending order.

      Hide data_frame_analytics attributes Show data_frame_analytics attributes object
      • analysis_stats object
        Hide analysis_stats attributes Show analysis_stats attributes object
        • classification_stats object
          Hide classification_stats attributes Show classification_stats attributes object
          • hyperparameters object Required
            Hide hyperparameters attributes Show hyperparameters attributes object
            • alpha number

              Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This parameter affects loss calculations by acting as a multiplier of the tree depth. Higher alpha values result in shallower trees and faster training times. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to zero.

            • lambda number

              Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularization term which applies to leaf weights of the individual trees in the forest. A high lambda value causes training to favor small leaf weights. This behavior makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. A small lambda value results in large individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

            • gamma number

              Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. A high gamma value causes training to prefer small trees. A small gamma value results in larger individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

            • eta number

              Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, larger forests cause slower training. By default, this value is calculated during hyperparameter optimization. It must be a value between 0.001 and 1.

            • eta_growth_rate_per_tree number

              Advanced configuration option. Specifies the rate at which eta increases for each new tree that is added to the forest. For example, a rate of 1.05 increases eta by 5% for each extra tree. By default, this value is calculated during hyperparameter optimization. It must be between 0.5 and 2.

            • feature_bag_fraction number

              Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.

            • downsample_factor number

              Advanced configuration option. Controls the fraction of data that is used to compute the derivatives of the loss function for tree training. A small value results in the use of a small fraction of the data. If this value is set to be less than 1, accuracy typically improves. However, too small a value may result in poor convergence for the ensemble and so require more trees. By default, this value is calculated during hyperparameter optimization. It must be greater than zero and less than or equal to 1.

            • max_attempts_to_add_tree number

              If the algorithm fails to determine a non-trivial tree (more than a single leaf), this parameter determines how many of such consecutive failures are tolerated. Once the number of attempts exceeds the threshold, the forest training stops.

            • max_optimization_rounds_per_hyperparameter number

              Advanced configuration option. A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter. By default, this value is calculated during hyperparameter optimization.

            • max_trees number

              Advanced configuration option. Defines the maximum number of decision trees in the forest. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.

            • num_folds number

              The maximum number of folds for the cross-validation procedure.

            • num_splits_per_feature number

              Determines the maximum number of splits for every feature that can occur in a decision tree when the tree is trained.

            • soft_tree_depth_limit number

              Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This soft limit combines with the soft_tree_depth_tolerance to penalize trees that exceed the specified depth; the regularized loss increases quickly beyond this depth. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.

            • soft_tree_depth_tolerance number

              Advanced configuration option. This option controls how quickly the regularized loss increases when the tree depth exceeds soft_tree_depth_limit. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.01.

          • iteration number Required

            The number of iterations on the analysis.

          • timestamp number

            Time unit for milliseconds

          • timing_stats object Required
            Hide timing_stats attributes Show timing_stats attributes object
          • validation_loss object Required
            Hide validation_loss attributes Show validation_loss attributes object
            • fold_values array[string] Required

              Validation loss values for every added decision tree during the forest growing procedure.

            • loss_type string Required

              The type of the loss metric. For example, binomial_logistic.

        • outlier_detection_stats object
          Hide outlier_detection_stats attributes Show outlier_detection_stats attributes object
          • parameters object Required
            Hide parameters attributes Show parameters attributes object
            • compute_feature_influence boolean

              Specifies whether the feature influence calculation is enabled.

            • feature_influence_threshold number

              The minimum outlier score that a document needs to have in order to calculate its feature influence score. Value range: 0-1

            • method string

              The method that outlier detection uses. Available methods are lof, ldof, distance_kth_nn, distance_knn, and ensemble. The default value is ensemble, which means that outlier detection uses an ensemble of different methods and normalises and combines their individual outlier scores to obtain the overall outlier score.

            • n_neighbors number

              Defines the value for how many nearest neighbors each method of outlier detection uses to calculate its outlier score. When the value is not set, different values are used for different ensemble members. This default behavior helps improve the diversity in the ensemble; only override it if you are confident that the value you choose is appropriate for the data set.

            • outlier_fraction number

              The proportion of the data set that is assumed to be outlying prior to outlier detection. For example, 0.05 means it is assumed that 5% of values are real outliers and 95% are inliers.

            • standardization_enabled boolean

              If true, the following operation is performed on the columns before computing outlier scores: (x_i - mean(x_i)) / sd(x_i).

          • timestamp number

            Time unit for milliseconds

          • timing_stats object Required
            Hide timing_stats attributes Show timing_stats attributes object
        • regression_stats object
          Hide regression_stats attributes Show regression_stats attributes object
          • hyperparameters object Required
            Hide hyperparameters attributes Show hyperparameters attributes object
            • alpha number

              Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This parameter affects loss calculations by acting as a multiplier of the tree depth. Higher alpha values result in shallower trees and faster training times. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to zero.

            • lambda number

              Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularization term which applies to leaf weights of the individual trees in the forest. A high lambda value causes training to favor small leaf weights. This behavior makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. A small lambda value results in large individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

            • gamma number

              Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. A high gamma value causes training to prefer small trees. A small gamma value results in larger individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

            • eta number

              Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, larger forests cause slower training. By default, this value is calculated during hyperparameter optimization. It must be a value between 0.001 and 1.

            • eta_growth_rate_per_tree number

              Advanced configuration option. Specifies the rate at which eta increases for each new tree that is added to the forest. For example, a rate of 1.05 increases eta by 5% for each extra tree. By default, this value is calculated during hyperparameter optimization. It must be between 0.5 and 2.

            • feature_bag_fraction number

              Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.

            • downsample_factor number

              Advanced configuration option. Controls the fraction of data that is used to compute the derivatives of the loss function for tree training. A small value results in the use of a small fraction of the data. If this value is set to be less than 1, accuracy typically improves. However, too small a value may result in poor convergence for the ensemble and so require more trees. By default, this value is calculated during hyperparameter optimization. It must be greater than zero and less than or equal to 1.

            • max_attempts_to_add_tree number

              If the algorithm fails to determine a non-trivial tree (more than a single leaf), this parameter determines how many of such consecutive failures are tolerated. Once the number of attempts exceeds the threshold, the forest training stops.

            • max_optimization_rounds_per_hyperparameter number

              Advanced configuration option. A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter. By default, this value is calculated during hyperparameter optimization.

            • max_trees number

              Advanced configuration option. Defines the maximum number of decision trees in the forest. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.

            • num_folds number

              The maximum number of folds for the cross-validation procedure.

            • num_splits_per_feature number

              Determines the maximum number of splits for every feature that can occur in a decision tree when the tree is trained.

            • soft_tree_depth_limit number

              Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This soft limit combines with the soft_tree_depth_tolerance to penalize trees that exceed the specified depth; the regularized loss increases quickly beyond this depth. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.

            • soft_tree_depth_tolerance number

              Advanced configuration option. This option controls how quickly the regularized loss increases when the tree depth exceeds soft_tree_depth_limit. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.01.

          • iteration number Required

            The number of iterations on the analysis.

          • timestamp number

            Time unit for milliseconds

          • timing_stats object Required
            Hide timing_stats attributes Show timing_stats attributes object
          • validation_loss object Required
            Hide validation_loss attributes Show validation_loss attributes object
            • fold_values array[string] Required

              Validation loss values for every added decision tree during the forest growing procedure.

            • loss_type string Required

              The type of the loss metric. For example, binomial_logistic.

      • assignment_explanation string

        For running jobs only, contains messages relating to the selection of a node to run the job.

      • data_counts object Required
        Hide data_counts attributes Show data_counts attributes object
        • skipped_docs_count number Required

          The number of documents that are skipped during the analysis because they contained values that are not supported by the analysis. For example, outlier detection does not support missing fields so it skips documents with missing fields. Likewise, all types of analysis skip documents that contain arrays with more than one element.

        • test_docs_count number Required

          The number of documents that are not used for training the model and can be used for testing.

        • training_docs_count number Required

          The number of documents that are used for training the model.

      • id string Required
      • memory_usage object Required
        Hide memory_usage attributes Show memory_usage attributes object
        • memory_reestimate_bytes number

          This value is present when the status is hard_limit and it is a new estimate of how much memory the job needs.

        • peak_usage_bytes number Required

          The number of bytes used at the highest peak of memory usage.

        • status string Required

          The memory usage status.

        • timestamp number

          Time unit for milliseconds

      • node object
        Hide node attributes Show node attributes object
      • progress array[object] Required

        The progress report of the data frame analytics job by phase.

        Hide progress attributes Show progress attributes object
        • phase string Required

          Defines the phase of the data frame analytics job.

        • progress_percent number Required

          The progress that the data frame analytics job has made expressed in percentage.

      • state string Required

        Values are started, stopped, starting, stopping, or failed.
GET /_ml/data_frame/analytics/{id}/_stats 
curl \
 --request GET 'http://api.example.com/_ml/data_frame/analytics/{id}/_stats' \
 --header "Authorization: $API_KEY"

Start a trained model deployment Added in 8.0.0

POST /_ml/trained_models/{model_id}/deployment/_start
Api key auth Basic auth Bearer auth

It allocates the model to every machine learning node.

Path parameters

  • model_id string Required

    The unique identifier of the trained model. Currently, only PyTorch models are supported.

Query parameters

  • cache_size number | string

    The inference cache size (in memory outside the JVM heap) per node for the model. The default value is the same size as the model_size_bytes. To disable the cache, 0b can be provided.

  • deployment_id string

    A unique identifier for the deployment of the model.

  • number_of_allocations number

    The number of model allocations on each node where the model is deployed. All allocations on a node share the same copy of the model in memory but use a separate set of threads to evaluate the model. Increasing this value generally increases the throughput. If this setting is greater than the number of hardware threads it will automatically be changed to a value less than the number of hardware threads. If adaptive_allocations is enabled, do not set this value, because it’s automatically set.

  • priority string

    The deployment priority.

    Values are normal or low.

  • queue_capacity number

    Specifies the number of inference requests that are allowed in the queue. After the number of requests exceeds this value, new requests are rejected with a 429 error.

  • threads_per_allocation number

    Sets the number of threads used by each model allocation during inference. This generally increases the inference speed. The inference process is a compute-bound process; any number greater than the number of available hardware threads on the machine does not increase the inference speed. If this setting is greater than the number of hardware threads it will automatically be changed to a value less than the number of hardware threads.

  • timeout string

    Specifies the amount of time to wait for the model to deploy.

  • wait_for string

    Specifies the allocation status to wait for before returning.

    Supported values include:

    • started: The trained model is started on at least one node.
    • starting: Trained model deployment is starting but it is not yet deployed on any nodes.
    • fully_allocated: Trained model deployment has started on all valid nodes.

    Values are started, starting, or fully_allocated.

application/json

Body

  • adaptive_allocations object
    Hide adaptive_allocations attributes Show adaptive_allocations attributes object
    • enabled boolean Required

      If true, adaptive_allocations is enabled

    • min_number_of_allocations number

      Specifies the minimum number of allocations to scale to. If set, it must be greater than or equal to 0. If not defined, the deployment scales to 0.

    • max_number_of_allocations number

      Specifies the maximum number of allocations to scale to. If set, it must be greater than or equal to min_number_of_allocations.

Responses

POST /_ml/trained_models/{model_id}/deployment/_start 
curl \
 --request POST 'http://api.example.com/_ml/trained_models/{model_id}/deployment/_start' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '{"adaptive_allocations":{"enabled":true,"min_number_of_allocations":42.0,"max_number_of_allocations":42.0}}'

Get deprecation information Added in 6.1.0

GET /_migration/deprecations
Api key auth Basic auth Bearer auth

Get information about different cluster, node, and index level settings that use deprecated features that will be removed or changed in the next major version.

TIP: This APIs is designed for indirect use by the Upgrade Assistant. You are strongly recommended to use the Upgrade Assistant.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • cluster_settings array[object] Required

      Cluster-level deprecation warnings.

      Hide cluster_settings attributes Show cluster_settings attributes object
      • details string

        Optional details about the deprecation warning.

      • level string Required

        Values are none, info, warning, or critical.

      • message string Required

        Descriptive information about the deprecation warning.

      • url string Required

        A link to the breaking change documentation, where you can find more information about this change.

      • resolve_during_rolling_upgrade boolean Required
      • _meta object
        Hide _meta attribute Show _meta attribute object
        • * object Additional properties
    • index_settings object Required

      Index warnings are sectioned off per index and can be filtered using an index-pattern in the query. This section includes warnings for the backing indices of data streams specified in the request path.

      Hide index_settings attribute Show index_settings attribute object
      • * array[object] Additional properties
        Hide * attributes Show * attributes object
        • details string

          Optional details about the deprecation warning.

        • level string Required

          Values are none, info, warning, or critical.

        • message string Required

          Descriptive information about the deprecation warning.

        • url string Required

          A link to the breaking change documentation, where you can find more information about this change.

        • resolve_during_rolling_upgrade boolean Required
        • _meta object
          Hide _meta attribute Show _meta attribute object
          • * object Additional properties
    • data_streams object Required
      Hide data_streams attribute Show data_streams attribute object
      • * array[object] Additional properties
        Hide * attributes Show * attributes object
        • details string

          Optional details about the deprecation warning.

        • level string Required

          Values are none, info, warning, or critical.

        • message string Required

          Descriptive information about the deprecation warning.

        • url string Required

          A link to the breaking change documentation, where you can find more information about this change.

        • resolve_during_rolling_upgrade boolean Required
        • _meta object
          Hide _meta attribute Show _meta attribute object
          • * object Additional properties
    • node_settings array[object] Required

      Node-level deprecation warnings. Since only a subset of your nodes might incorporate these settings, it is important to read the details section for more information about which nodes are affected.

      Hide node_settings attributes Show node_settings attributes object
      • details string

        Optional details about the deprecation warning.

      • level string Required

        Values are none, info, warning, or critical.

      • message string Required

        Descriptive information about the deprecation warning.

      • url string Required

        A link to the breaking change documentation, where you can find more information about this change.

      • resolve_during_rolling_upgrade boolean Required
      • _meta object
        Hide _meta attribute Show _meta attribute object
        • * object Additional properties
    • ml_settings array[object] Required

      Machine learning-related deprecation warnings.

      Hide ml_settings attributes Show ml_settings attributes object
      • details string

        Optional details about the deprecation warning.

      • level string Required

        Values are none, info, warning, or critical.

      • message string Required

        Descriptive information about the deprecation warning.

      • url string Required

        A link to the breaking change documentation, where you can find more information about this change.

      • resolve_during_rolling_upgrade boolean Required
      • _meta object
        Hide _meta attribute Show _meta attribute object
        • * object Additional properties
    • templates object Required

      Template warnings are sectioned off per template and include deprecations for both component templates and index templates.

      Hide templates attribute Show templates attribute object
      • * array[object] Additional properties
        Hide * attributes Show * attributes object
        • details string

          Optional details about the deprecation warning.

        • level string Required

          Values are none, info, warning, or critical.

        • message string Required

          Descriptive information about the deprecation warning.

        • url string Required

          A link to the breaking change documentation, where you can find more information about this change.

        • resolve_during_rolling_upgrade boolean Required
        • _meta object
          Hide _meta attribute Show _meta attribute object
          • * object Additional properties
    • ilm_policies object Required

      ILM policy warnings are sectioned off per policy.

      Hide ilm_policies attribute Show ilm_policies attribute object
      • * array[object] Additional properties
        Hide * attributes Show * attributes object
        • details string

          Optional details about the deprecation warning.

        • level string Required

          Values are none, info, warning, or critical.

        • message string Required

          Descriptive information about the deprecation warning.

        • url string Required

          A link to the breaking change documentation, where you can find more information about this change.

        • resolve_during_rolling_upgrade boolean Required
        • _meta object
          Hide _meta attribute Show _meta attribute object
          • * object Additional properties
GET /_migration/deprecations 
curl \
 --request GET 'http://api.example.com/_migration/deprecations' \
 --header "Authorization: $API_KEY"
Response examples (200)
An abbreviated response from `GET /_migration/deprecations`. 
{
  "cluster_settings": [
    {
      "level": "critical",
      "message": "Cluster name cannot contain ':'",
      "url": "https://www.elastic.co/guide/en/elasticsearch/reference/7.0/breaking-changes-7.0.html#_literal_literal_is_no_longer_allowed_in_cluster_name",
      "details": "This cluster is named [mycompany:logging], which contains the illegal character ':'."
    }
  ],
  "node_settings": [],
  "index_settings": {
    "logs:apache": [
      {
        "level": "warning",
        "message": "Index name cannot contain ':'",
        "url": "https://www.elastic.co/guide/en/elasticsearch/reference/7.0/breaking-changes-7.0.html#_literal_literal_is_no_longer_allowed_in_index_name",
        "details": "This index is named [logs:apache], which contains the illegal character ':'."
      }
    ]
  },
  "ml_settings": []
}

Run a scrolling search

GET /_search/scroll/{scroll_id}
Api key auth Basic auth Bearer auth

IMPORTANT: The scroll API is no longer recommend for deep pagination. If you need to preserve the index state while paging through more than 10,000 hits, use the search_after parameter with a point in time (PIT).

The scroll API gets large sets of results from a single scrolling search request. To get the necessary scroll ID, submit a search API request that includes an argument for the scroll query parameter. The scroll parameter indicates how long Elasticsearch should retain the search context for the request. The search response returns a scroll ID in the _scroll_id response body parameter. You can then use the scroll ID with the scroll API to retrieve the next batch of results for the request. If the Elasticsearch security features are enabled, the access to the results of a specific scroll ID is restricted to the user or API key that submitted the search.

You can also use the scroll API to specify a new scroll parameter that extends or shortens the retention period for the search context.

IMPORTANT: Results from a scrolling search reflect the state of the index at the time of the initial search request. Subsequent indexing or document changes only affect later search and scroll requests.

External documentation

Path parameters

Query parameters

  • scroll string

    The period to retain the search context for scrolling.

  • scroll_id string Deprecated

    The scroll ID for scrolled search

  • rest_total_hits_as_int boolean

    If true, the API response’s hit.total property is returned as an integer. If false, the API response’s hit.total property is returned as an object.

application/json

Body

  • scroll string

    A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

  • scroll_id string Required

Responses

GET /_search/scroll/{scroll_id} 
curl \
 --request GET 'http://api.example.com/_search/scroll/{scroll_id}' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '"{\n  \"scroll_id\" : \"DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ==\"\n}"'
Request example
Run `GET /_search/scroll` to get the next batch of results for a scrolling search. 
{
  "scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ=="
}

Count search results

GET /{index}/_count
Api key auth Basic auth Bearer auth

Get the number of documents matching a query.

The query can be provided either by using a simple query string as a parameter, or by defining Query DSL within the request body. The query is optional. When no query is provided, the API uses match_all to count all the documents.

The count API supports multi-target syntax. You can run a single count API search across multiple data streams and indices.

The operation is broadcast across all shards. For each shard ID group, a replica is chosen and the search is run against it. This means that replicas increase the scalability of the count.

Path parameters

  • index string | array[string] Required

    A comma-separated list of data streams, indices, and aliases to search. It supports wildcards (*). To search all data streams and indices, omit this parameter or use * or _all.

Query parameters

  • allow_no_indices boolean

    If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.

  • analyzer string

    The analyzer to use for the query string. This parameter can be used only when the q query string parameter is specified.

  • analyze_wildcard boolean

    If true, wildcard and prefix queries are analyzed. This parameter can be used only when the q query string parameter is specified.

  • default_operator string

    The default operator for query string query: AND or OR. This parameter can be used only when the q query string parameter is specified.

    Values are and, AND, or, or OR.

  • df string

    The field to use as a default when no field prefix is given in the query string. This parameter can be used only when the q query string parameter is specified.

  • expand_wildcards string | array[string]

    The type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. It supports comma-separated values, such as open,hidden.

    Supported values include:

    • all: Match any data stream or index, including hidden ones.
    • open: Match open, non-hidden indices. Also matches any non-hidden data stream.
    • closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.
    • hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.
    • none: Wildcard expressions are not accepted.
  • ignore_throttled boolean Deprecated

    If true, concrete, expanded, or aliased indices are ignored when frozen.

  • ignore_unavailable boolean

    If false, the request returns an error if it targets a missing or closed index.

  • lenient boolean

    If true, format-based query failures (such as providing text to a numeric field) in the query string will be ignored. This parameter can be used only when the q query string parameter is specified.

  • min_score number

    The minimum _score value that documents must have to be included in the result.

  • preference string

    The node or shard the operation should be performed on. By default, it is random.

  • routing string

    A custom value used to route operations to a specific shard.

  • terminate_after number

    The maximum number of documents to collect for each shard. If a query reaches this limit, Elasticsearch terminates the query early. Elasticsearch collects documents before sorting.

    IMPORTANT: Use with caution. Elasticsearch applies this parameter to each shard handling the request. When possible, let Elasticsearch perform early termination automatically. Avoid specifying this parameter for requests that target data streams with backing indices across multiple data tiers.

  • q string

    The query in Lucene query string syntax. This parameter cannot be used with a request body.

application/json

Body

Responses

GET /{index}/_count 
curl \
 --request GET 'http://api.example.com/{index}/_count' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '"{\n  \"query\" : {\n    \"term\" : { \"user.id\" : \"kimchy\" }\n  }\n}"'
Request example
Run `GET /my-index-000001/_count?q=user:kimchy`. Alternatively, run `GET /my-index-000001/_count` with the same query in the request body. Both requests count the number of documents in `my-index-000001` with a `user.id` of `kimchy`. 
{
  "query" : {
    "term" : { "user.id" : "kimchy" }
  }
}
Response examples (200)
A successful response from `GET /my-index-000001/_count?q=user:kimchy`. 
{
  "count": 1,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  }
}