Get the autoscaling capacity Added in 7.11.0

GET /_autoscaling/capacity
Api key auth Basic auth Bearer auth

NOTE: This feature is designed for indirect use by Elasticsearch Service, Elastic Cloud Enterprise, and Elastic Cloud on Kubernetes. Direct use is not supported.

This API gets the current autoscaling capacity based on the configured autoscaling policy. It will return information to size the cluster appropriately to the current workload.

The required_capacity is calculated as the maximum of the required_capacity result of all individual deciders that are enabled for the policy.

The operator should verify that the current_nodes match the operator’s knowledge of the cluster to avoid making autoscaling decisions based on stale or incomplete information.

The response contains decider-specific information you can use to diagnose how and why autoscaling determined a certain capacity was required. This information is provided for diagnosis only. Do not use this information to make autoscaling decisions.

External documentation

Query parameters

  • master_timeout string

    Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

    Values are -1 or 0.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • policies object Required
      Hide policies attribute Show policies attribute object
      • * object Additional properties
        Hide * attributes Show * attributes object
        • required_capacity object Required
          Hide required_capacity attributes Show required_capacity attributes object
          • node object Required
            Hide node attributes Show node attributes object
          • total object Required
            Hide total attributes Show total attributes object
        • current_capacity object Required
          Hide current_capacity attributes Show current_capacity attributes object
          • node object Required
            Hide node attributes Show node attributes object
          • total object Required
            Hide total attributes Show total attributes object
        • current_nodes array[object] Required
          Hide current_nodes attribute Show current_nodes attribute object
        • deciders object Required
          Hide deciders attribute Show deciders attribute object
GET /_autoscaling/capacity 
GET /_autoscaling/capacity

curl \
 --request GET 'http://api.example.com/_autoscaling/capacity' \
 --header "Authorization: $API_KEY"
Response examples (200)
This may be a response to `GET /_autoscaling/capacity`. 
{
  policies: {}
}

Get index information

GET /_cat/indices
Api key auth Basic auth Bearer auth

Get high-level information about indices in a cluster, including backing indices for data streams.

Use this request to get the following information for each index in a cluster:

  • shard count
  • document count
  • deleted document count
  • primary store size
  • total store size of all shards, including shard replicas

These metrics are retrieved directly from Lucene, which Elasticsearch uses internally to power indexing and search. As a result, all document counts include hidden nested documents. To get an accurate count of Elasticsearch documents, use the cat count or count APIs.

CAT APIs are only intended for human consumption using the command line or Kibana console. They are not intended for use by applications. For application consumption, use an index endpoint.

Query parameters

  • bytes string

    The unit used to display byte values.

    Values are b, kb, mb, gb, tb, or pb.

  • expand_wildcards string | array[string]

    The type of index that wildcard patterns can match.

    Supported values include:

    • all: Match any data stream or index, including hidden ones.
    • open: Match open, non-hidden indices. Also matches any non-hidden data stream.
    • closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.
    • hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.
    • none: Wildcard expressions are not accepted.

    Values are all, open, closed, hidden, or none.

  • health string

    The health status used to limit returned indices. By default, the response includes indices of any health status.

    Supported values include:

    • green (or GREEN): All shards are assigned.
    • yellow (or YELLOW): All primary shards are assigned, but one or more replica shards are unassigned. If a node in the cluster fails, some data could be unavailable until that node is repaired.
    • red (or RED): One or more primary shards are unassigned, so some data is unavailable. This can occur briefly during cluster startup as primary shards are assigned.

    Values are green, GREEN, yellow, YELLOW, red, or RED.

  • include_unloaded_segments boolean

    If true, the response includes information from segments that are not loaded into memory.

  • pri boolean

    If true, the response only includes information from primary shards.

  • time string

    The unit used to display time values.

    Values are nanos, micros, ms, s, m, h, or d.

  • master_timeout string

    Period to wait for a connection to the master node.

    Values are -1 or 0.

  • h string | array[string]

    List of columns to appear in the response. Supports simple wildcards.

  • s string | array[string]

    List of columns that determine how the table should be sorted. Sorting defaults to ascending and can be changed by setting :asc or :desc as a suffix to the column name.

Responses

GET /_cat/indices 
GET /_cat/indices/my-index-*?v=true&s=index&format=json

curl \
 --request GET 'http://api.example.com/_cat/indices' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response from `GET /_cat/indices/my-index-*?v=true&s=index&format=json`. 
[
  {
    "health": "yellow",
    "status": "open",
    "index": "my-index-000001",
    "uuid": "u8FNjxh8Rfy_awN11oDKYQ",
    "pri": "1",
    "rep": "1",
    "docs.count": "1200",
    "docs.deleted": "0",
    "store.size": "88.1kb",
    "pri.store.size": "88.1kb",
    "dataset.size": "88.1kb"
  },
  {
    "health": "green",
    "status": "open",
    "index": "my-index-000002",
    "uuid": "nYFWZEO7TUiOjLQXBaYJpA ",
    "pri": "1",
    "rep": "0",
    "docs.count": "0",
    "docs.deleted": "0",
    "store.size": "260b",
    "pri.store.size": "260b",
    "dataset.size": "260b"
  }
]

Get trained models Added in 7.7.0

GET /_cat/ml/trained_models/{model_id}
Api key auth Basic auth Bearer auth

Get configuration and usage information about inference trained models.

IMPORTANT: CAT APIs are only intended for human consumption using the Kibana console or command line. They are not intended for use by applications. For application consumption, use the get trained models statistics API.

Path parameters

  • model_id string Required

    A unique identifier for the trained model.

Query parameters

  • allow_no_match boolean

    Specifies what to do when the request: contains wildcard expressions and there are no models that match; contains the _all string or no identifiers and there are no matches; contains wildcard expressions and there are only partial matches. If true, the API returns an empty array when there are no matches and the subset of results when there are partial matches. If false, the API returns a 404 status code when there are no matches or only partial matches.

  • bytes string

    The unit used to display byte values.

    Values are b, kb, mb, gb, tb, or pb.

  • h string | array[string]

    A comma-separated list of column names to display.

    Supported values include:

    • create_time (or ct): The time when the trained model was created.
    • created_by (or c, createdBy): Information on the creator of the trained model.
    • data_frame_analytics_id (or df, dataFrameAnalytics, dfid): Identifier for the data frame analytics job that created the model. Only displayed if it is still available.
    • description (or d): The description of the trained model.
    • heap_size (or hs, modelHeapSize): The estimated heap size to keep the trained model in memory.
    • id: Identifier for the trained model.
    • ingest.count (or ic, ingestCount): The total number of documents that are processed by the model.
    • ingest.current (or icurr, ingestCurrent): The total number of document that are currently being handled by the trained model.
    • ingest.failed (or if, ingestFailed): The total number of failed ingest attempts with the trained model.
    • ingest.pipelines (or ip, ingestPipelines): The total number of ingest pipelines that are referencing the trained model.
    • ingest.time (or it, ingestTime): The total time that is spent processing documents with the trained model.
    • license (or l): The license level of the trained model.
    • operations (or o, modelOperations): The estimated number of operations to use the trained model. This number helps measuring the computational complexity of the model.
    • version (or v): The Elasticsearch version number in which the trained model was created.

    Values are create_time, ct, created_by, c, createdBy, data_frame_analytics_id, df, dataFrameAnalytics, dfid, description, d, heap_size, hs, modelHeapSize, id, ingest.count, ic, ingestCount, ingest.current, icurr, ingestCurrent, ingest.failed, if, ingestFailed, ingest.pipelines, ip, ingestPipelines, ingest.time, it, ingestTime, license, l, operations, o, modelOperations, version, or v.

  • s string | array[string]

    A comma-separated list of column names or aliases used to sort the response.

    Supported values include:

    • create_time (or ct): The time when the trained model was created.
    • created_by (or c, createdBy): Information on the creator of the trained model.
    • data_frame_analytics_id (or df, dataFrameAnalytics, dfid): Identifier for the data frame analytics job that created the model. Only displayed if it is still available.
    • description (or d): The description of the trained model.
    • heap_size (or hs, modelHeapSize): The estimated heap size to keep the trained model in memory.
    • id: Identifier for the trained model.
    • ingest.count (or ic, ingestCount): The total number of documents that are processed by the model.
    • ingest.current (or icurr, ingestCurrent): The total number of document that are currently being handled by the trained model.
    • ingest.failed (or if, ingestFailed): The total number of failed ingest attempts with the trained model.
    • ingest.pipelines (or ip, ingestPipelines): The total number of ingest pipelines that are referencing the trained model.
    • ingest.time (or it, ingestTime): The total time that is spent processing documents with the trained model.
    • license (or l): The license level of the trained model.
    • operations (or o, modelOperations): The estimated number of operations to use the trained model. This number helps measuring the computational complexity of the model.
    • version (or v): The Elasticsearch version number in which the trained model was created.

    Values are create_time, ct, created_by, c, createdBy, data_frame_analytics_id, df, dataFrameAnalytics, dfid, description, d, heap_size, hs, modelHeapSize, id, ingest.count, ic, ingestCount, ingest.current, icurr, ingestCurrent, ingest.failed, if, ingestFailed, ingest.pipelines, ip, ingestPipelines, ingest.time, it, ingestTime, license, l, operations, o, modelOperations, version, or v.

  • from number

    Skips the specified number of transforms.

  • size number

    The maximum number of transforms to display.

  • time string

    Unit used to display time values.

    Values are nanos, micros, ms, s, m, h, or d.

Responses

GET /_cat/ml/trained_models/{model_id} 
GET _cat/ml/trained_models?v=true&format=json

curl \
 --request GET 'http://api.example.com/_cat/ml/trained_models/{model_id}' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response from `GET _cat/ml/trained_models?v=true&format=json`. 
[
  {
    "id": "ddddd-1580216177138",
    "heap_size": "0b",
    "operations": "196",
    "create_time": "2025-03-25T00:01:38.662Z",
    "type": "pytorch",
    "ingest.pipelines": "0",
    "data_frame.id": "__none__"
  },
  {
    "id": "lang_ident_model_1",
    "heap_size": "1mb",
    "operations": "39629",
    "create_time": "2019-12-05T12:28:34.594Z",
    "type": "lang_ident",
    "ingest.pipelines": "0",
    "data_frame.id": "__none__"
  }
]

Get node attribute information

GET /_cat/nodeattrs
Api key auth Basic auth Bearer auth

Get information about custom node attributes. IMPORTANT: cat APIs are only intended for human consumption using the command line or Kibana console. They are not intended for use by applications. For application consumption, use the nodes info API.

Query parameters

  • h string | array[string]

    List of columns to appear in the response. Supports simple wildcards.

  • s string | array[string]

    List of columns that determine how the table should be sorted. Sorting defaults to ascending and can be changed by setting :asc or :desc as a suffix to the column name.

  • local boolean

    If true, the request computes the list of selected nodes from the local cluster state. If false the list of selected nodes are computed from the cluster state of the master node. In both cases the coordinating node will send requests for further information to each selected node.

  • master_timeout string

    Period to wait for a connection to the master node.

    Values are -1 or 0.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • node string

      The node name.

    • id string

      The unique node identifier.

    • pid string

      The process identifier.

    • host string

      The host name.

    • ip string

      The IP address.

    • port string

      The bound transport port.

    • attr string

      The attribute name.

    • value string

      The attribute value.
GET /_cat/nodeattrs 
GET /_cat/nodeattrs?v=true&format=json

curl \
 --request GET 'http://api.example.com/_cat/nodeattrs' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response from `GET /_cat/nodeattrs?v=true&format=json`. The `node`, `host`, and `ip` columns provide basic information about each node. The `attr` and `value` columns return custom node attributes, one per line. 
[
  {
    "node": "node-0",
    "host": "127.0.0.1",
    "ip": "127.0.0.1",
    "attr": "testattr",
    "value": "test"
  }
]
A successful response from `GET /_cat/nodeattrs?v=true&h=name,pid,attr,value`. It returns the `name`, `pid`, `attr`, and `value` columns. 
[
  {
    "name": "node-0",
    "pid": "19566",
    "attr": "testattr",
    "value": "test"
  }
]

Get node information Added in 1.3.0

GET /_nodes/{node_id}
Api key auth Basic auth Bearer auth

By default, the API returns all attributes and core settings for cluster nodes.

Path parameters

  • node_id string | array[string] Required

    Comma-separated list of node IDs or names used to limit returned information.

Query parameters

  • flat_settings boolean

    If true, returns settings in flat format.

  • timeout string

    Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

    Values are -1 or 0.

Responses

GET /_nodes/{node_id} 
curl \
 --request GET 'http://api.example.com/_nodes/{node_id}' \
 --header "Authorization: $API_KEY"
Response examples (200)
An abbreviated response when requesting cluster nodes information. 
{
    "_nodes": {},
    "cluster_name": "elasticsearch",
    "nodes": {
      "USpTGYaBSIKbgSUJR2Z9lg": {
        "name": "node-0",
        "transport_address": "192.168.17:9300",
        "host": "node-0.elastic.co",
        "ip": "192.168.17",
        "version": "{version}",
        "transport_version": 100000298,
        "index_version": 100000074,
        "component_versions": {
          "ml_config_version": 100000162,
          "transform_config_version": 100000096
        },
        "build_flavor": "default",
        "build_type": "{build_type}",
        "build_hash": "587409e",
        "roles": [
          "master",
          "data",
          "ingest"
        ],
        "attributes": {},
        "plugins": [
          {
            "name": "analysis-icu",
            "version": "{version}",
            "description": "The ICU Analysis plugin integrates Lucene ICU
  module into elasticsearch, adding ICU relates analysis components.",
            "classname":
  "org.elasticsearch.plugin.analysis.icu.AnalysisICUPlugin",
            "has_native_controller": false
          }
        ],
        "modules": [
          {
            "name": "lang-painless",
            "version": "{version}",
            "description": "An easy, safe and fast scripting language for
  Elasticsearch",
            "classname": "org.elasticsearch.painless.PainlessPlugin",
            "has_native_controller": false
          }
        ]
      }
    }
}

Update the connector is_native flag Beta

PUT /_connector/{connector_id}/_native
Api key auth Basic auth Bearer auth

Path parameters

  • connector_id string Required

    The unique identifier of the connector to be updated

application/json

Body Required

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • result string Required

      Values are created, updated, deleted, not_found, or noop.
PUT /_connector/{connector_id}/_native 
curl \
 --request PUT 'http://api.example.com/_connector/{connector_id}/_native' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '{"is_native":true}'

Update the connector pipeline Beta

PUT /_connector/{connector_id}/_pipeline
Api key auth Basic auth Bearer auth

When you create a new connector, the configuration of an ingest pipeline is populated with default settings.

Path parameters

  • connector_id string Required

    The unique identifier of the connector to be updated

application/json

Body Required

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • result string Required

      Values are created, updated, deleted, not_found, or noop.
PUT /_connector/{connector_id}/_pipeline 
PUT _connector/my-connector/_pipeline
{
    "pipeline": {
        "extract_binary_content": true,
        "name": "my-connector-pipeline",
        "reduce_whitespace": true,
        "run_ml_inference": true
    }
}
curl \
 --request PUT 'http://api.example.com/_connector/{connector_id}/_pipeline' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '"{\n    \"pipeline\": {\n        \"extract_binary_content\": true,\n        \"name\": \"my-connector-pipeline\",\n        \"reduce_whitespace\": true,\n        \"run_ml_inference\": true\n    }\n}"'
Request example 
{
    "pipeline": {
        "extract_binary_content": true,
        "name": "my-connector-pipeline",
        "reduce_whitespace": true,
        "run_ml_inference": true
    }
}
Response examples (200) 
{
  "result": "updated"
}

Update the connector service type Beta

PUT /_connector/{connector_id}/_service_type
Api key auth Basic auth Bearer auth

Path parameters

  • connector_id string Required

    The unique identifier of the connector to be updated

application/json

Body Required

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • result string Required

      Values are created, updated, deleted, not_found, or noop.
PUT /_connector/{connector_id}/_service_type 
PUT _connector/my-connector/_service_type
{
    "service_type": "sharepoint_online"
}
curl \
 --request PUT 'http://api.example.com/_connector/{connector_id}/_service_type' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '"{\n    \"service_type\": \"sharepoint_online\"\n}"'
Request example 
{
    "service_type": "sharepoint_online"
}
Response examples (200) 
{
  "result": "updated"
}

Create a data stream Added in 7.9.0

PUT /_data_stream/{name}
Api key auth Basic auth Bearer auth

You must have a matching index template with data stream enabled.

Path parameters

  • name string Required

    Name of the data stream, which must meet the following criteria: Lowercase only; Cannot include \, /, *, ?, ", <, >, |, ,, #, :, or a space character; Cannot start with -, _, +, or .ds-; Cannot be . or ..; Cannot be longer than 255 bytes. Multi-byte characters count towards this limit faster.

Query parameters

  • master_timeout string

    Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

    Values are -1 or 0.

  • timeout string

    Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

    Values are -1 or 0.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • acknowledged boolean Required

      For a successful response, this value is always true. On failure, an exception is returned instead.
PUT /_data_stream/{name} 
curl \
 --request PUT 'http://api.example.com/_data_stream/{name}' \
 --header "Authorization: $API_KEY"

Explore graph analytics

GET /{index}/_graph/explore
Api key auth Basic auth Bearer auth

Extract and summarize information about the documents and terms in an Elasticsearch data stream or index. The easiest way to understand the behavior of this API is to use the Graph UI to explore connections. An initial request to the _explore API contains a seed query that identifies the documents of interest and specifies the fields that define the vertices and connections you want to include in the graph. Subsequent requests enable you to spider out from one more vertices of interest. You can exclude vertices that have already been returned.

External documentation

Path parameters

  • index string | array[string] Required

    Name of the index.

Query parameters

  • routing string

    Custom value used to route operations to a specific shard.

  • timeout string

    Specifies the period of time to wait for a response from each shard. If no response is received before the timeout expires, the request fails and returns an error. Defaults to no timeout.

    Values are -1 or 0.

application/json

Body

  • connections object
    Hide connections attributes Show connections attributes object
    • connections object
    • query object

      An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

      External documentation
    • vertices array[object] Required

      Contains the fields you are interested in.

      Hide vertices attributes Show vertices attributes object
      • exclude array[string]

        Prevents the specified terms from being included in the results.

      • field string Required

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • include array[object]

        Identifies the terms of interest that form the starting points from which you want to spider out.

        Hide include attributes Show include attributes object
      • min_doc_count number

        Specifies how many documents must contain a pair of terms before it is considered to be a useful connection. This setting acts as a certainty threshold.

      • shard_min_doc_count number

        Controls how many documents on a particular shard have to contain a pair of terms before the connection is returned for global consideration.

      • size number

        Specifies the maximum number of vertex terms returned for each field.

  • controls object
    Hide controls attributes Show controls attributes object
    • sample_diversity object
      Hide sample_diversity attributes Show sample_diversity attributes object
      • field string Required

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • max_docs_per_value number Required
    • sample_size number

      Each hop considers a sample of the best-matching documents on each shard. Using samples improves the speed of execution and keeps exploration focused on meaningfully-connected terms. Very small values (less than 50) might not provide sufficient weight-of-evidence to identify significant connections between terms. Very large sample sizes can dilute the quality of the results and increase execution times.

    • timeout string

      A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • use_significance boolean Required

      Filters associated terms so only those that are significantly associated with your query are included.

  • query object

    An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

    External documentation
  • vertices array[object]

    Specifies one or more fields that contain the terms you want to include in the graph as vertices.

    Hide vertices attributes Show vertices attributes object
    • exclude array[string]

      Prevents the specified terms from being included in the results.

    • field string Required

      Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • include array[object]

      Identifies the terms of interest that form the starting points from which you want to spider out.

      Hide include attributes Show include attributes object
    • min_doc_count number

      Specifies how many documents must contain a pair of terms before it is considered to be a useful connection. This setting acts as a certainty threshold.

    • shard_min_doc_count number

      Controls how many documents on a particular shard have to contain a pair of terms before the connection is returned for global consideration.

    • size number

      Specifies the maximum number of vertex terms returned for each field.

Responses

GET /{index}/_graph/explore 
POST clicklogs/_graph/explore
{
  "query": {
    "match": {
      "query.raw": "midi"
    }
  },
  "vertices": [
    {
      "field": "product"
    }
  ],
  "connections": {
    "vertices": [
      {
        "field": "query.raw"
      }
    ]
  }
}
curl \
 --request GET 'http://api.example.com/{index}/_graph/explore' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '"{\n  \"query\": {\n    \"match\": {\n      \"query.raw\": \"midi\"\n    }\n  },\n  \"vertices\": [\n    {\n      \"field\": \"product\"\n    }\n  ],\n  \"connections\": {\n    \"vertices\": [\n      {\n        \"field\": \"query.raw\"\n      }\n    ]\n  }\n}"'
Request example
Run `POST clicklogs/_graph/explore` for a basic exploration An initial graph explore query typically begins with a query to identify strongly related terms. Seed the exploration with a query. This example is searching `clicklogs` for people who searched for the term `midi`.Identify the vertices to include in the graph. This example is looking for product codes that are significantly associated with searches for `midi`. Find the connections. This example is looking for other search terms that led people to click on the products that are associated with searches for `midi`. 
{
  "query": {
    "match": {
      "query.raw": "midi"
    }
  },
  "vertices": [
    {
      "field": "product"
    }
  ],
  "connections": {
    "vertices": [
      {
        "field": "query.raw"
      }
    ]
  }
}

Stop the ILM plugin Added in 6.6.0

POST /_ilm/stop
Api key auth Basic auth Bearer auth

Halt all lifecycle management operations and stop the index lifecycle management plugin. This is useful when you are performing maintenance on the cluster and need to prevent ILM from performing any actions on your indices.

The API returns as soon as the stop request has been acknowledged, but the plugin might continue to run until in-progress operations complete and the plugin can be safely stopped. Use the get ILM status API to check whether ILM is running.

Query parameters

  • master_timeout string

    Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

    Values are -1 or 0.

  • timeout string

    Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

    Values are -1 or 0.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • acknowledged boolean Required

      For a successful response, this value is always true. On failure, an exception is returned instead.
POST /_ilm/stop 
curl \
 --request POST 'http://api.example.com/_ilm/stop' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response when stopping the ILM plugin. 
{
  "acknowledged": true
}

Perform chat completion inference Added in 8.18.0

POST /_inference/chat_completion/{inference_id}/_stream
Api key auth Basic auth Bearer auth

The chat completion inference API enables real-time responses for chat completion tasks by delivering answers incrementally, reducing response times during computation. It only works with the chat_completion task type for openai and elastic inference services.

NOTE: The chat_completion task type is only available within the _stream API and only supports streaming. The Chat completion inference API and the Stream inference API differ in their response structure and capabilities. The Chat completion inference API provides more comprehensive customization options through more fields and function calling support. If you use the openai service or the elastic service, use the Chat completion inference API.

Path parameters

Query parameters

  • timeout string

    Specifies the amount of time to wait for the inference request to complete.

    Values are -1 or 0.

application/json

Body Required

  • messages array[object] Required

    A list of objects representing the conversation. Requests should generally only add new messages from the user (role user). The other message roles (assistant, system, or tool) should generally only be copied from the response to a previous completion request, such that the messages array is built up throughout a conversation.

    Hide messages attributes Show messages attributes object
  • model string

    The ID of the model to use.

  • max_completion_tokens number

    The upper bound limit for the number of tokens that can be generated for a completion request.

  • stop array[string]

    A sequence of strings to control when the model should stop generating additional tokens.

  • temperature number

    The sampling temperature to use.

  • tool_choice string | object

    One of:
    CompletionToolType string CompletionToolChoice object
  • tools array[object]

    A list of tools that the model can call.

    Hide tools attributes Show tools attributes object
    • type string Required

      The type of tool.

    • function object Required
      Hide function attributes Show function attributes object
      • description string

        A description of what the function does. This is used by the model to choose when and how to call the function.

      • name string Required

        The name of the function.

      • parameters object

        The parameters the functional accepts. This should be formatted as a JSON object.

      • strict boolean

        Whether to enable schema adherence when generating the function call.

  • top_p number

    Nucleus sampling, an alternative to sampling with temperature.

Responses

POST /_inference/chat_completion/{inference_id}/_stream 
POST _inference/chat_completion/openai-completion/_stream
{
  "model": "gpt-4o",
  "messages": [
      {
          "role": "user",
          "content": "What is Elastic?"
      }
  ]
}
curl \
 --request POST 'http://api.example.com/_inference/chat_completion/{inference_id}/_stream' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '"{\n  \"model\": \"gpt-4o\",\n  \"messages\": [\n      {\n          \"role\": \"user\",\n          \"content\": \"What is Elastic?\"\n      }\n  ]\n}"'
Request examples
Run `POST _inference/chat_completion/openai-completion/_stream` to perform a chat completion on the example question with streaming. 
{
  "model": "gpt-4o",
  "messages": [
      {
          "role": "user",
          "content": "What is Elastic?"
      }
  ]
}
Run `POST POST _inference/chat_completion/openai-completion/_stream` to perform a chat completion using an Assistant message with `tool_calls`. 
{
  "messages": [
      {
          "role": "assistant",
          "content": "Let's find out what the weather is",
          "tool_calls": [ 
              {
                  "id": "call_KcAjWtAww20AihPHphUh46Gd",
                  "type": "function",
                  "function": {
                      "name": "get_current_weather",
                      "arguments": "{\"location\":\"Boston, MA\"}"
                  }
              }
          ]
      },
      { 
          "role": "tool",
          "content": "The weather is cold",
          "tool_call_id": "call_KcAjWtAww20AihPHphUh46Gd"
      }
  ]
}
Run `POST POST _inference/chat_completion/openai-completion/_stream` to perform a chat completion using a User message with `tools` and `tool_choice`. 
{
  "messages": [
      {
          "role": "user",
          "content": [
              {
                  "type": "text",
                  "text": "What's the price of a scarf?"
              }
          ]
      }
  ],
  "tools": [
      {
          "type": "function",
          "function": {
              "name": "get_current_price",
              "description": "Get the current price of a item",
              "parameters": {
                  "type": "object",
                  "properties": {
                      "item": {
                          "id": "123"
                      }
                  }
              }
          }
      }
  ],
  "tool_choice": {
      "type": "function",
      "function": {
          "name": "get_current_price"
      }
  }
}
Response examples (200)
A successful response when performing a chat completion task using a User message with `tools` and `tool_choice`. 
event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":"","role":"assistant"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":Elastic"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":" is"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}

(...)

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","usage":{"completion_tokens":28,"prompt_tokens":16,"total_tokens":44}}} 

event: message
data: [DONE]

Perform completion inference on the service Added in 8.11.0

POST /_inference/completion/{inference_id}
Api key auth Basic auth Bearer auth

Path parameters

Query parameters

  • timeout string

    Specifies the amount of time to wait for the inference request to complete.

    Values are -1 or 0.

application/json

Body

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • completion array[object] Required
      Hide completion attribute Show completion attribute object
POST /_inference/completion/{inference_id} 
POST _inference/completion/openai_chat_completions
{
  "input": "What is Elastic?"
}
curl \
 --request POST 'http://api.example.com/_inference/completion/{inference_id}' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '"{\n  \"input\": \"What is Elastic?\"\n}"'
Request example
Run `POST _inference/completion/openai_chat_completions` to perform a completion on the example question. 
{
  "input": "What is Elastic?"
}
Response examples (200)
A successful response from `POST _inference/completion/openai_chat_completions`. 
{
  "completion": [
    {
      "result": "Elastic is a company that provides a range of software solutions for search, logging, security, and analytics. Their flagship product is Elasticsearch, an open-source, distributed search engine that allows users to search, analyze, and visualize large volumes of data in real-time. Elastic also offers products such as Kibana, a data visualization tool, and Logstash, a log management and pipeline tool, as well as various other tools and solutions for data analysis and management."
    }
  ]
}

Get calendar configuration info Added in 6.2.0

GET /_ml/calendars/{calendar_id}
Api key auth Basic auth Bearer auth

Path parameters

  • calendar_id string Required

    A string that uniquely identifies a calendar. You can get information for multiple calendars by using a comma-separated list of ids or a wildcard expression. You can get information for all calendars by using _all or * or by omitting the calendar identifier.

Query parameters

  • from number

    Skips the specified number of calendars. This parameter is supported only when you omit the calendar identifier.

  • size number

    Specifies the maximum number of calendars to obtain. This parameter is supported only when you omit the calendar identifier.

application/json

Body

  • page object
    Hide page attributes Show page attributes object
    • from number

      Skips the specified number of items.

    • size number

      Specifies the maximum number of items to obtain.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • calendars array[object] Required
      Hide calendars attributes Show calendars attributes object
    • count number Required
GET /_ml/calendars/{calendar_id} 
curl \
 --request GET 'http://api.example.com/_ml/calendars/{calendar_id}' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '{"page":{"from":42.0,"size":42.0}}'

Create a calendar Added in 6.2.0

PUT /_ml/calendars/{calendar_id}
Api key auth Basic auth Bearer auth

Path parameters

  • calendar_id string Required

    A string that uniquely identifies a calendar.

application/json

Body

  • job_ids array[string]

    An array of anomaly detection job identifiers.

  • description string

    A description of the calendar.

Responses

PUT /_ml/calendars/{calendar_id} 
curl \
 --request PUT 'http://api.example.com/_ml/calendars/{calendar_id}' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '{"job_ids":["string"],"description":"string"}'

Delete a datafeed Added in 5.4.0

DELETE /_ml/datafeeds/{datafeed_id}
Api key auth Basic auth Bearer auth

Path parameters

  • datafeed_id string Required

    A numerical character string that uniquely identifies the datafeed. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters.

Query parameters

  • force boolean

    Use to forcefully delete a started datafeed; this method is quicker than stopping and deleting the datafeed.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • acknowledged boolean Required

      For a successful response, this value is always true. On failure, an exception is returned instead.
DELETE /_ml/datafeeds/{datafeed_id} 
curl \
 --request DELETE 'http://api.example.com/_ml/datafeeds/{datafeed_id}' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response when deleting a datafeed. 
{
  "acknowledged": true
}

Delete expired ML data Added in 5.4.0

DELETE /_ml/_delete_expired_data
Api key auth Basic auth Bearer auth

Delete all job results, model snapshots and forecast data that have exceeded their retention days period. Machine learning state documents that are not associated with any job are also deleted. You can limit the request to a single or set of anomaly detection jobs by using a job identifier, a group name, a comma-separated list of jobs, or a wildcard expression. You can delete expired data for all anomaly detection jobs by using _all, by specifying * as the <job_id>, or by omitting the <job_id>.

Query parameters

  • requests_per_second number

    The desired requests per second for the deletion processes. The default behavior is no throttling.

  • timeout string

    How long can the underlying delete processes run until they are canceled.

    Values are -1 or 0.

application/json

Body

  • requests_per_second number

    The desired requests per second for the deletion processes. The default behavior is no throttling.

  • timeout string

    A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
DELETE /_ml/_delete_expired_data 
curl \
 --request DELETE 'http://api.example.com/_ml/_delete_expired_data' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '{"requests_per_second":42.0,"timeout":"string"}'
Response examples (200)
A successful response when deleting expired and unused anomaly detection data. 
{
  "deleted": true
}

Get datafeeds configuration info Added in 5.5.0

GET /_ml/datafeeds
Api key auth Basic auth Bearer auth

You can get information for multiple datafeeds in a single API request by using a comma-separated list of datafeeds or a wildcard expression. You can get information for all datafeeds by using _all, by specifying * as the <feed_id>, or by omitting the <feed_id>. This API returns a maximum of 10,000 datafeeds.

Query parameters

  • allow_no_match boolean

    Specifies what to do when the request:

    1. Contains wildcard expressions and there are no datafeeds that match.
    2. Contains the _all string or no identifiers and there are no matches.
    3. Contains wildcard expressions and there are only partial matches.

    The default value is true, which returns an empty datafeeds array when there are no matches and the subset of results when there are partial matches. If this parameter is false, the request returns a 404 status code when there are no matches or only partial matches.

  • exclude_generated boolean

    Indicates if certain fields should be removed from the configuration on retrieval. This allows the configuration to be in an acceptable format to be retrieved and then added to another cluster.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • count number Required
    • datafeeds array[object] Required
      Hide datafeeds attributes Show datafeeds attributes object
      • aggregations object
      • authorization object
        Hide authorization attributes Show authorization attributes object
        • api_key object
          Hide api_key attributes Show api_key attributes object
          • id string Required

            The identifier for the API key.

          • name string Required

            The name of the API key.

        • roles array[string]

          If a user ID was used for the most recent update to the datafeed, its roles at the time of the update are listed in the response.

        • service_account string

          If a service account was used for the most recent update to the datafeed, the account name is listed in the response.

      • chunking_config object
        Hide chunking_config attributes Show chunking_config attributes object
        • mode string Required

          Values are auto, manual, or off.

        • time_span string

          A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • datafeed_id string Required
      • frequency string

        A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • indices array[string] Required
      • indexes array[string]
      • job_id string Required
      • max_empty_searches number
      • query_delay string

        A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • script_fields object
        Hide script_fields attribute Show script_fields attribute object
        • * object Additional properties
          Hide * attributes Show * attributes object
          • script object Required
            Hide script attributes Show script attributes object
            • source
            • id string
            • params object

              Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

            • lang
            • options object
          • ignore_failure boolean
      • scroll_size number
      • delayed_data_check_config object Required
        Hide delayed_data_check_config attributes Show delayed_data_check_config attributes object
        • check_window string

          A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • enabled boolean Required

          Specifies whether the datafeed periodically checks for delayed data.

      • runtime_mappings object
        Hide runtime_mappings attribute Show runtime_mappings attribute object
        • * object Additional properties
          Hide * attributes Show * attributes object
          • fields object

            For type composite

            Hide fields attribute Show fields attribute object
            • * object Additional properties
          • fetch_fields array[object]

            For type lookup

          • format string

            A custom format for date type runtime fields.

          • input_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • target_index string
          • script object
            Hide script attributes Show script attributes object
            • source
            • id string
            • params object

              Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

            • lang
            • options object
          • type string Required

            Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

      • indices_options object
        Hide indices_options attributes Show indices_options attributes object
        • allow_no_indices boolean

          If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.

        • expand_wildcards string | array[string]
        • ignore_unavailable boolean

          If true, missing or closed indices are not included in the response.

        • ignore_throttled boolean

          If true, concrete, expanded or aliased indices are ignored when frozen.

      • query object Required

        The Elasticsearch query domain-specific language (DSL). This value corresponds to the query object in an Elasticsearch search POST body. All the options that are supported by Elasticsearch can be used, as this object is passed verbatim to Elasticsearch. By default, this property has the following value: {"match_all": {"boost": 1}}.

        Query DSL
GET /_ml/datafeeds 
curl \
 --request GET 'http://api.example.com/_ml/datafeeds' \
 --header "Authorization: $API_KEY"

Get anomaly detection job results for influencers Added in 5.4.0

GET /_ml/anomaly_detectors/{job_id}/results/influencers
Api key auth Basic auth Bearer auth

Influencers are the entities that have contributed to, or are to blame for, the anomalies. Influencer results are available only if an influencer_field_name is specified in the job configuration.

Path parameters

  • job_id string Required

    Identifier for the anomaly detection job.

Query parameters

  • desc boolean

    If true, the results are sorted in descending order.

  • end string | number

    Returns influencers with timestamps earlier than this time. The default value means it is unset and results are not limited to specific timestamps.

  • exclude_interim boolean

    If true, the output excludes interim results. By default, interim results are included.

  • influencer_score number

    Returns influencers with anomaly scores greater than or equal to this value.

  • from number

    Skips the specified number of influencers.

  • size number

    Specifies the maximum number of influencers to obtain.

  • sort string

    Specifies the sort field for the requested influencers. By default, the influencers are sorted by the influencer_score value.

  • start string | number

    Returns influencers with timestamps after this time. The default value means it is unset and results are not limited to specific timestamps.

application/json

Body

  • page object
    Hide page attributes Show page attributes object
    • from number

      Skips the specified number of items.

    • size number

      Specifies the maximum number of items to obtain.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • count number Required
    • influencers array[object] Required

      Array of influencer objects

      Hide influencers attributes Show influencers attributes object
      • bucket_span number

        Time unit for seconds

      • influencer_score number Required

        A normalized score between 0-100, which is based on the probability of the influencer in this bucket aggregated across detectors. Unlike initial_influencer_score, this value is updated by a re-normalization process as new data is analyzed.

      • influencer_field_name string Required

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • influencer_field_value string Required

        The entity that influenced, contributed to, or was to blame for the anomaly.

      • initial_influencer_score number Required

        A normalized score between 0-100, which is based on the probability of the influencer aggregated across detectors. This is the initial value that was calculated at the time the bucket was processed.

      • is_interim boolean Required

        If true, this is an interim result. In other words, the results are calculated based on partial input data.

      • job_id string Required
      • probability number Required

        The probability that the influencer has this behavior, in the range 0 to 1. This value can be held to a high precision of over 300 decimal places, so the influencer_score is provided as a human-readable and friendly interpretation of this value.

      • result_type string Required

        Internal. This value is always set to influencer.

      • timestamp number

        Time unit for milliseconds

      • foo string

        Additional influencer properties are added, depending on the fields being analyzed. For example, if it’s analyzing user_name as an influencer, a field user_name is added to the result document. This information enables you to filter the anomaly results more easily.
GET /_ml/anomaly_detectors/{job_id}/results/influencers 
curl \
 --request GET 'http://api.example.com/_ml/anomaly_detectors/{job_id}/results/influencers' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '{"page":{"from":42.0,"size":42.0}}'

Evaluate data frame analytics Added in 7.3.0

POST /_ml/data_frame/_evaluate
Api key auth Basic auth Bearer auth

The API packages together commonly used evaluation metrics for various types of machine learning features. This has been designed for use on indexes created by data frame analytics. Evaluation requires both a ground truth field and an analytics result field to be present.

application/json

Body Required

  • evaluation object Required
    Hide evaluation attributes Show evaluation attributes object
    • classification object
      Hide classification attributes Show classification attributes object
      • actual_field string Required

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • predicted_field string

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • top_classes_field string

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • metrics object
        Hide metrics attributes Show metrics attributes object
        • auc_roc object
          Hide auc_roc attributes Show auc_roc attributes object
          • class_name string
          • include_curve boolean

            Whether or not the curve should be returned in addition to the score. Default value is false.

        • precision object

          Precision of predictions (per-class and average).

          Hide precision attribute Show precision attribute object
          • * object Additional properties
        • recall object

          Recall of predictions (per-class and average).

          Hide recall attribute Show recall attribute object
          • * object Additional properties
        • accuracy object

          Accuracy of predictions (per-class and overall).

          Hide accuracy attribute Show accuracy attribute object
          • * object Additional properties
        • multiclass_confusion_matrix object

          Multiclass confusion matrix.

          Hide multiclass_confusion_matrix attribute Show multiclass_confusion_matrix attribute object
          • * object Additional properties
    • outlier_detection object
      Hide outlier_detection attributes Show outlier_detection attributes object
      • actual_field string Required

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • predicted_probability_field string Required

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • metrics object
        Hide metrics attributes Show metrics attributes object
        • auc_roc object
          Hide auc_roc attributes Show auc_roc attributes object
          • class_name string
          • include_curve boolean

            Whether or not the curve should be returned in addition to the score. Default value is false.

        • precision object

          Precision of predictions (per-class and average).

          Hide precision attribute Show precision attribute object
          • * object Additional properties
        • recall object

          Recall of predictions (per-class and average).

          Hide recall attribute Show recall attribute object
          • * object Additional properties
        • confusion_matrix object

          Accuracy of predictions (per-class and overall).

          Hide confusion_matrix attribute Show confusion_matrix attribute object
          • * object Additional properties
    • regression object
      Hide regression attributes Show regression attributes object
      • actual_field string Required

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • predicted_field string Required

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • metrics object
        Hide metrics attributes Show metrics attributes object
        • mse object

          Average squared difference between the predicted values and the actual (ground truth) value. For more information, read this wiki article.

          Hide mse attribute Show mse attribute object
          • * object Additional properties
        • msle object
          Hide msle attribute Show msle attribute object
          • offset number

            Defines the transition point at which you switch from minimizing quadratic error to minimizing quadratic log error. Defaults to 1.

        • huber object
          Hide huber attribute Show huber attribute object
          • delta number

            Approximates 1/2 (prediction - actual)2 for values much less than delta and approximates a straight line with slope delta for values much larger than delta. Defaults to 1. Delta needs to be greater than 0.

        • r_squared object

          Proportion of the variance in the dependent variable that is predictable from the independent variables.

          Hide r_squared attribute Show r_squared attribute object
          • * object Additional properties
  • index string Required
  • query object

    An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

    External documentation

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • classification object
      Hide classification attributes Show classification attributes object
    • outlier_detection object
      Hide outlier_detection attributes Show outlier_detection attributes object
      • auc_roc object
        Hide auc_roc attributes Show auc_roc attributes object
      • precision object

        Set the different thresholds of the outlier score at where the metric is calculated.

        Hide precision attribute Show precision attribute object
        • * number Additional properties
      • recall object

        Set the different thresholds of the outlier score at where the metric is calculated.

        Hide recall attribute Show recall attribute object
        • * number Additional properties
      • confusion_matrix object

        Set the different thresholds of the outlier score at where the metrics (tp - true positive, fp - false positive, tn - true negative, fn - false negative) are calculated.

        Hide confusion_matrix attribute Show confusion_matrix attribute object
        • * object Additional properties
          Hide * attributes Show * attributes object
          • tp number Required

            True Positive

          • fp number Required

            False Positive

          • tn number Required

            True Negative

          • fn number Required

            False Negative

    • regression object
      Hide regression attributes Show regression attributes object
      • huber object
        Hide huber attribute Show huber attribute object
      • mse object
        Hide mse attribute Show mse attribute object
      • msle object
        Hide msle attribute Show msle attribute object
      • r_squared object
        Hide r_squared attribute Show r_squared attribute object
POST /_ml/data_frame/_evaluate 
POST _ml/data_frame/_evaluate
{
  "index": "animal_classification",
  "evaluation": {
    "classification": {
      "actual_field": "animal_class",
      "predicted_field": "ml.animal_class_prediction",
      "metrics": {
        "multiclass_confusion_matrix": {}
      }
    }
  }
}
curl \
 --request POST 'http://api.example.com/_ml/data_frame/_evaluate' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '"{\n  \"index\": \"animal_classification\",\n  \"evaluation\": {\n    \"classification\": {\n      \"actual_field\": \"animal_class\",\n      \"predicted_field\": \"ml.animal_class_prediction\",\n      \"metrics\": {\n        \"multiclass_confusion_matrix\": {}\n      }\n    }\n  }\n}"'
Request examples
Run `POST _ml/data_frame/_evaluate` to evaluate a a classification job for an annotated index. The `actual_field` contains the ground truth for classification. The `predicted_field` contains the predicted value calculated by the classification analysis. 
{
  "index": "animal_classification",
  "evaluation": {
    "classification": {
      "actual_field": "animal_class",
      "predicted_field": "ml.animal_class_prediction",
      "metrics": {
        "multiclass_confusion_matrix": {}
      }
    }
  }
}
Run `POST _ml/data_frame/_evaluate` to evaluate a classification job with AUC ROC metrics for an annotated index. The `actual_field` contains the ground truth value for the actual animal classification. This is required in order to evaluate results. The `class_name` specifies the class name that is treated as positive during the evaluation, all the other classes are treated as negative. 
{
  "index": "animal_classification",
  "evaluation": {
    "classification": {
      "actual_field": "animal_class",
      "metrics": {
        "auc_roc": {
          "class_name": "dog"
        }
      }
    }
  }
}
Run `POST _ml/data_frame/_evaluate` to evaluate an outlier detection job for an annotated index. 
{
  "index": "my_analytics_dest_index",
  "evaluation": {
    "outlier_detection": {
      "actual_field": "is_outlier",
      "predicted_probability_field": "ml.outlier_score"
    }
  }
}
Run `POST _ml/data_frame/_evaluate` to evaluate the testing error of a regression job for an annotated index. The term query in the body limits evaluation to be performed on the test split only. The `actual_field` contains the ground truth for house prices. The `predicted_field` contains the house price calculated by the regression analysis. 
{
  "index": "house_price_predictions",
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "ml.is_training": false
          }
        }
      ]
    }
  },
  "evaluation": {
    "regression": {
      "actual_field": "price",
      "predicted_field": "ml.price_prediction",
      "metrics": {
        "r_squared": {},
        "mse": {},
        "msle": {
          "offset": 10
        },
        "huber": {
          "delta": 1.5
        }
      }
    }
  }
}
Run `POST _ml/data_frame/_evaluate` to evaluate the training error of a regression job for an annotated index. The term query in the body limits evaluation to be performed on the training split only. The `actual_field` contains the ground truth for house prices. The `predicted_field` contains the house price calculated by the regression analysis. 
{
  "index": "house_price_predictions",
  "query": {
    "term": {
      "ml.is_training": {
        "value": true
      }
    }
  },
  "evaluation": {
    "regression": {
      "actual_field": "price",
      "predicted_field": "ml.price_prediction",
      "metrics": {
        "r_squared": {},
        "mse": {},
        "msle": {},
        "huber": {}
      }
    }
  }
}
Response examples (200)
A succesful response from `POST _ml/data_frame/_evaluate` to evaluate a classification analysis job for an annotated index. The `actual_class` contains the name of the class the analysis tried to predict. The `actual_class_doc_count` is the number of documents in the index belonging to the `actual_class`. The `predicted_classes` object contains the list of the predicted classes and the number of predictions associated with the class. 
{
  "classification": {
    "multiclass_confusion_matrix": {
      "confusion_matrix": [
        {
          "actual_class": "cat",
          "actual_class_doc_count": 12,
          "predicted_classes": [
            {
              "predicted_class": "cat",
              "count": 12
            },
            {
              "predicted_class": "dog",
              "count": 0
            }
          ],
          "other_predicted_class_doc_count": 0
        },
        {
          "actual_class": "dog",
          "actual_class_doc_count": 11,
          "predicted_classes": [
            {
              "predicted_class": "dog",
              "count": 7
            },
            {
              "predicted_class": "cat",
              "count": 4
            }
          ],
          "other_predicted_class_doc_count": 0
        }
      ],
      "other_actual_class_count": 0
    }
  }
}
A succesful response from `POST _ml/data_frame/_evaluate` to evaluate a classification analysis job with the AUC ROC metrics for an annotated index. 
{
  "classification": {
    "auc_roc": {
      "value": 0.8941788639536681
    }
  }
}
A successful response from `POST _ml/data_frame/_evaluate` to evaluate an outlier detection job. 
{
  "outlier_detection": {
    "auc_roc": {
      "value": 0.9258475774641445
    },
    "confusion_matrix": {
      "0.25": {
        "tp": 5,
        "fp": 9,
        "tn": 204,
        "fn": 5
      },
      "0.5": {
        "tp": 1,
        "fp": 5,
        "tn": 208,
        "fn": 9
      },
      "0.75": {
        "tp": 0,
        "fp": 4,
        "tn": 209,
        "fn": 10
      }
    },
    "precision": {
      "0.25": 0.35714285714285715,
      "0.5": 0.16666666666666666,
      "0.75": 0
    },
    "recall": {
      "0.25": 0.5,
      "0.5": 0.1,
      "0.75": 0
    }
  }
}

Get data frame analytics job stats Added in 7.3.0

GET /_ml/data_frame/analytics/_stats
Api key auth Basic auth Bearer auth

Query parameters

  • allow_no_match boolean

    Specifies what to do when the request:

    1. Contains wildcard expressions and there are no data frame analytics jobs that match.
    2. Contains the _all string or no identifiers and there are no matches.
    3. Contains wildcard expressions and there are only partial matches.

    The default value returns an empty data_frame_analytics array when there are no matches and the subset of results when there are partial matches. If this parameter is false, the request returns a 404 status code when there are no matches or only partial matches.

  • from number

    Skips the specified number of data frame analytics jobs.

  • size number

    Specifies the maximum number of data frame analytics jobs to obtain.

  • verbose boolean

    Defines whether the stats response should be verbose.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • count number Required
    • data_frame_analytics array[object] Required

      An array of objects that contain usage information for data frame analytics jobs, which are sorted by the id value in ascending order.

      Hide data_frame_analytics attributes Show data_frame_analytics attributes object
      • analysis_stats object
        Hide analysis_stats attributes Show analysis_stats attributes object
        • classification_stats object
          Hide classification_stats attributes Show classification_stats attributes object
          • hyperparameters object Required
            Hide hyperparameters attributes Show hyperparameters attributes object
            • alpha number

              Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This parameter affects loss calculations by acting as a multiplier of the tree depth. Higher alpha values result in shallower trees and faster training times. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to zero.

            • lambda number

              Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularization term which applies to leaf weights of the individual trees in the forest. A high lambda value causes training to favor small leaf weights. This behavior makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. A small lambda value results in large individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

            • gamma number

              Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. A high gamma value causes training to prefer small trees. A small gamma value results in larger individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

            • eta number

              Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, larger forests cause slower training. By default, this value is calculated during hyperparameter optimization. It must be a value between 0.001 and 1.

            • eta_growth_rate_per_tree number

              Advanced configuration option. Specifies the rate at which eta increases for each new tree that is added to the forest. For example, a rate of 1.05 increases eta by 5% for each extra tree. By default, this value is calculated during hyperparameter optimization. It must be between 0.5 and 2.

            • feature_bag_fraction number

              Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.

            • downsample_factor number

              Advanced configuration option. Controls the fraction of data that is used to compute the derivatives of the loss function for tree training. A small value results in the use of a small fraction of the data. If this value is set to be less than 1, accuracy typically improves. However, too small a value may result in poor convergence for the ensemble and so require more trees. By default, this value is calculated during hyperparameter optimization. It must be greater than zero and less than or equal to 1.

            • max_attempts_to_add_tree number

              If the algorithm fails to determine a non-trivial tree (more than a single leaf), this parameter determines how many of such consecutive failures are tolerated. Once the number of attempts exceeds the threshold, the forest training stops.

            • max_optimization_rounds_per_hyperparameter number

              Advanced configuration option. A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter. By default, this value is calculated during hyperparameter optimization.

            • max_trees number

              Advanced configuration option. Defines the maximum number of decision trees in the forest. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.

            • num_folds number

              The maximum number of folds for the cross-validation procedure.

            • num_splits_per_feature number

              Determines the maximum number of splits for every feature that can occur in a decision tree when the tree is trained.

            • soft_tree_depth_limit number

              Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This soft limit combines with the soft_tree_depth_tolerance to penalize trees that exceed the specified depth; the regularized loss increases quickly beyond this depth. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.

            • soft_tree_depth_tolerance number

              Advanced configuration option. This option controls how quickly the regularized loss increases when the tree depth exceeds soft_tree_depth_limit. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.01.

          • iteration number Required

            The number of iterations on the analysis.

          • timestamp number

            Time unit for milliseconds

          • timing_stats object Required
            Hide timing_stats attributes Show timing_stats attributes object
          • validation_loss object Required
            Hide validation_loss attributes Show validation_loss attributes object
            • fold_values array[string] Required

              Validation loss values for every added decision tree during the forest growing procedure.

            • loss_type string Required

              The type of the loss metric. For example, binomial_logistic.

        • outlier_detection_stats object
          Hide outlier_detection_stats attributes Show outlier_detection_stats attributes object
          • parameters object Required
            Hide parameters attributes Show parameters attributes object
            • compute_feature_influence boolean

              Specifies whether the feature influence calculation is enabled.

            • feature_influence_threshold number

              The minimum outlier score that a document needs to have in order to calculate its feature influence score. Value range: 0-1

            • method string

              The method that outlier detection uses. Available methods are lof, ldof, distance_kth_nn, distance_knn, and ensemble. The default value is ensemble, which means that outlier detection uses an ensemble of different methods and normalises and combines their individual outlier scores to obtain the overall outlier score.

            • n_neighbors number

              Defines the value for how many nearest neighbors each method of outlier detection uses to calculate its outlier score. When the value is not set, different values are used for different ensemble members. This default behavior helps improve the diversity in the ensemble; only override it if you are confident that the value you choose is appropriate for the data set.

            • outlier_fraction number

              The proportion of the data set that is assumed to be outlying prior to outlier detection. For example, 0.05 means it is assumed that 5% of values are real outliers and 95% are inliers.

            • standardization_enabled boolean

              If true, the following operation is performed on the columns before computing outlier scores: (x_i - mean(x_i)) / sd(x_i).

          • timestamp number

            Time unit for milliseconds

          • timing_stats object Required
            Hide timing_stats attributes Show timing_stats attributes object
        • regression_stats object
          Hide regression_stats attributes Show regression_stats attributes object
          • hyperparameters object Required
            Hide hyperparameters attributes Show hyperparameters attributes object
            • alpha number

              Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This parameter affects loss calculations by acting as a multiplier of the tree depth. Higher alpha values result in shallower trees and faster training times. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to zero.

            • lambda number

              Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularization term which applies to leaf weights of the individual trees in the forest. A high lambda value causes training to favor small leaf weights. This behavior makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. A small lambda value results in large individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

            • gamma number

              Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. A high gamma value causes training to prefer small trees. A small gamma value results in larger individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

            • eta number

              Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, larger forests cause slower training. By default, this value is calculated during hyperparameter optimization. It must be a value between 0.001 and 1.

            • eta_growth_rate_per_tree number

              Advanced configuration option. Specifies the rate at which eta increases for each new tree that is added to the forest. For example, a rate of 1.05 increases eta by 5% for each extra tree. By default, this value is calculated during hyperparameter optimization. It must be between 0.5 and 2.

            • feature_bag_fraction number

              Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.

            • downsample_factor number

              Advanced configuration option. Controls the fraction of data that is used to compute the derivatives of the loss function for tree training. A small value results in the use of a small fraction of the data. If this value is set to be less than 1, accuracy typically improves. However, too small a value may result in poor convergence for the ensemble and so require more trees. By default, this value is calculated during hyperparameter optimization. It must be greater than zero and less than or equal to 1.

            • max_attempts_to_add_tree number

              If the algorithm fails to determine a non-trivial tree (more than a single leaf), this parameter determines how many of such consecutive failures are tolerated. Once the number of attempts exceeds the threshold, the forest training stops.

            • max_optimization_rounds_per_hyperparameter number

              Advanced configuration option. A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter. By default, this value is calculated during hyperparameter optimization.

            • max_trees number

              Advanced configuration option. Defines the maximum number of decision trees in the forest. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.

            • num_folds number

              The maximum number of folds for the cross-validation procedure.

            • num_splits_per_feature number

              Determines the maximum number of splits for every feature that can occur in a decision tree when the tree is trained.

            • soft_tree_depth_limit number

              Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This soft limit combines with the soft_tree_depth_tolerance to penalize trees that exceed the specified depth; the regularized loss increases quickly beyond this depth. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.

            • soft_tree_depth_tolerance number

              Advanced configuration option. This option controls how quickly the regularized loss increases when the tree depth exceeds soft_tree_depth_limit. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.01.

          • iteration number Required

            The number of iterations on the analysis.

          • timestamp number

            Time unit for milliseconds

          • timing_stats object Required
            Hide timing_stats attributes Show timing_stats attributes object
          • validation_loss object Required
            Hide validation_loss attributes Show validation_loss attributes object
            • fold_values array[string] Required

              Validation loss values for every added decision tree during the forest growing procedure.

            • loss_type string Required

              The type of the loss metric. For example, binomial_logistic.

      • assignment_explanation string

        For running jobs only, contains messages relating to the selection of a node to run the job.

      • data_counts object Required
        Hide data_counts attributes Show data_counts attributes object
        • skipped_docs_count number Required

          The number of documents that are skipped during the analysis because they contained values that are not supported by the analysis. For example, outlier detection does not support missing fields so it skips documents with missing fields. Likewise, all types of analysis skip documents that contain arrays with more than one element.

        • test_docs_count number Required

          The number of documents that are not used for training the model and can be used for testing.

        • training_docs_count number Required

          The number of documents that are used for training the model.

      • id string Required
      • memory_usage object Required
        Hide memory_usage attributes Show memory_usage attributes object
        • memory_reestimate_bytes number

          This value is present when the status is hard_limit and it is a new estimate of how much memory the job needs.

        • peak_usage_bytes number Required

          The number of bytes used at the highest peak of memory usage.

        • status string Required

          The memory usage status.

        • timestamp number

          Time unit for milliseconds

      • node object
        Hide node attributes Show node attributes object
      • progress array[object] Required

        The progress report of the data frame analytics job by phase.

        Hide progress attributes Show progress attributes object
        • phase string Required

          Defines the phase of the data frame analytics job.

        • progress_percent number Required

          The progress that the data frame analytics job has made expressed in percentage.

      • state string Required

        Values are started, stopped, starting, stopping, or failed.
GET /_ml/data_frame/analytics/_stats 
curl \
 --request GET 'http://api.example.com/_ml/data_frame/analytics/_stats' \
 --header "Authorization: $API_KEY"

Start a data frame analytics job Added in 7.3.0

POST /_ml/data_frame/analytics/{id}/_start
Api key auth Basic auth Bearer auth

A data frame analytics job can be started and stopped multiple times throughout its lifecycle. If the destination index does not exist, it is created automatically the first time you start the data frame analytics job. The index.number_of_shards and index.number_of_replicas settings for the destination index are copied from the source index. If there are multiple source indices, the destination index copies the highest setting values. The mappings for the destination index are also copied from the source indices. If there are any mapping conflicts, the job fails to start. If the destination index exists, it is used as is. You can therefore set up the destination index in advance with custom settings and mappings.

Path parameters

  • id string Required

    Identifier for the data frame analytics job. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters.

Query parameters

  • timeout string

    Controls the amount of time to wait until the data frame analytics job starts.

    Values are -1 or 0.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
POST /_ml/data_frame/analytics/{id}/_start 
curl \
 --request POST 'http://api.example.com/_ml/data_frame/analytics/{id}/_start' \
 --header "Authorization: $API_KEY"

Clear trained model deployment cache Added in 8.5.0

POST /_ml/trained_models/{model_id}/deployment/cache/_clear
Api key auth Basic auth Bearer auth

Cache will be cleared on all nodes where the trained model is assigned. A trained model deployment may have an inference cache enabled. As requests are handled by each allocated node, their responses may be cached on that individual node. Calling this API clears the caches without restarting the deployment.

Path parameters

  • model_id string Required

    The unique identifier of the trained model.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
POST /_ml/trained_models/{model_id}/deployment/cache/_clear 
curl \
 --request POST 'http://api.example.com/_ml/trained_models/{model_id}/deployment/cache/_clear' \
 --header "Authorization: $API_KEY"
Response examples (200)
A successful response when clearing the inference cache. 
{
  "cleared": true
}

Migration