Creates an inference endpoint to perform an inference task.

Requestedit

PUT /_inference/<task_type>/<inference_id>

Prerequisitesedit

Descriptionedit

The create inference API enables you to create an inference endpoint and configure a machine learning model to perform a specific inference task.

The following services are available through the inference API:

Path parametersedit

Request bodyedit

Examplesedit

This section contains example API calls for every service type.

Cohere serviceedit

The following example shows how to create an inference endpoint called cohere-embeddings to perform a text_embedding task type.

resp = client.inference.put_model(
    task_type="text_embedding",
    inference_id="cohere-embeddings",
    body={
        "service": "cohere",
        "service_settings": {
            "api_key": "<api_key>",
            "model_id": "embed-english-light-v3.0",
            "embedding_type": "byte",
        },
    },
)
print(resp)

PUT _inference/text_embedding/cohere-embeddings
{
    "service": "cohere",
    "service_settings": {
        "api_key": "<api_key>",
        "model_id": "embed-english-light-v3.0",
        "embedding_type": "byte"
    }
}

The following example shows how to create an inference endpoint called cohere-rerank to perform a rerank task type.

PUT _inference/rerank/cohere-rerank
{
    "service": "cohere",
    "service_settings": {
        "api_key": "<API-KEY>",
        "model_id": "rerank-english-v3.0"
    },
    "task_settings": {
        "top_n": 10,
        "return_documents": true
    }
}

For more examples, also review the Cohere documentation.

E5 via the elasticsearch serviceedit

The following example shows how to create an inference endpoint called my-e5-model to perform a text_embedding task type.

resp = client.inference.put_model(
    task_type="text_embedding",
    inference_id="my-e5-model",
    body={
        "service": "elasticsearch",
        "service_settings": {
            "num_allocations": 1,
            "num_threads": 1,
            "model_id": ".multilingual-e5-small",
        },
    },
)
print(resp)

PUT _inference/text_embedding/my-e5-model
{
  "service": "elasticsearch",
  "service_settings": {
    "num_allocations": 1,
    "num_threads": 1,
    "model_id": ".multilingual-e5-small" 
  }
}

ELSER serviceedit

The following example shows how to create an inference endpoint called my-elser-model to perform a sparse_embedding task type.

resp = client.inference.put_model(
    task_type="sparse_embedding",
    inference_id="my-elser-model",
    body={
        "service": "elser",
        "service_settings": {"num_allocations": 1, "num_threads": 1},
    },
)
print(resp)

PUT _inference/sparse_embedding/my-elser-model
{
  "service": "elser",
  "service_settings": {
    "num_allocations": 1,
    "num_threads": 1
  }
}

Example response:

{
  "inference_id": "my-elser-model",
  "task_type": "sparse_embedding",
  "service": "elser",
  "service_settings": {
    "num_allocations": 1,
    "num_threads": 1
  },
  "task_settings": {}
}

Hugging Face serviceedit

The following example shows how to create an inference endpoint called hugging-face-embeddings to perform a text_embedding task type.

resp = client.inference.put_model(
    task_type="text_embedding",
    inference_id="hugging-face-embeddings",
    body={
        "service": "hugging_face",
        "service_settings": {
            "api_key": "<access_token>",
            "url": "<url_endpoint>",
        },
    },
)
print(resp)

PUT _inference/text_embedding/hugging-face-embeddings
{
  "service": "hugging_face",
  "service_settings": {
    "api_key": "<access_token>", 
    "url": "<url_endpoint>" 
  }
}

Create a new inference endpoint on the Hugging Face endpoint page to get an endpoint URL. Select the model you want to use on the new endpoint creation page - for example intfloat/e5-small-v2 - then select the Sentence Embeddings task under the Advanced configuration section. Create the endpoint. Copy the URL after the endpoint initialization has been finished.

The list of recommended models for the Hugging Face service:

Models uploaded by Eland via the elasticsearch serviceedit

The following example shows how to create an inference endpoint called my-msmarco-minilm-model to perform a text_embedding task type.

resp = client.inference.put_model(
    task_type="text_embedding",
    inference_id="my-msmarco-minilm-model",
    body={
        "service": "elasticsearch",
        "service_settings": {
            "num_allocations": 1,
            "num_threads": 1,
            "model_id": "msmarco-MiniLM-L12-cos-v5",
        },
    },
)
print(resp)

PUT _inference/text_embedding/my-msmarco-minilm-model
{
  "service": "elasticsearch",
  "service_settings": {
    "num_allocations": 1,
    "num_threads": 1,
    "model_id": "msmarco-MiniLM-L12-cos-v5" 
  }
}

OpenAI serviceedit

The following example shows how to create an inference endpoint called openai-embeddings to perform a text_embedding task type.

PUT _inference/text_embedding/openai-embeddings
{
    "service": "openai",
    "service_settings": {
        "api_key": "<api_key>",
        "model_id": "text-embedding-ada-002"
    }
}

The next example shows how to create an inference endpoint called openai-completion to perform a completion task type.

PUT _inference/completion/openai-completion
{
    "service": "openai",
    "service_settings": {
        "api_key": "<api_key>",
        "model_id": "gpt-3.5-turbo"
    }
}

Azure OpenAI serviceedit

The following example shows how to create an inference endpoint called azure_openai_embeddings to perform a text_embedding task type. Note that we do not specify a model here, as it is defined already via our Azure OpenAI deployment.

The list of embeddings models that you can choose from in your deployment can be found in the Azure models documentation.

PUT _inference/text_embedding/azure_openai_embeddings
{
    "service": "azureopenai",
    "service_settings": {
        "api_key": "<api_key>",
        "resource_name": "<resource_name>",
        "deployment_id": "<deployment_id>",
        "api_version": "2024-02-01"
    }
}