Perform dense embedding inference on the service Technical preview

POST /_inference/embedding/{inference_id}
Api key auth

Path parameters

  • inference_id string Required

    The inference Id

Query parameters

application/json

Body Required

  • input string | array[string] | object | array[object] Required

    Inference input. Either a string, an array of strings, a content object, or an array of content objects. content objects may contain a single item or an array of items. Models that support multiple items per content object will return a single embedding for each content object, regardless of how many items it contains.

    string example:

    "input": "Some text"

    string array example:

    "input": ["Some text", "Some more text"]

    content object example:

    "input": {
    "content": {
      "type": "image",
      "format": "base64",
      "value": "data:image/jpeg;base64,..."
    }
  }

    content object array example:

    "input": [
  {
    "content": {
      "type": "text",
      "format": "text",
      "value": "Some text to generate an embedding"
    }
  },
  {
    "content": {
      "type": "image",
      "format": "base64",
      "value": "data:image/jpeg;base64,..."
    }
  }
]

    Multiple items in one content object example:

    "input": [
  {
    "content": [
      {
        "type": "image",
        "format": "base64",
        "value": "data:image/jpeg;base64,..."
      },
      {
        "type": "text",
        "value": "Some text to create an embedding"
      }
    ]
  }
]
    One of:
    string-1 string array-2 array[string] EmbeddingContentObject object array-2 array[object]
  • input_type string

    The input data type for the embedding model. Possible values include:

    • SEARCH
    • INGEST
    • CLASSIFICATION
    • CLUSTERING

    Not all models support all values. Unsupported values will trigger a validation exception. Accepted values depend on the configured inference service, refer to the relevant service-specific documentation for more info.


    The input_type parameter specified on the root level of the request body will take precedence over the input_type parameter specified in task_settings.

  • task_settings object

    Task settings for the individual inference request. These settings are specific to the you specified and override the task settings specified when initializing the service.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • embeddings_bytes array[object]
      Hide embeddings_bytes attribute Show embeddings_bytes attribute object

      The dense embedding result object for byte representation

      • embedding array[number] Required

        Dense Embedding results containing bytes are represented as Dense Vectors of bytes.

    • embeddings_bits array[object]
      Hide embeddings_bits attribute Show embeddings_bits attribute object

      The dense embedding result object for byte representation

      • embedding array[number] Required

        Dense Embedding results containing bytes are represented as Dense Vectors of bytes.

    • embeddings array[object]
      Hide embeddings attribute Show embeddings attribute object

      The dense embedding result object for float representation

      • embedding array[number] Required

        Dense Embedding results are represented as Dense Vectors of floats.
POST /_inference/embedding/{inference_id} 
POST _inference/embedding/my-multimodal-endpoint
{
  "input": [
      {
          "content": {
              "type": "image",
              "format": "base64",
              "value": "data:image/jpeg;base64,..."
          }
      },
      {
          "content": {
              "type": "text",
              "value": "Some text to create an embedding"
          }
      }
  ]
}
resp = client.inference.embedding(
    inference_id="my-multimodal-endpoint",
    embedding={
        "input": [
            {
                "content": {
                    "type": "image",
                    "format": "base64",
                    "value": "data:image/jpeg;base64,..."
                }
            },
            {
                "content": {
                    "type": "text",
                    "value": "Some text to create an embedding"
                }
            }
        ]
    },
)
const response = await client.inference.embedding({
  inference_id: "my-multimodal-endpoint",
  embedding: {
    input: [
      {
        content: {
          type: "image",
          format: "base64",
          value: "data:image/jpeg;base64,...",
        },
      },
      {
        content: {
          type: "text",
          value: "Some text to create an embedding",
        },
      },
    ],
  },
});
response = client.inference.embedding(
  inference_id: "my-multimodal-endpoint",
  body: {
    "input": [
      {
        "content": {
          "type": "image",
          "format": "base64",
          "value": "data:image/jpeg;base64,..."
        }
      },
      {
        "content": {
          "type": "text",
          "value": "Some text to create an embedding"
        }
      }
    ]
  }
)
$resp = $client->inference()->embedding([
    "inference_id" => "my-multimodal-endpoint",
    "body" => [
        "input" => array(
            [
                "content" => [
                    "type" => "image",
                    "format" => "base64",
                    "value" => "data:image/jpeg;base64,...",
                ],
            ],
            [
                "content" => [
                    "type" => "text",
                    "value" => "Some text to create an embedding",
                ],
            ],
        ),
    ],
]);
curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"input":[{"content":{"type":"image","format":"base64","value":"data:image/jpeg;base64,..."}},{"content":{"type":"text","value":"Some text to create an embedding"}}]}' "$ELASTICSEARCH_URL/_inference/embedding/my-multimodal-endpoint"
client.inference().embedding(e -> e
    .inferenceId("my-multimodal-endpoint")
    .embedding(em -> em
        .input(i -> i
            .content(List.of(EmbeddingContentObject.of(emb -> emb
                    .content(c -> c
                        .type(EmbeddingContentType.Image)
                        .format(EmbeddingContentFormat.Base64)
                        .value("data:image/jpeg;base64,...")
                    )
                ),EmbeddingContentObject.of(emb -> emb
                    .content(c -> c
                        .type(EmbeddingContentType.Text)
                        .value("Some text to create an embedding")
                    )
                )))
        )
    )
);
Request examples
Run `POST _inference/embedding/my-multimodal-endpoint` to generate embeddings from the example text and image 
{
  "input": [
      {
          "content": {
              "type": "image",
              "format": "base64",
              "value": "data:image/jpeg;base64,..."
          }
      },
      {
          "content": {
              "type": "text",
              "value": "Some text to create an embedding"
          }
      }
  ]
}
Run `POST _inference/embedding/my-text-only-endpoint` to generate embeddings from the example text 
{
  "input": ["The first text", "The second text"]
}
Run `POST _inference/embedding/my-multimodal-endpoint` to generate a single embedding from the example text and image 
{
  "input": [
    {
      "content": [
        {
          "type": "image",
          "format": "base64",
          "value": "data:image/jpeg;base64,..."
        },
        {
          "type": "text",
          "value": "Some text to create an embedding"
        }
      ]
    }
  ]
}
Response examples (200)
An abbreviated response from `POST _inference/embedding/my-multimodal-endpoint`. 
{
  "embeddings": [
    {
      "embedding": [
        -0.0189209,
        -0.04174805,
        0.00854492,
        0.01556396,
        0.01928711,
        -0.00616455,
        -0.00460815,
        0.01477051,
        -0.00656128,
        0.05419922
      ]
    },
    {
      "embedding": [
        -0.01379395,
        -0.02368164,
        0.01068115,
        0.0279541,
        0.01043701,
        -7.7057E-4,
        0.04150391,
        0.00836182,
        -0.01135254,
        0.0246582
      ]
    }
  ]
}
An abbreviated response from `POST _inference/embedding/my-text-only-endpoint`. 
{
  "embeddings": [
    {
      "embedding": [
        0.00854492,
        -0.00616455,
        -0.0189209,
        0.01556396,
        -0.00460815,
        0.01477051,
        -0.04174805,
        0.01928711,
        -0.00656128,
        0.05419922
      ]
    },
    {
      "embedding": [
        -0.01135254,
        0.0279541,
        -0.02368164,
        0.01068115,
        0.01043701,
        0.04150391,
        0.00836182,
        -7.7057E-4,
        -0.01379395,
        0.0246582
      ]
    }
  ]
}