Multimodal search in Elasticsearch

Multimodal search finds results by meaning across more than one content type. Common use cases include text-to-image product search, finding similar images, searching media libraries by natural language, and retrieving screenshots or document pages with text or image queries.

Multimodal search builds on vector search. A multimodal embedding model maps each supported input into a dense vector so content with similar meaning is nearby in vector space, even when the media types differ.

Multimodal embedding models map different media types into a shared vector space. Text, images, and other supported modalities (such as audio, video, or PDF, depending on the model) become dense vectors that live in the same space. That shared space is what makes cross-modal retrieval possible: a text query can match an image embedding, and an image query can match text or other images.

Use the same model (and compatible endpoint settings) at ingest and at search time. Mixing models breaks similarity comparisons because the vector spaces are not interchangeable.

In Elasticsearch, the model is exposed through an inference endpoint that uses the embedding task type. The endpoint determines which modalities you can index and query.

Common multimodal search use cases include:

Product and catalog visual search

Shoppers describe an item in natural language ("leather crossbody bag with gold zipper") and retrieve matching product photos, even when titles and tags are incomplete. Combine with filters for price, brand, or availability.

Similar-image and reverse image search

A user uploads a photo or selects an existing asset and finds visually similar products, duplicates, or near-duplicates in a catalog or media library.

Digital asset management

Search large image and media collections by meaning instead of filenames or manual tags. For example, find campaign assets that match a brief written in plain language.

Multilingual text-to-image search

Query an image index in one language and retrieve the same visual results you would get in another, when the embedding model is trained for multilingual text-image matching.

PDF and document-page search

Embed document pages (or PDFs) so queries can match layout and visual content (diagrams, tables, scanned pages), not only extracted plain text.

Multimodal retrieval for RAG

Retrieve images, charts, or document pages alongside text passages so a downstream generative model can ground answers in visual evidence as well as prose.

Compare the multimodal embedding models available with Elasticsearch:

For the full Jina catalog, deployment matrix, and input examples, refer to Jina models. For EIS availability by stack version, refer to Supported models on EIS.

You can access Jina models in these ways:

• Elastic Inference Service (EIS): Elastic hosts the model. Use this when you want managed inference without provisioning ML nodes.
• Jina API: The model runs on the hosted Jina platform. Use this when you want Jina-hosted inference outside EIS.
• On-prem: You run the model in Docker on your own infrastructure. Use this for air-gapped, offline, or compliance scenarios.
• Cloud marketplace endpoints: Call models you deployed in your cloud provider account. Use this when the model runs in your AWS, Azure, or Google Cloud account.

The semantic field type simplifies semantic and multimodal search across text, images, audio, video, and PDF files. With a compatible multimodal embedding model, you can search from any supported input type to any other supported input type. The field automatically:

• Generates embeddings when you index field values, without an ingest pipeline or inference processor.
• Splits long text into smaller passages, called chunks.
• Indexes the generated embeddings using default index options that optimize for common use cases.
• Searches the embeddings generated for each value or text chunk.

For field parameters, defaults, supported input types, and limitations, refer to the semantic field documentation.

Here's an example using the Jina Embeddings v5 Omni Small endpoint:

				PUT my-multimodal-index
					{
  "mappings": {
    "properties": {
      "content": {
        "type": "semantic",
        "inference_id": ".jina-embeddings-v5-omni-small"
      }
    }
  }
}
		

1. Required. A semantic field has no default inference endpoint. You must specify an embedding endpoint ID.

• Tutorial: Build multimodal search with a semantic field: Index images and search them with text, image, and PDF input
• semantic field type: Review the mapping reference of the semantic field tpye.

• Jina models: Learn about Jina models, how to deploy them, and how to structure multimodal input payloads.
• Elastic Inference Service: Learn how to use hosted inference without managing ML nodes.
• Inference API: Learn how to create and manage inference endpoints.