Multistage retrieval: The challenge of modern search

Modern search has evolved beyond simple keyword matching. Today's search applications need to understand intent, handle natural language, and combine multiple ranking signals to deliver the best results.

Retrieval of the most relevant results happens in multiple stages, with each stage gradually refining the result set. This wasn’t the case in the past, where most use cases would require one or two stages of retrieval: an initial query to get results and a potential rescoring phase.

We start with an initial retrieval, where we cast a wide net to gather results that are relevant to our query. Since we need to sieve through all the data, we should use techniques that return results fast, even when we index billions of documents.

We therefore employ trusted techniques, such as lexical search that Elasticsearch has supported and optimized since the beginning, or vector search, where Elasticsearch excels in speed and accuracy.

Lexical search using BM25 is quite fast and best at exact term matching or phrase matching, and vector or semantic search is better suited for handling natural language queries. Hybrid search combines lexical and vector search results to bring the best from both. The challenge that hybrid search solves is that vector and lexical search have completely different and incompatible scoring functions which produce values in different intervals, following different distributions. A vector search score close to 1 can mean a very close match, but it doesn’t mean the same for lexical search. Hybrid search methods, such as reciprocal rank fusion (RRF) and linear combination of scores, assign new scores that blend the original scores from lexical and vector search.

After hybrid search, we can employ techniques such as semantic reranking and Learning To Rank (LTR), which use specialized machine learning models to rerank the result.

With our most relevant results, we can use large language models (LLMs) to further enrich our response or pass the most relevant results as context to LLMs in agentic workflows in tools such as Elastic Agent Builder.

ES|QL is able to handle all these stages of retrieval. By design, ES|QL is a piped language, where each command transforms the input and sends the output to the next command. Each stage of retrieval is represented by one or more consecutive ES|QL commands. In this article, we show how each stage is supported in ES|QL.

Vector search

In Elasticsearch 9.2, we introduced tech preview support for dense vector search in ES|QL. This is as simple as calling the knn function, which only requires a dense_vector field and a query vector:

This query executes an approximate nearest neighbor search, retrieving 100 documents that are the most similar to the query_vector.

Hybrid search: Reciprocal rank fusion

In Elasticsearch 9.2, we introduced support for hybrid search using RRF and linear combination of results in ES|QL.

This allows combining vector search and lexical search results into a single result set.

To achieve this in ES|QL, we need to use the FORK and FUSE commands. FORK runs multiple branches of execution, and FUSE merges the results and assigns new relevance scores using RRF or linear combination.

In the following example, we use FORK to run two separate branches, where one is doing a lexical search using the match function, while the other is doing a vector search using the knn function. We then merge the results together using FUSE:

Let's decompose the query to better understand the execution model and first look at the output of the FORK command:

The FORK commands outputs the results from both branches and adds a _fork discriminator column:

_id

title

_score

_fork

4001

The Hobbit

0.88

fork1

3999

The Fellowship of the Ring

0.88

fork1

4005

The Two Towers

0.86

fork1

4006

The Return of the King

0.84

fork1

4123

The Silmarillion

0.78

fork1

4144

The Children of Húrin

0.79

fork1

4001

The Hobbit

4.55

fork2

3999

The Fellowship of the Ring

4.25

fork2

4123

The Silmarillion

4.11

fork2

4005

The Two Towers

3.8

fork2

4006

The Return of the King

4.1

fork2

As you’ll notice, certain documents appear twice, which is why we then use FUSE to merge rows that represent the same documents and assign new relevance scores. FUSE is executed in two stages:

• For each row, FUSE assigns a new relevance score, depending on the hybrid search algorithm that is being used.

• Rows that represent the same document are merged together, and a new score is computed.

In our example, we’re using RRF. As a first step, FUSE assigns a new score to each row using the RRF formula:

Where the rank_constant takes a default value of 60 and rank(doc)represents the position of the document in the result set.

In the first phase, our results become:

_id

title

_score

_fork

4001

The Hobbit

1 / (60 + 1) = 0.01639

fork1

3999

The Fellowship of the Ring

1 / (60 + 2) = 0.01613

fork1

4005

The Two Towers

1 / (60 + 3) = 0.01587

fork1

4006

The Return of the King

1 / (60 + 4) = 0.01563

fork1

4123

The Silmarillion

1 / (60 + 5) = 0.01538

fork1

4144

The Children of Húrin

1 / (60 + 6) = 0.01515

fork1

4001

The Hobbit

1 / (60 + 1) = 0.01639

fork2

3999

The Fellowship of the Ring

1 / (60 + 2) = 0.01613

fork2

4123

The Silmarillion

1 / (60 + 3) = 0.01587

fork2

4005

The Two Towers

1 / (60 + 4) = 0.01563

fork2

4006

The Return of the King

1 / (60 + 5) = 0.01538

fork2

Then the rows are merged together and a new score is assigned. Since a SORT _score DESC follows the FUSE command, the final results are:

_id

title

_score

4001

The Hobbit

0.01639 + 0.01639 = 0.03279

3999

The Fellowship of the Ring

0.01613 + 0.01613 = 0.03226

4005

The Two Towers

0.01587 + 0.01563 = 0.0315

4123

The Silmarillion

0.01538 + 0.01587 = 0.03125

4006

The Return of the King

0.01563 + 0.01538 = 0.03101

4144

The Children of Húrin

0.01515

Hybrid search: Linear combination of scores

Reciprocal rank fusion is the simplest way to do hybrid search, but it isn’t the only hybrid search method that we support in ES|QL.

In the following example, we use FUSE to combine lexical and semantic search results using linear combination of scores:

Let's first decompose the query and take a look at the input of the FUSE command when we only run the FORK command.

Notice that we use the match function, which is able to not only query lexical fields, such as text or keyword, but also semantic_text fields.

The first FORK branch executes a semantic query by querying a semantic_text field, while the second one executes a lexical query:

The output of the FORK command can contain rows with the same _id and _index values representing the same Elasticsearch document:

_id

title

_score

_fork

4001

The Hobbit

0.88

fork1

3999

The Fellowship of the Ring

0.88

fork1

4005

The Two Towers

0.86

fork1

4006

The Return of the King

0.84

fork1

4123

The Silmarillion

0.78

fork1

4144

The Children of Húrin

0.79

fork1

4001

The Hobbit

4.55

fork2

3999

The Fellowship of the Ring

4.25

fork2

4123

The Silmarillion

4.11

fork2

4005

The Two Towers

3.8

fork2

4006

The Return of the King

4.1

fork2

In the next step, we use FUSE to merge rows that have the same _id and _index values, and assign new relevance scores.

The new score is a linear combination of the scores the row had in each FORK branch:

Here, _score1 and _score2 represent the score a document has in the first FORK branch and the second FORK branch, respectively.

Notice that we also apply custom weights, giving more weight to the semantic score over the lexical one, resulting in this set of documents:

_id

title

_score

4001

The Hobbit

0.7 * 0.88 + 0.3 * 4.55 = 1.981

3999

The Fellowship of the Ring

0.7 * 0.88 + 0.3 * 4.25 = 1.891

4006

The Return of the King

0.7 * 0.84 + 0.3 * 4.1 = 1.818

4123

The Silmarillion

0.7 * 0.78 + 0.3 * 4.11 = 1.779

4005

The Two Towers

0.7 * 0.86 + 0.3 * 3.8 = 1.742

4144

The Children of Húrin

0.7 * 0.79 + 0.3 * 0 = 0.553

One challenge is that the semantic and lexical scores can be incompatible to apply the linear combination, since they can follow completely different distributions. To mitigate this, we first need to normalize the scores, employing score normalization methods, such as minmax. This ensures that the scores from each FORK branch are first normalized to take values between 0 and 1, before applying the linear combination formula.

To achieve this with FUSE, we need to specify the normalizer option:

Semantic reranking

At this stage, after hybrid search, we should be left with the most relevant documents. We can now use semantic reranking to reorder the results using the RERANK command. By default, RERANK uses the latest Elastic semantic reranking machine learning model, so no additional configuration is needed:

We now have our best results, sorted by relevance.

One key feature that sets the RERANK command apart from other products that offer semantic reranking integrations is that it doesn’t require the input to represent a mapped field from an index. RERANK only expects an expression that evaluates to a string value, making it possible to do semantic reranking using multiple fields:

LLM completions

Now we have a set of highly relevant, reranked results.

At this stage, you might simply decide to return the results back to your application or you might want to further enhance your results using LLM completions.

If you’re using ES|QL as part of a retrieval-augmented generation (RAG) workflow, you can choose to call your favorite LLM directly from ES|QL. To achieve this, we’ve added a new COMPLETION command that takes in a prompt, a completion inference ID which designates which LLM to call, and a column identifier to specify where to output the LLM response.

In the following example, we’re using COMPLETION to add a new _completion column that contains the summary of the content column:

Each row now contains a summary:

_id

title

_score

summary

4001

The Hobbit

0.03279

Bilbo helps dwarves reclaim Erebor from the dragon Smaug.

3999

The Fellowship of the Ring

0.03226

Frodo begins the quest to destroy the One Ring.

4005

The Two Towers

0.0315

The Fellowship splits; war comes to Rohan; Frodo nears Mordor.

4123

The Silmarillion

0.03125

Ancient myths and history of Middle-earth's First Age.

4006

The Return of the King

0.3101

Sauron is defeated and Aragorn is crowned King.

4144

The Children of Húrin

0.01515

The tragic tale of Túrin Turambar's cursed life.

In another use case, you may simply want to answer a question using the proprietary data that you have indexed in Elasticsearch. In this case, the best search results that we’ve computed in the previous stage can be used as context for the prompt:

Since the COMPLETION command unlocks the ability to send any prompt to an LLM, the possibilities are endless. Although we’re only showing a few examples, the COMPLETION command can be used in a wide range of scenarios, from security analysts using it to assign scores depending on whether a log event can represent a malicious action or data scientists using it to analyze data, to cases where you just need to generate Chuck Norris facts based on your data.

This is only the beginning

In the future, we’ll be expanding ES|QL to improve semantic reranking for long documents, better conditional execution of the ES|QL queries using multiple FORK commands, support sparse vector queries, removing close duplicate results to enhance result diversity, allowing full text search on runtime generated columns, and many other scenarios.

Additional tutorials and guides:

• ES|QL for search

• ES|QL for search tutorial

• Semantic_text field type

• FORK and FUSE documentation

• ES|QL search functions

Do you want to start using semantic search for your data, but focus on your model and results instead of on the technical details? We’ve introduced the semantic_text field type that will take care of the details and infrastructure that you need.

Semantic search is a sophisticated technique designed to enhance the relevance of search results by utilizing machine learning models. Unlike traditional keyword-based search, semantic search focuses on understanding the meaning of words and the context in which they are used. This is achieved through the application of machine learning models that provide a deeper semantic understanding of the text.

These models generate vector embeddings, which are numeric representations capturing the text meaning. These embeddings are stored alongside your document data, enabling vector search techniques that take into account the word meaning and context instead of pure lexical matches.

How to perform semantic search

To perform semantic search, you need to go through the following steps:

• Choose an inference model to create embeddings, both for indexing documents and performing queries.

• Create your index mapping to store the inference results, so they can be efficiently searched afterwards.

• Setting up indexing so inference results are calculated for new documents added to your index.

• Automatically handle long text documents, so search can be accurate and cover the entire document.

• Querying your data to retrieve results.

Configuring semantic search from the ground up can be complex. It requires setting up mappings, ingestion pipelines, and queries tailored to your chosen inference model. Each step offers opportunities for fine-tuning and optimization, but also demands careful configuration to ensure all components work together seamlessly.

While this offers a great degree of control, it makes using semantic search a detailed and deliberate process, requiring you to configure separate pieces that are all related to each other and to the inference model.

semantic_text simplifies this process by focusing on what matters: the inference model. Once you have selected the inference model, semantic_text will make it easy to start using semantic search by providing sensible defaults, so you can focus on your search and not on how to index, generate, or query your embeddings.

Let's take a look at each of these steps, and how semantic_text simplifies this setup.

Choosing an inference model

The inference model will generate embeddings for your documents and queries. Different models have different tradeoffs in terms of:

• Accuracy and relevance of the results

• Scalability and performance

• Language and multilingual support

• Cost

Elasticsearch supports both internal and external inference services:

• Internal services are deployed in the Elasticsearch cluster. You can use already included models like ELSER and E5, or import external models into the cluster using eland.

• External services are deployed by model providers. Elasticsearch supports the following:

  • Cohere

  • Hugging Face

  • Mistral

  • OpenAI

  • Azure AI Studio

  • Azure OpenAI

  • Google AI Studio

Once you have chosen the inference mode, create an inference endpoint for it. The inference endpoint identifier will be the only configuration detail that you will need to set up semantic_text.

Creating your index mapping

Elasticsearch will need to index the embeddings generated by the model so they can be efficiently queried later.

Before semantic_text, you needed to understand about the two main field types used for storing embeddings information:

• sparse_vector: It indexes sparse vector embeddings, like the ones generated by ELSER. Each embedding consists of pairs of tokens and weights. There is a small number of tokens generated per embedding.

• dense_vector: It indexes vectors of numbers, which contains the embedding information. A model produces vectors of a fixed size, called the vector dimension.

The field type to use is conditioned by the model you have chosen. If using dense vectors, you will need to configure the field to include the dimension count, the similarity function used to calculate vectors proximity, and storage customizations like quantization or the specific data type used for each element.

Now, if you're using semantic_text, you define a semantic_text field mapping by just specifying the inference endpoint identifier for your model:

That's it. No need for you to define other mapping options, or to understand which field type you need to use.

Setting up indexing

Once your index is ready to store the embeddings, it's time to generate them.

Before semantic_text, to generate embeddings automatically on document ingestion you needed to set up an ingestion pipeline.

Ingestion pipelines are used to automatically enrich or transform documents when ingested into an index, or when explicitly specified as part of the ingestion process.

You need to use the inference processor to generate embeddings for your fields. The processor needs to be configured using:

• The text fields from which to generate the embeddings

• The output fields where the generated embeddings will be added

• Specific inference configuration for text embeddings or sparse embeddings, depending on the model type

With semantic_text, you simply add documents to your index. semantic_text fields will automatically calculate the embeddings using the specified inference endpoint.

This means there's no need to create an inference pipeline to generate the embeddings. Using bulk, index, or update APIs will do that for you automatically:

Inference requests in semantic_text fields are also batched. If you have 10 documents in a bulk API request, and each document contains 2 semantic_text fields, then that request will perform a single inference request with 20 texts to your inference service in one go, instead of making 10 separate inference requests of 2 texts each.

Automatically handling long text passages

Part of the challenge of selecting a model is the number of tokens that the model can generate embeddings for. Models have a limited number of tokens they can process. This is referred to as the model’s context window.

If the text you need to work with is longer than the model’s context window, you may truncate the text and use just part of it to generate embeddings. This is not ideal as you'll lose information; the resulting embeddings will not capture the full context of the input text.

Even if you have a long context window, having a long text means a lot of content will be reduced to a single embedding, making it an inaccurate representation.

Also, returning a long text will be difficult for the users to understand, as they will have to scan the text to check it's what they are looking for. Using smaller snippets would be preferable instead.

Another option is to use chunking to divide long texts into smaller fragments. These smaller chunks are added to each document to provide a better representation of the complete text. You can then use a nested query to search over all the individual fragments and retrieve the documents that contain the best-scoring chunks.

Before semantic_text, chunking was not done out of the box - the inference processor did not support chunking. If you needed to use chunking, you needed to do it before ingesting your documents or use the script processor to perform the chunking in Elasticsearch.

Using semantic_text means that chunking will be done on your behalf when indexing. Long documents will be split into 250-word sections with a 100-word overlap so that each section shares 100 words with the previous section. This overlap ensures continuity and prevents vital contextual information in the input text from being lost by a hard break.

If the model and inference service support batching the chunked inputs are automatically batched together into as few requests as possible, each optimally sized for the Inference Service. The resulting chunks will be stored in a nested object structure so you can check the text contained in each chunk.

Querying your data

Now that the documents and their embeddings are indexed in Elasticsearch, it's time to do some queries!

Before semantic_text, you needed to use a different query depending on the type of embeddings the model generates (dense or sparse). A sparse vector query is needed to query sparse_vector field types, and either a knn search or a knn query can be used to search dense_vector field types.

The query process can be further customized for performance and relevance. For example, sparse vector queries can define token pruning to avoid considering irrelevant tokens. Knn queries can specify the number of candidates to consider and the top k results to be returned from each shard.

You don't need to deal with those details when using semantic_text. You use a single query type to search your documents:

Just include the field and the query text. There’s no need to decide between sparse vector and knn queries, semantic text does this for you.

Compare this with using a specific knn search with all its configuration parameters:

Under the hood: How semantic_text works

To understand how semantic_text works, you can create a semantic_text index and check what happens when you ingest a document. When the first document is ingested, the inference endpoint calculates the embeddings. When indexed, you will notice changes in the index mapping:

Now there is additional information about the model settings. Text embedding models will also include information like the number of dimensions or the similarity function for the model.

You can check the document already includes the embedding results:

The field does not just contain the input text, but also a structure storing the original text, the model settings, and information for each chunk the input text has been divided into.

This structure consists of an object with two elements:

• text: Contains the original input text

• inference: Inference information added by the inference endpoint, that consists of:

  • inference_id of the inference endpoint

  • model_settings that contain model properties

  • chunks: Nested object that contains an element for each chunk that has been created from the input text. Each chunk contains:

    • The text for the chunk

    • The calculated embeddings for the chunk text

Customizing semantic_text

semantic_text simplifies semantic search by making default decisions about indexing and querying your data:

• uses sparse_vector or dense_vector field types depending on the inference model type

• Automatically defines the number of dimensions and similarity according to the inference results

• Uses int8_hnsw index type for dense vector field types to leverage scalar quantization.

• Uses query defaults. No token pruning is applied for sparse_vector queries, nor custom k and num_candidates are set for knn queries.

Those are sensible defaults and allow you to quickly and easily start working with semantic search. Over time, you may want to customize your queries and data types to optimize search relevance, index and query performance, and index storage.

Query customization

There are no customization options - yet - for semantic queries. If you want to customize queries against semantic_text fields, you can perform advanced semantic_text search using explicit knn and sparse vector queries.

We're planning to add retrievers support for semantic_text, and adding configuration options to the semantic_text field so they won't be needed at query time. Stay tuned!

Data type customization

If you need deeper customization for the data indexing, you can use the sparse_vector or dense_vector field types. These field types give you full control over how embeddings are generated, indexed, and queried.

You need to create an ingest pipeline with an inference processor to generate the embeddings. This tutorial walks you through the process.

What's next with semantic_text?

We're just getting started with semantic_text! There are quite a few enhancements that we will keep working on, including:

• Better inference error handling

• Customize the chunking strategy

• Hiding embeddings in _source by default, to avoid cluttering the search responses

• Inner hits support, to retrieve the relevant chunks of information for a query

• Filtering and retrievers support

• Kibana support

Try it out!

semantic_textis available on Elasticsearch Serverless now! It will be available soon on Elasticsearch 8.15 version for Elastic Cloud and on Elasticsearch downloads.

If you already have an Elasticsearch serverless cluster, you can see a complete example for testing semantic search using semantic_text in this tutorial, or try it with this notebook.

We'd love to hear about your experience with semantic_text! Let us know what you think in the forums, or open an issue in the GitHub repository. Let's make semantic search easier together!