Multistage retrieval: The challenge of modern search

Modern search has evolved beyond simple keyword matching. Today's search applications need to understand intent, handle natural language, and combine multiple ranking signals to deliver the best results.

Retrieval of the most relevant results happens in multiple stages, with each stage gradually refining the result set. This wasn’t the case in the past, where most use cases would require one or two stages of retrieval: an initial query to get results and a potential rescoring phase.

We start with an initial retrieval, where we cast a wide net to gather results that are relevant to our query. Since we need to sieve through all the data, we should use techniques that return results fast, even when we index billions of documents.

We therefore employ trusted techniques, such as lexical search that Elasticsearch has supported and optimized since the beginning, or vector search, where Elasticsearch excels in speed and accuracy.

Lexical search using BM25 is quite fast and best at exact term matching or phrase matching, and vector or semantic search is better suited for handling natural language queries. Hybrid search combines lexical and vector search results to bring the best from both. The challenge that hybrid search solves is that vector and lexical search have completely different and incompatible scoring functions which produce values in different intervals, following different distributions. A vector search score close to 1 can mean a very close match, but it doesn’t mean the same for lexical search. Hybrid search methods, such as reciprocal rank fusion (RRF) and linear combination of scores, assign new scores that blend the original scores from lexical and vector search.

After hybrid search, we can employ techniques such as semantic reranking and Learning To Rank (LTR), which use specialized machine learning models to rerank the result.

With our most relevant results, we can use large language models (LLMs) to further enrich our response or pass the most relevant results as context to LLMs in agentic workflows in tools such as Elastic Agent Builder.

ES|QL is able to handle all these stages of retrieval. By design, ES|QL is a piped language, where each command transforms the input and sends the output to the next command. Each stage of retrieval is represented by one or more consecutive ES|QL commands. In this article, we show how each stage is supported in ES|QL.

Vector search

In Elasticsearch 9.2, we introduced tech preview support for dense vector search in ES|QL. This is as simple as calling the knn function, which only requires a dense_vector field and a query vector:

This query executes an approximate nearest neighbor search, retrieving 100 documents that are the most similar to the query_vector.

Hybrid search: Reciprocal rank fusion

In Elasticsearch 9.2, we introduced support for hybrid search using RRF and linear combination of results in ES|QL.

This allows combining vector search and lexical search results into a single result set.

To achieve this in ES|QL, we need to use the FORK and FUSE commands. FORK runs multiple branches of execution, and FUSE merges the results and assigns new relevance scores using RRF or linear combination.

In the following example, we use FORK to run two separate branches, where one is doing a lexical search using the match function, while the other is doing a vector search using the knn function. We then merge the results together using FUSE:

Let's decompose the query to better understand the execution model and first look at the output of the FORK command:

The FORK commands outputs the results from both branches and adds a _fork discriminator column:

_id

title

_score

_fork

4001

The Hobbit

0.88

fork1

3999

The Fellowship of the Ring

0.88

fork1

4005

The Two Towers

0.86

fork1

4006

The Return of the King

0.84

fork1

4123

The Silmarillion

0.78

fork1

4144

The Children of Húrin

0.79

fork1

4001

The Hobbit

4.55

fork2

3999

The Fellowship of the Ring

4.25

fork2

4123

The Silmarillion

4.11

fork2

4005

The Two Towers

3.8

fork2

4006

The Return of the King

4.1

fork2

As you’ll notice, certain documents appear twice, which is why we then use FUSE to merge rows that represent the same documents and assign new relevance scores. FUSE is executed in two stages:

• For each row, FUSE assigns a new relevance score, depending on the hybrid search algorithm that is being used.

• Rows that represent the same document are merged together, and a new score is computed.

In our example, we’re using RRF. As a first step, FUSE assigns a new score to each row using the RRF formula:

Where the rank_constant takes a default value of 60 and rank(doc)represents the position of the document in the result set.

In the first phase, our results become:

_id

title

_score

_fork

4001

The Hobbit

1 / (60 + 1) = 0.01639

fork1

3999

The Fellowship of the Ring

1 / (60 + 2) = 0.01613

fork1

4005

The Two Towers

1 / (60 + 3) = 0.01587

fork1

4006

The Return of the King

1 / (60 + 4) = 0.01563

fork1

4123

The Silmarillion

1 / (60 + 5) = 0.01538

fork1

4144

The Children of Húrin

1 / (60 + 6) = 0.01515

fork1

4001

The Hobbit

1 / (60 + 1) = 0.01639

fork2

3999

The Fellowship of the Ring

1 / (60 + 2) = 0.01613

fork2

4123

The Silmarillion

1 / (60 + 3) = 0.01587

fork2

4005

The Two Towers

1 / (60 + 4) = 0.01563

fork2

4006

The Return of the King

1 / (60 + 5) = 0.01538

fork2

Then the rows are merged together and a new score is assigned. Since a SORT _score DESC follows the FUSE command, the final results are:

_id

title

_score

4001

The Hobbit

0.01639 + 0.01639 = 0.03279

3999

The Fellowship of the Ring

0.01613 + 0.01613 = 0.03226

4005

The Two Towers

0.01587 + 0.01563 = 0.0315

4123

The Silmarillion

0.01538 + 0.01587 = 0.03125

4006

The Return of the King

0.01563 + 0.01538 = 0.03101

4144

The Children of Húrin

0.01515

Hybrid search: Linear combination of scores

Reciprocal rank fusion is the simplest way to do hybrid search, but it isn’t the only hybrid search method that we support in ES|QL.

In the following example, we use FUSE to combine lexical and semantic search results using linear combination of scores:

Let's first decompose the query and take a look at the input of the FUSE command when we only run the FORK command.

Notice that we use the match function, which is able to not only query lexical fields, such as text or keyword, but also semantic_text fields.

The first FORK branch executes a semantic query by querying a semantic_text field, while the second one executes a lexical query:

The output of the FORK command can contain rows with the same _id and _index values representing the same Elasticsearch document:

_id

title

_score

_fork

4001

The Hobbit

0.88

fork1

3999

The Fellowship of the Ring

0.88

fork1

4005

The Two Towers

0.86

fork1

4006

The Return of the King

0.84

fork1

4123

The Silmarillion

0.78

fork1

4144

The Children of Húrin

0.79

fork1

4001

The Hobbit

4.55

fork2

3999

The Fellowship of the Ring

4.25

fork2

4123

The Silmarillion

4.11

fork2

4005

The Two Towers

3.8

fork2

4006

The Return of the King

4.1

fork2

In the next step, we use FUSE to merge rows that have the same _id and _index values, and assign new relevance scores.

The new score is a linear combination of the scores the row had in each FORK branch:

Here, _score1 and _score2 represent the score a document has in the first FORK branch and the second FORK branch, respectively.

Notice that we also apply custom weights, giving more weight to the semantic score over the lexical one, resulting in this set of documents:

_id

title

_score

4001

The Hobbit

0.7 * 0.88 + 0.3 * 4.55 = 1.981

3999

The Fellowship of the Ring

0.7 * 0.88 + 0.3 * 4.25 = 1.891

4006

The Return of the King

0.7 * 0.84 + 0.3 * 4.1 = 1.818

4123

The Silmarillion

0.7 * 0.78 + 0.3 * 4.11 = 1.779

4005

The Two Towers

0.7 * 0.86 + 0.3 * 3.8 = 1.742

4144

The Children of Húrin

0.7 * 0.79 + 0.3 * 0 = 0.553

One challenge is that the semantic and lexical scores can be incompatible to apply the linear combination, since they can follow completely different distributions. To mitigate this, we first need to normalize the scores, employing score normalization methods, such as minmax. This ensures that the scores from each FORK branch are first normalized to take values between 0 and 1, before applying the linear combination formula.

To achieve this with FUSE, we need to specify the normalizer option:

Semantic reranking

At this stage, after hybrid search, we should be left with the most relevant documents. We can now use semantic reranking to reorder the results using the RERANK command. By default, RERANK uses the latest Elastic semantic reranking machine learning model, so no additional configuration is needed:

We now have our best results, sorted by relevance.

One key feature that sets the RERANK command apart from other products that offer semantic reranking integrations is that it doesn’t require the input to represent a mapped field from an index. RERANK only expects an expression that evaluates to a string value, making it possible to do semantic reranking using multiple fields:

LLM completions

Now we have a set of highly relevant, reranked results.

At this stage, you might simply decide to return the results back to your application or you might want to further enhance your results using LLM completions.

If you’re using ES|QL as part of a retrieval-augmented generation (RAG) workflow, you can choose to call your favorite LLM directly from ES|QL. To achieve this, we’ve added a new COMPLETION command that takes in a prompt, a completion inference ID which designates which LLM to call, and a column identifier to specify where to output the LLM response.

In the following example, we’re using COMPLETION to add a new _completion column that contains the summary of the content column:

Each row now contains a summary:

_id

title

_score

summary

4001

The Hobbit

0.03279

Bilbo helps dwarves reclaim Erebor from the dragon Smaug.

3999

The Fellowship of the Ring

0.03226

Frodo begins the quest to destroy the One Ring.

4005

The Two Towers

0.0315

The Fellowship splits; war comes to Rohan; Frodo nears Mordor.

4123

The Silmarillion

0.03125

Ancient myths and history of Middle-earth's First Age.

4006

The Return of the King

0.3101

Sauron is defeated and Aragorn is crowned King.

4144

The Children of Húrin

0.01515

The tragic tale of Túrin Turambar's cursed life.

In another use case, you may simply want to answer a question using the proprietary data that you have indexed in Elasticsearch. In this case, the best search results that we’ve computed in the previous stage can be used as context for the prompt:

Since the COMPLETION command unlocks the ability to send any prompt to an LLM, the possibilities are endless. Although we’re only showing a few examples, the COMPLETION command can be used in a wide range of scenarios, from security analysts using it to assign scores depending on whether a log event can represent a malicious action or data scientists using it to analyze data, to cases where you just need to generate Chuck Norris facts based on your data.

This is only the beginning

In the future, we’ll be expanding ES|QL to improve semantic reranking for long documents, better conditional execution of the ES|QL queries using multiple FORK commands, support sparse vector queries, removing close duplicate results to enhance result diversity, allowing full text search on runtime generated columns, and many other scenarios.

Additional tutorials and guides:

• ES|QL for search

• ES|QL for search tutorial

• Semantic_text field type

• FORK and FUSE documentation

• ES|QL search functions

Let’s build something fun to show it off: a Chuck Norris fact generator. We'll combine movie descriptions with a GPT model to generate facts so legendary even Rambo would be impressed.

What you'll need

• Access to an LLM (like OpenAI’s GPT-4o in our example below)

• A dataset of movie descriptions

You can download a sample dataset from Kaggle and upload it to your Elasticsearch cluster using the Data Visualizer in Kibana or the _bulk API.

Setting up the inference endpoint

Before you can run the COMPLETION command, you need to create an inference endpoint for the model you want to use via the _inference API.

Here’s how to set up GPT-4o with OpenAI:

Once this is in place, you can reference my-gpt-4o-endpoint directly in your query.

The query

Here’s the magic in action. This single ES|QL query handles the entire workflow: it finds a movie based on your input, constructs a prompt from its description, and then calls the LLM to generate a legendary Chuck Norris fact. Below is the full ES|QL query that powers our Chuck Norris fact generator. It takes in a movie query, retrieves the most relevant description, turns it into a prompt, and sends it off to the LLM—all in a single, piped query.

Here’s what comes back:

Yes, the model really said that. 💪🐐🚁

Dissecting the query

Let’s dissect the query and break down what’s happening, step by step.

Step 1: Retrieve relevant movie data

We begin by searching for the most relevant movie for the user query. We use the MATCH function to search both the title and overview fields for the text provided by the query parameter, keeping only the first result, sorted by relevance using the metadata _score field:

This narrows down our dataset to the best match, giving us the movie's title and description, which will become the context for the LLM.

Step 2: Build the prompt from the context

Now we create the input prompt for the LLM by concatenating a static instruction provided as a query parameter, denoted by ?instruction, with the movie’s overview:

This creates a new prompt column combining the provided instruction with the overview field from the returned document, which for our request looks a bit like this:

You can easily swap in different instructions to change the tone or style of what the LLM generates by tweaking the instruction parameter. And because the prompt is just another ES|QL expression, you can compose it with any string-generating function—whether it’s simple concatenation, conditional logic, or even formatting based on your document content.

Step 3: Generate text using the LLM

Finally, we pass the prompt to the inference endpoint connected to our model using our new COMPLETION command, and select which fields to return:

The result? A Chuck Norris fact, rooted in your movie data without any extra tooling required.

This example also demonstrates the full power of ES|QL's piped structure. Each step flows naturally into the next, letting you express a full retrieval augmented generation (RAG) pipeline in a single, declarative query. It’s clean, composable, and stays entirely inside Elasticsearch.

What’s next?

While the COMPLETION command is still a tech preview, this new feature unlocks a whole new world of possibilities—from summarization and content generation to enrichment and storytelling. Try it yourself! Point it at your favorite movie, tweak the prompt, or go wild and generate haikus from SQL errors. The power is yours.

Let us know what you build! 💬

This blog post will explain how this new feature can help in improving your document ranking in text search and how to implement it in Elasticsearch.

Whether you are trying to optimize an eCommerce search, build the best context for a Retrieval Augmented Generation(RAG) application or craft a question answering based search on millions of academic papers, you have probably realized how challenging it can be to accurately optimize document ranking in a search engine. That's where Learning to Rank comes in.

Understanding relevance features and how to build a scoring function

Relevance features are the signals to determine how well a document matches a user's query or interest, all of which impact search relevance. These features can vary significantly depending on the context, but they generally fall into several categories. Let’s take a look at some common relevance features used across different domains:

• Text Relevance Scores (e.g., BM25, TF-IDF): Scores derived from text matching algorithms that measure the similarity of document content to the search query. These scores can be obtained from Elasticsearch.

• Document Properties (e.g., price of a product, publication date): Features that can be extracted directly from the stored document.

• Popularity Metrics (e.g., click-through rate, views): Indicators of how popular or frequently accessed a document is. Popularity metrics can be obtained with Search analytics tools, of which Elasticsearch provides out-of-the-box.

The scoring function combines these features to produce a final relevance score for each document. Documents with higher scores are ranked higher in search results.

When using the Elasticsearch Query DSL, you are implicitly writing a scoring function that weights relevance features and ultimately defines your search relevance

Scoring in the Elasticsearch Query DSL

Consider the following example query:

This query translates into the following scoring function:

While this approach works well, it has a few limitations:

• Weights are estimated: The weights assigned to each feature are often based on heuristics or intuition. These guesses may not accurately reflect the true importance of each feature in determining relevance.

• Uniform Weights Across Documents: Manually assigned weights apply uniformly to all documents, ignoring potential interactions between features and how their importance might vary across different queries or document types. For instance, the relevance of recency might be more significant for news articles but less so for academic papers.

As the number of features and documents increases, these limitations become more pronounced, making it increasingly challenging to determine accurate weights. Ultimately, the chosen weights become a compromise, potentially leading to suboptimal ranking in many scenarios.

A compelling alternative is to replace the scoring function that uses manual weights by a ML-based model that computes the score using relevance features.

Hello Learning To Rank (LTR)!

LambdaMART is a popular and effective LTR technique that uses gradient boosting decision trees (GBDT) to learn the optimal scoring function from a judgment list.

The judgment list is a dataset that contains pairs of queries and documents, along with their corresponding relevance labels or grades. Relevance labels are typically either binary, (e.g. relevant/irrelevant) or graded (e.g between 0 for completely irrelevant and 4 for highly relevant). Judgment lists can be created manually by humans or be generated from user engagement data, such as clicks or conversions.

The example below uses a graded relevance judgment.

LambdaMART treats the ranking problem as a regression task using a decision tree where the inner nodes of the tree are conditions over the relevance features, and the leaves are the predicted scores.

LambdaMART uses a gradient boosted tree approach, and in the training process it builds multiple decision trees where each tree corrects errors of its predecessors. This process aims to optimize a ranking metric like NDCG, based on examples from the judgment list. The final model is a weighted sum of individual trees.

XGBoost is a well known library that provides an implementation of LambdaMART, making it a popular choice to implement ranking based on gradient boosting decision trees.

Getting started with LTR in Elasticsearch

Starting with version 8.13, Learning To Rank is integrated directly into Elasticsearch and associated tooling as a technical preview feature.

Train and deploy an LTR model to Elasticsearch

Eland is our Python client and toolkit for DataFrames and machine learning in Elasticsearch. Eland is compatible with most of the standard Python data science tools like Pandas, scikit-learn and XGBoost.

We highly recommend using it to train and deploy your LTR XGBoost model, as it provides features to simplify this process:

1. The first step of the training process is to define the relevant features of the LTR model. Using the Python code below, you can specify the relevant features using the Elasticsearch Query DSL.

1. The second step of the process is to build your training dataset. At this step you will compute and add relevance features for each rows of your judgment list:

To help you with this task, Eland provides the FeatureLogger class:

1. When the training dataset is built, the model is trained very easily (as also shown in the notebook):

1. Deploy your model to Elasticsearch once the training process is complete:

To learn more about how our tooling can help you to train and deploy the model, check out this end-to-end notebook.

Use your LTR model as a rescorer in Elasticsearch

Once you deploy your model in Elasticsearch, you can enhance your search results through a rescorer. The rescorer allows you to refine a first-pass ranking of search results using the more sophisticated scoring provided by your LTR model:

In this example:

• First-pass query: The multi_match query retrieves documents that match the query the quick brown fox in the title and content fields. This query is designed to be fast and capture a large set of potentially relevant documents.

• Rescore phase: The learning_to_rank rescorer refines the top results from the first-pass query using the LTR model.

  • model_id: Specifies the ID of the deployed LTR model (ltr-model-xgboost in our example).

  • params: Provides any parameters required by the LTR model to extract features relevant to the query. Here query_text allows you to specify the query issued by the user that some of our features extractors expect.

  • window_size: Defines the number of top documents from the search results issued by the first-pass query to be rescored. In this example, the top 100 documents will be rescored.

By integrating LTR as a two stage retrieval process, you can can optimize both performance and accuracy of your retrieval process by combining:

• Speed of Traditional Search: The first-pass query retrieves a large number of documents with a broad match very quickly, ensuring fast response times.

• Precision of Machine Learning Models: The LTR model is applied only to the top results, refining their ranking to ensure optimal relevance. This targeted application of the model enhances precision without compromising overall performance.

Try LTR yourself!?

Whether you are struggling to configure search relevance for an eCommerce platform, aiming to improve the context relevance of your RAG application, or you are simply curious about enhancing your existing search engine's performance, you should consider LTR seriously.

To start your journey with implementing LTR, make sure to visit our notebook detailing how to train, deploy, and use an LTR model in Elasticsearch and to read our documentation. Let us know if you built anything based on this blog post or if you have questions on our Discuss forums and the community Slack channel.