How MATCH and TO_TEXT enable full-text search on any ES|QL expression

Let's start with a query that was impossible in Elasticsearch 9.4, which uses the EVAL command:

In this example, summary has no mapping or analyzer configuration. It’s also not associated with any inverted index. It exists only for the lifetime of this query, but now you can search it anyway. Two additions make this work.

First, MATCH now accepts any expression as its first argument, not just a mapped field. That includes columns produced by EVAL and function results used inline. It also includes unmapped fields loaded directly from the original document. Furthermore, all data types normally accepted by MATCH are supported in this new use case.

The second part of this is the new TO_TEXT function, which is the first ES|QL conversion function that produces output of type text. Until now, text columns could only come from indexed mapped fields, and all strings produced by ES|QL expressions were keyword values rather than text. The distinction matters because MATCH treats the two differently: text values are analyzed, while keyword values are compared exactly, mirroring how a MATCH query on an indexed keyword field rewrites to a term query. TO_TEXT(x) is how you tell ES|QL: treat this string as full text.

This ships as a technical preview in Elasticsearch 9.5, and as such, it has some limitations:

• It’s currently filtering only. A MATCH on an expression doesn't contribute to the relevance score yet; only matches on indexed fields affect the score.

• Querying options like fuzziness and others aren't yet supported when matching an expression.

• Runtime text is analyzed with the standard analyzer. This isn’t configurable yet.

Work is underway to address these limitations.

Why use full-text search instead of LIKE or RLIKE in ES|QL?

ES|QL already had two ways to search strings without an index: LIKE (wildcard patterns) and RLIKE (regular expressions). Both work on any string expression, so it's fair to ask what MATCH adds. The answer is analysis, a more advanced form of search which uses techniques such as stemming and synonyms. It also uses stopword handling.

LIKE is simple substring matching, without any understanding of the words that comprise a string. Say, for example, you're looking for log messages about a fox:

This misses "Fox spotted near the henhouse" due to the capitalization, while matching "Outfoxed by the competition", which isn't about a fox at all. It fails in both directions, with false negatives on capitalization and false positives on substrings buried inside other words.

Regular expressions can patch the case problem, but the word-boundary problem gets ugly fast. Something like:

And even that's not right yet. It misses a fox at the end of a sentence followed by ! or ?, and it says nothing about tabs, quotes, or parentheses. Each fix makes the pattern longer, and the next person to read the query has to reverse-engineer what it’s actually doing.

MATCH makes the problem go away, because it runs both the query and the value through an analyzer, which tokenizes text into lowercase terms and then matches term against term:

This query will match values like "The quick brown fox" and "FOX spotted near the henhouse" but not "Outfoxed by the competition" or “FOXTROT protocol enabled", regardless of any punctuation surrounding the words. Of course, this all works for multi-term queries, like MATCH(TO_TEXT(message), "brown fox"), too, just the way you’d expect it to.

Work is underway to enable the use of the 36 dedicated language analyzers, with support for natural languages on data that was never indexed or mapped.

Full-text search use cases for unindexed and unmapped data

The examples above searched values computed from mapped fields. . The more interesting use cases for ES|QL MATCH on expressions involve data that was never searchable at all. Let's walk through a few.

How to search unmapped fields in ES|QL without adding a mapping

Sometimes you deliberately leave a field out of your mappings, such as a verbose stack trace or a raw request payload. You might even leave out a debug blob. Indexing one of these would cost disk and heap space on every document and wouldn’t be worth it for a field you might query once a quarter.

That decision has always been final, because unmapped fields were invisible to queries entirely. In Elasticsearch 9.5, you can use SET unmapped_fields="load" to make ES|QL load unmapped fields directly from the source document as keywords. Follow that up by wrapping it in TO_TEXT, and now you can run full-text search on it:

Here, stack_trace was never mapped. Every value is fetched from the original documents and analyzed on the fly. They’re matched row by row. That’s real work, and it will never be as fast as an inverted index lookup. But now, that field you didn't index is no longer unsearchable. You get to keep the mapping small for the everyday case and still answer the once-a-quarter question when it matters.

Full-text search on a keyword field without reindexing

Keyword fields can do a lot. They give you exact matching, fast aggregations, and sorting, which is why so many fields end up mapped that way. But mappings are decided when data arrives, and it’s easy to end up in a situation where you want to do something different with your data than you had originally intended. Maybe product_name was mapped as a keyword because the dashboards aggregate on it, and then, after receiving a year’s worth of product data, someone wants to be able to search within product_name values.

The old answer was to change the mapping to text (or add a multi-field) and reindex everything. This can be both time-consuming and costly, and in many cases, users simply won’t want to bother with it. The new answer is one function call:

TO_TEXT converts the keyword values to text on the fly, so MATCH analyzes them instead of comparing them exactly. This allows you to query a keyword field without creating a mapping or reindexing the source document. If the search becomes an everyday query, indexing the field as text is still the right long-term move, but TO_TEXT gets you an answer today, without any extra work.

Searching the same field across indices with different mappings

ES|QL can span many indices, and the same field doesn't need to always look the same in all of them. When the same field has different types in different indices, ES|QL treats it as a union type, and a conversion function resolves the conflict. Let’s consider an example in which the message field has type text in this year's index template but was a keyword in last year's:

Every value is analyzed at query time, whether it came from the text index or the keyword one. The keyword values from the older indices are tokenized and lowercased like everything else, so "connection reset" finds “Connection RESET by peer”, no matter which index it lives in.

Another interesting case is when a field is mapped in only one index but also present (and unmapped) in the other:

There's a nuance worth calling out here. If error_details is mapped in logs-2026 but not in logs-2025, Elasticsearch cannot push this query down to Lucene, because the indices where the field is unmapped would silently return no matches. Instead, the planner notices that the field is potentially unmapped and evaluates the whole MATCH row by row, wherever the rows came from. You don't have to know which of your indices have the field mapped; the query just answers the question.

How ES|QL analyzes text at query time without an inverted index

When ES|QL plans a MATCH against an expression, it analyzes the query string once, up front, into a set of terms. How each row is then evaluated depends on the expression's type:

Expression type

Processing

Matching behavior

text (via TO_TEXT)

Analyzer tokenizes value into lowercase terms

Token-against-token comparison; a row matches if any token equals any query term (OR semantics)

keyword,ip, date, numeric

No analysis; query constant converted once to the native type

Exact comparison per row

Both paths bypass Lucene entirely and evaluate values row by row. The non-text path mirrors exactly what a match query does when pushed down to Lucene against those field types, so the semantics stay consistent regardless of whether your query hits an index.

An inverted-index lookup does its work at ingest time and never touches non-matching documents at query time. A runtime MATCH does that analysis at query time, for every row that reaches it. One is fast because the work already happened; the other is flexible because the data doesn't need to have been indexed at all.

What's next for ES|QL full-text search

Everything in this post is the first installment of a larger effort to make search in ES|QL work on anything, not just on what you indexed ahead of time. The limitations called out earlier are actively being worked on, and the roadmap goes further:

• Scoring. Runtime matches will contribute to _score, so you can sort by relevance even when the data was never indexed.

• MATCH_PHRASE on expressions. Already available in Elastic Cloud Serverless, and coming to the Elastic Stack in 9.6.

• Configurable analyzers. Analyzer support for MATCH and MATCH_PHRASE on expressions, enabling language analyzers, stemming, and synonyms at query time.

• Match options. Options like fuzziness and operator for runtime matches.

• Vector search. Generating embeddings per row and running k-nearest neighbors (kNN) on runtime dense_vector expressions, bringing semantic search to unindexed data, too.

Try ES|QL full-text search on expressions today

You can try runtime search today. It's available now in Elastic Cloud Serverless, where new ES|QL capabilities land first, and it ships as a technical preview in Elasticsearch 9.5. Start with the search functions reference, and check the ES|QL limitations page for the current boundaries. It's a technical preview because we want your feedback: If you search something that was never indexed and it surprises you, either positively or negatively, we'd love to hear about it.

Multistage retrieval: The challenge of modern search

Modern search has evolved beyond simple keyword matching. Today's search applications need to understand intent, handle natural language, and combine multiple ranking signals to deliver the best results.

Retrieval of the most relevant results happens in multiple stages, with each stage gradually refining the result set. This wasn’t the case in the past, where most use cases would require one or two stages of retrieval: an initial query to get results and a potential rescoring phase.

We start with an initial retrieval, where we cast a wide net to gather results that are relevant to our query. Since we need to sieve through all the data, we should use techniques that return results fast, even when we index billions of documents.

We therefore employ trusted techniques, such as lexical search that Elasticsearch has supported and optimized since the beginning, or vector search, where Elasticsearch excels in speed and accuracy.

Lexical search using BM25 is quite fast and best at exact term matching or phrase matching, and vector or semantic search is better suited for handling natural language queries. Hybrid search combines lexical and vector search results to bring the best from both. The challenge that hybrid search solves is that vector and lexical search have completely different and incompatible scoring functions which produce values in different intervals, following different distributions. A vector search score close to 1 can mean a very close match, but it doesn’t mean the same for lexical search. Hybrid search methods, such as reciprocal rank fusion (RRF) and linear combination of scores, assign new scores that blend the original scores from lexical and vector search.

After hybrid search, we can employ techniques such as semantic reranking and Learning To Rank (LTR), which use specialized machine learning models to rerank the result.

With our most relevant results, we can use large language models (LLMs) to further enrich our response or pass the most relevant results as context to LLMs in agentic workflows in tools such as Elastic Agent Builder.

ES|QL is able to handle all these stages of retrieval. By design, ES|QL is a piped language, where each command transforms the input and sends the output to the next command. Each stage of retrieval is represented by one or more consecutive ES|QL commands. In this article, we show how each stage is supported in ES|QL.

Vector search

In Elasticsearch 9.2, we introduced tech preview support for dense vector search in ES|QL. This is as simple as calling the knn function, which only requires a dense_vector field and a query vector:

This query executes an approximate nearest neighbor search, retrieving 100 documents that are the most similar to the query_vector.

Hybrid search: Reciprocal rank fusion

In Elasticsearch 9.2, we introduced support for hybrid search using RRF and linear combination of results in ES|QL.

This allows combining vector search and lexical search results into a single result set.

To achieve this in ES|QL, we need to use the FORK and FUSE commands. FORK runs multiple branches of execution, and FUSE merges the results and assigns new relevance scores using RRF or linear combination.

In the following example, we use FORK to run two separate branches, where one is doing a lexical search using the match function, while the other is doing a vector search using the knn function. We then merge the results together using FUSE:

Let's decompose the query to better understand the execution model and first look at the output of the FORK command:

The FORK commands outputs the results from both branches and adds a _fork discriminator column:

_id

title

_score

_fork

4001

The Hobbit

0.88

fork1

3999

The Fellowship of the Ring

0.88

fork1

4005

The Two Towers

0.86

fork1

4006

The Return of the King

0.84

fork1

4123

The Silmarillion

0.78

fork1

4144

The Children of Húrin

0.79

fork1

4001

The Hobbit

4.55

fork2

3999

The Fellowship of the Ring

4.25

fork2

4123

The Silmarillion

4.11

fork2

4005

The Two Towers

3.8

fork2

4006

The Return of the King

4.1

fork2

As you’ll notice, certain documents appear twice, which is why we then use FUSE to merge rows that represent the same documents and assign new relevance scores. FUSE is executed in two stages:

• For each row, FUSE assigns a new relevance score, depending on the hybrid search algorithm that is being used.

• Rows that represent the same document are merged together, and a new score is computed.

In our example, we’re using RRF. As a first step, FUSE assigns a new score to each row using the RRF formula:

Where the rank_constant takes a default value of 60 and rank(doc)represents the position of the document in the result set.

In the first phase, our results become:

_id

title

_score

_fork

4001

The Hobbit

1 / (60 + 1) = 0.01639

fork1

3999

The Fellowship of the Ring

1 / (60 + 2) = 0.01613

fork1

4005

The Two Towers

1 / (60 + 3) = 0.01587

fork1

4006

The Return of the King

1 / (60 + 4) = 0.01563

fork1

4123

The Silmarillion

1 / (60 + 5) = 0.01538

fork1

4144

The Children of Húrin

1 / (60 + 6) = 0.01515

fork1

4001

The Hobbit

1 / (60 + 1) = 0.01639

fork2

3999

The Fellowship of the Ring

1 / (60 + 2) = 0.01613

fork2

4123

The Silmarillion

1 / (60 + 3) = 0.01587

fork2

4005

The Two Towers

1 / (60 + 4) = 0.01563

fork2

4006

The Return of the King

1 / (60 + 5) = 0.01538

fork2

Then the rows are merged together and a new score is assigned. Since a SORT _score DESC follows the FUSE command, the final results are:

_id

title

_score

4001

The Hobbit

0.01639 + 0.01639 = 0.03279

3999

The Fellowship of the Ring

0.01613 + 0.01613 = 0.03226

4005

The Two Towers

0.01587 + 0.01563 = 0.0315

4123

The Silmarillion

0.01538 + 0.01587 = 0.03125

4006

The Return of the King

0.01563 + 0.01538 = 0.03101

4144

The Children of Húrin

0.01515

Hybrid search: Linear combination of scores

Reciprocal rank fusion is the simplest way to do hybrid search, but it isn’t the only hybrid search method that we support in ES|QL.

In the following example, we use FUSE to combine lexical and semantic search results using linear combination of scores:

Let's first decompose the query and take a look at the input of the FUSE command when we only run the FORK command.

Notice that we use the match function, which is able to not only query lexical fields, such as text or keyword, but also semantic_text fields.

The first FORK branch executes a semantic query by querying a semantic_text field, while the second one executes a lexical query:

The output of the FORK command can contain rows with the same _id and _index values representing the same Elasticsearch document:

_id

title

_score

_fork

4001

The Hobbit

0.88

fork1

3999

The Fellowship of the Ring

0.88

fork1

4005

The Two Towers

0.86

fork1

4006

The Return of the King

0.84

fork1

4123

The Silmarillion

0.78

fork1

4144

The Children of Húrin

0.79

fork1

4001

The Hobbit

4.55

fork2

3999

The Fellowship of the Ring

4.25

fork2

4123

The Silmarillion

4.11

fork2

4005

The Two Towers

3.8

fork2

4006

The Return of the King

4.1

fork2

In the next step, we use FUSE to merge rows that have the same _id and _index values, and assign new relevance scores.

The new score is a linear combination of the scores the row had in each FORK branch:

Here, _score1 and _score2 represent the score a document has in the first FORK branch and the second FORK branch, respectively.

Notice that we also apply custom weights, giving more weight to the semantic score over the lexical one, resulting in this set of documents:

_id

title

_score

4001

The Hobbit

0.7 * 0.88 + 0.3 * 4.55 = 1.981

3999

The Fellowship of the Ring

0.7 * 0.88 + 0.3 * 4.25 = 1.891

4006

The Return of the King

0.7 * 0.84 + 0.3 * 4.1 = 1.818

4123

The Silmarillion

0.7 * 0.78 + 0.3 * 4.11 = 1.779

4005

The Two Towers

0.7 * 0.86 + 0.3 * 3.8 = 1.742

4144

The Children of Húrin

0.7 * 0.79 + 0.3 * 0 = 0.553

One challenge is that the semantic and lexical scores can be incompatible to apply the linear combination, since they can follow completely different distributions. To mitigate this, we first need to normalize the scores, employing score normalization methods, such as minmax. This ensures that the scores from each FORK branch are first normalized to take values between 0 and 1, before applying the linear combination formula.

To achieve this with FUSE, we need to specify the normalizer option:

Semantic reranking

At this stage, after hybrid search, we should be left with the most relevant documents. We can now use semantic reranking to reorder the results using the RERANK command. By default, RERANK uses the latest Elastic semantic reranking machine learning model, so no additional configuration is needed:

We now have our best results, sorted by relevance.

One key feature that sets the RERANK command apart from other products that offer semantic reranking integrations is that it doesn’t require the input to represent a mapped field from an index. RERANK only expects an expression that evaluates to a string value, making it possible to do semantic reranking using multiple fields:

LLM completions

Now we have a set of highly relevant, reranked results.

At this stage, you might simply decide to return the results back to your application or you might want to further enhance your results using LLM completions.

If you’re using ES|QL as part of a retrieval-augmented generation (RAG) workflow, you can choose to call your favorite LLM directly from ES|QL. To achieve this, we’ve added a new COMPLETION command that takes in a prompt, a completion inference ID which designates which LLM to call, and a column identifier to specify where to output the LLM response.

In the following example, we’re using COMPLETION to add a new _completion column that contains the summary of the content column:

Each row now contains a summary:

_id

title

_score

summary

4001

The Hobbit

0.03279

Bilbo helps dwarves reclaim Erebor from the dragon Smaug.

3999

The Fellowship of the Ring

0.03226

Frodo begins the quest to destroy the One Ring.

4005

The Two Towers

0.0315

The Fellowship splits; war comes to Rohan; Frodo nears Mordor.

4123

The Silmarillion

0.03125

Ancient myths and history of Middle-earth's First Age.

4006

The Return of the King

0.3101

Sauron is defeated and Aragorn is crowned King.

4144

The Children of Húrin

0.01515

The tragic tale of Túrin Turambar's cursed life.

In another use case, you may simply want to answer a question using the proprietary data that you have indexed in Elasticsearch. In this case, the best search results that we’ve computed in the previous stage can be used as context for the prompt:

Since the COMPLETION command unlocks the ability to send any prompt to an LLM, the possibilities are endless. Although we’re only showing a few examples, the COMPLETION command can be used in a wide range of scenarios, from security analysts using it to assign scores depending on whether a log event can represent a malicious action or data scientists using it to analyze data, to cases where you just need to generate Chuck Norris facts based on your data.

This is only the beginning

In the future, we’ll be expanding ES|QL to improve semantic reranking for long documents, better conditional execution of the ES|QL queries using multiple FORK commands, support sparse vector queries, removing close duplicate results to enhance result diversity, allowing full text search on runtime generated columns, and many other scenarios.

Additional tutorials and guides:

• ES|QL for search

• ES|QL for search tutorial

• Semantic_text field type

• FORK and FUSE documentation

• ES|QL search functions

With Elasticsearch 8.18 and 9.0, ES|QL adds a host of new functionalities, including:

• support for scoring

• semantic search

• more configuration options for the match function

• a new KQL function

In this blog, we will review these 8.18 features and other exciting new features that we plan to add to ES|QL, reinforcing our investment in making ES|QL a modern search language ready to fit your needs, whether you are building a search application powered by ES|QL or analyzing your data in Kibana Discover.

Introducing scoring

In 8.17 we added the ability to filter documents using full text functions. If you are unfamiliar with full text filtering in ES|QL, we suggest reading our original blog post about it.

With 8.18 and 9.0 we introduce support for scoring, making it possible to return documents in order of their relevance. To access the score for each document, simply add the metadata _score field to your ES|QL query:

We retrieve the same scores we get from the equivalent search API query:

Full text search functions such as match, qstr and kql can only be used in the context of a WHERE condition and are the only ones that contribute to the score.

The _score column can not only be used to sort documents by relevance, but also in custom scoring formulas. In the next example, we keep only the most relevant results using a score threshold and then add a score boost based on the reader rating:

Improving the match function

In ES|QL, the match function simply translates to a Query DSL match query. In 8.18 and 9.0, we expanded the match function's capabilities to include all options that are currently available in Query DSL. It is now possible to set well-known match options such as boost, fuzziness and operator in ES|QL too:

Enter semantic search

The 8.18 release comes with the exciting announcement that semantic search is now generally available. We've expanded the match function to support querying over semantic_text field types.

In ES|QL, executing a semantic query is now as simple as performing a full-text query, as shown in this example:

In this example, we set semantic_title to use the semantic_text field type.

Mapping your index fields as semantic_text is all it takes to set up your index for semantic search.

Check our search with semantic text tutorial for more details.

Hybrid search with ES|QL

ES|QL makes it straightforward to do both semantic and lexical search at the same time. It is also possible to set different boosts, prioritizing results from semantic search or lexical search, depending on your use case:

Transitioning from KQL

If you are a long-term user of Kibana Discover and use KQL (Kibana Query Language) to query and visualize your data and you'd like to try ES|QL but don't know where to start, don't worry, we got you!

In 8.18 and 9.0, ES|QL adds a new function which allows you to use KQL inside ES|QL. This is as simple as:

ES|QL is already available in Kibana Discover.

This way, you get the best of both worlds: you can continue to use KQL and start getting more familiar with ES|QL at your own pace.

Check out our getting started with ES|QL guide for more information.

Beyond 8.18 and 9.0

In future releases, we'll be adding more and more search capabilities to ES|QL, including vector search, semantic reranking, enhanced score customization options, and additional methods for combining hybrid search results, such as Reciprocal Rank Fusion (RRF).

Try it out yourself

These changes are available starting with Elasticsearch 8.18, but they are already available in Elasticsearch Serverless. For Elasticsearch Serverless, start a free trial cloud today or try Elastic on your local machine now!

Follow the Search and filter in ES|QL tutorial for a hands-on introduction to the features described in this blog post!