The changes are about storage and performance, along with the out-of-the-box (OOTB) experience. Up until 9.5.0, Elasticsearch operated as a document-based search engine by default. It could be set up to behave like a columnar system storage-wise, but that wasn’t the OOTB experience. Columnar index modes make a number of fundamental changes that allow Elasticsearch to optimize columnar analytic and search use cases:

• Fields are stored once as doc values only and are no longer indexed by default.

• New multi-value semantics. The original ordering of multiple values per field per document (for example, in arrays) is preserved by default.

• Mappings are always flat, and object and passthrough fields in mappings are always auto-flattened.

How columnar index modes fit into Elasticsearch

Many of the columnar index mode changes originate from time series data streams (TSDSs). As part of making TSDB a competitive metrics solution, we improved doc values format on disk and only store dimensions and metric fields once as doc values. We also improved query performance. TSDB is already columnar today. Essentially, this makes TSDB’s storage mode columnar. The lessons learned from TSDB are now being applied more broadly to Elasticsearch. 

Note that columnar index modes are opt-in and columnar, and document-based indices can coexist in the same cluster. An enterprise search use case can use a document-oriented index mode, while a logging use case can use logsdb_columnar index mode and be fully columnar, all in the same cluster. In fact, there are currently seven index modes, and indices can all use them in the same cluster.

How columnar storage stays fast without an inverted index

Indexed fields, either an inverted index for string-based fields or block k-dimensional (BKD) tree for numeric fields, allow Elasticsearch to query or filter by field very efficiently. However, the cost for this is an additional expensive data structure that uses a lot of disk space and is expensive to build at index time and at merge time. With the columnar index modes, fields are no longer indexed by default, so what did we do for query performance to be still acceptable on fields that were no longer indexed?

One major change was improving doc values scanning performance. This is key and is the cornerstone that any columnar system relies on. Previously, the scanning of doc values was essentially document by document. Lucene’s doc values API only allowed for looking up one value at a time. Historically, this fit the execution model of a search engine. In our own doc value format, we build the capability to allow bulk reading of values for Elasticsearch Query Language (ES|QL) queries. Also over recent minor Lucene releases, Lucene doc values API added support for bulk reading. Without this, fast columnar scanning wouldn’t have been possible.

Secondly, we fully adopted doc value skippers, a hierarchical skiplist over doc values. Contrary to an inverted index or a BKD tree, doc values skippers are lightweight data structures. At its core, a skipper allows queries to skip over a range of documents that don’t match a query. It can do this because it stores information, like min and max values. So, for example, when a range query is executed, an interval of documents can be skipped based on the intermediate result and a doc value skipper’s min and max values. The effectiveness of doc value skippers depends on the order in which documents are laid out on disk. This is why index sorting should be enabled or altered to match the use case.

By significantly improving our columnar scanning and doubling down on doc values skippers, we’re able to avoid indexing fields by default. Note that a field can still be indexed; the index mapping attribute just defaults to false in columnar mode. The exceptions to this rule are text-based fields, which are still indexed by default. This is because text fields provide free text search, which includes text analysis, along with phrase and wildcard matching. This is different from just filtering.

How Elasticsearch handles high and low cardinality fields

When setting up a schema with string fields, an important configuration parameter is often cardinality; that is, whether many unique values or a few unique string values are expected. Many systems have dedicated field or column types that target low and high cardinality string fields.

Fields that have low cardinality are typically stored with a dictionary, containing all unique values. Then, for each row offset, the offset into the dictionary containing the term the row has is stored. This is often called an ordinal. For low cardinality fields, this works well, as storing an ordinal per document takes up much less space. Encoding techniques, like delta encoding, offset encoding, and bitpacking, work well for ordinals to compact the per-document storage to just a few bits. 

However, for high cardinality fields, the dictionary and ordinal approach can work counterintuitively. If a larger percentage of the documents have a unique value, building the dictionary becomes expensive and storage savings diminish. The dictionary then becomes another level of indirection for reading values. This is why most systems in that case store values in a columnar fashion using block-based compression. For example, values of multiple rows are stored in 128KB blocks using a sliding-window dictionary-based compression algorithm (like zstandard or lz4). This, in general, is a simple and effective method to store higher cardinality fields and avoids building and maintaining a dictionary. 

With document-based Elasticsearch, there are two ways to map a string: using either the keyword field mapping or one of the text-based field mappings. The former stores an inverted index and dictionary-based doc values. The latter only stores an inverted index. This is why, typically, a text field mapper is often used in combination with a keyword mapper as a multi-field. Also, keyword field mapper (as the name suggests) is meant for keywords or fields that have a lower cardinality and uses dictionary-based doc values implementation. However, in practice, keyword field mappers are also used for high cardinality.

In columnar mode, every field is only stored once by default. For keyword fields, this means only doc values are stored with no inverted index. Text-based fields now also store doc values and an inverted index by default. Text-based field mappers are different from the keyword field mapper, as these are not automatically used via Elasticsearch’s dynamic mapping logic for columnar indices, and therefore text fields keep storing an inverted index by default.

For both keyword- and text-based fields, we didn’t choose to expose a cardinality mapping attribute. It’s not always possible to know ahead of time whether a field is low or high cardinality. When flushing and merging segments to disk, Elasticsearch sees all values and can determine the cardinality of a field. This is why we’re choosing to automatically determine whether the usage of a dictionary and ordinal-based encoding is beneficial over block-based compression using a simple cardinality threshold. If a field is below this threshold, dictionary and ordinal-based encoding is used; otherwise block-based compression is used. This simplifies configuration of the mappings and makes it possible to automatically optimize storage as data evolves, since some segments may use dictionaries while others may use blocks for the same field. However, this is currently not ready yet and so, as part of 9.5.0, in columnar mode, both keyword- and text-based field mappers store values in doc values in a block-based compressed layout on disk.

The two approaches compare as follows:

Dictionary and ordinal encoding

Block-based compression

Suits

Low cardinality fields

High cardinality fields

What’s stored

A dictionary of unique values, plus one ordinal per document

Values for many documents compressed together in blocks

Compression

Delta encoding, offset encoding, and bitpacking reduce each ordinal to a few bits

Sliding-window dictionary compression, such as zstandard or lz4, typically over 128KB blocks

Read path

Resolve the ordinal, and then look up the value in the dictionary

Decompress the block, and then read the value directly

Cost as cardinality rises

Dictionary grows large, savings shrink, and the extra indirection stays

Stable, with no dictionary to build or maintain

Used in columnar mode tech preview

Not yet

Yes, for both keyword and text fields

Columnar mapping attributes: multi_value, nullability, on_failure

The columnar index modes provide more control over how data is stored as doc values. By default, Elasticsearch is lenient and accepts all non-malformed values (for example, nulls and multiple values per field and document). If documents have fields with multiple values per document or no value, doc values store additional data structures to deal with them and therefore implicitly increase costs. 

With columnar, new mapping attributes provide additional control. Note that these new mapping attributes are currently only available with the columnar index modes but will eventually also be available for all index modes.

Three new mapping attributes are involved:

Attribute

Default

Enforces

On violation

Available

multi_value

true

One value per document per field

Document indexing fails

9.5.0

nullability

true

Field must have a value

Document indexing fails

9.5.0

on_failure

fail

How the above failures are handled

Sets fail or ignore behavior

Next minor release

multi_value: Enforcing single-valued fields

By default, Elasticsearch accepts multiple values per document. To understand the implications of this, we first should take a look at how Elasticsearch (using Lucene’s doc values) stores a dense numeric field where all documents have a single value:

With this layout, all values are stored in blocks. The number of values per block depends on the index mode but is typically 128 values and is always the same within an index. All values in a block are encoded using various encoding techniques, like delta encoding and bit packing, so each block can have a different size, depending on how well the encoding techniques compress the values. This is why a block index is required. 

Lucene has the notion of a docid (internal numbering for a document), which is essentially a row identifier.

1. Queries produce matching docids.

2. In case of a dense field, the block id can be resolved from the docid directly.

3. The offset of a block can be resolved from the block index.

4. Once that has been looked up, the target block gets decoded and all values are available.

5. Finally, from docid, the ordinal within the decoded values array can be resolved, which produces the final value.

Now let’s have a look at how the data layout changes when documents have multiple values per document:

To determine how many values belong to a single docid, an offset lookup is required.

In this case, a docid has one or more offsets. Each offset points to a block index. Values for a single document are adjacent but can stretch over blocks. In general, compaction of values works well in blocks because values are similar. However, multi-value fields can cause the compaction of values to be less efficient if the number of values per field and document is large and values aren’t similar. This and the additional storage of offsets result in multi-value fields typically having a higher storage footprint on disk.

If a field is truly single-valued, you may want to enforce this property. The multi_value mapping attribute makes that possible now. An example is a log level field. Logs typically have one log level (such as debug, info, or error). Enforcing that this field is single-valued in your mappings can help avoid accidentally using more storage than anticipated. A mapping snippet example that disallows the field log.level to have multiple values per document:

Note that even if a field allows multiple values, this doesn’t mean an offset lookup is stored. This only happens when a Lucene segment has at least one document with two or more values. The multi_value mapping attribute exists just for enforcement.

nullability: Requiring every document to have a value

By default, Elasticsearch accepts documents with fields that have no value or null value. Just as  multiple values per document require additional accounting, documents with no value require additional accounting to identify which of them have at least one value.

Doc values store a docid to offset lookup (known as IndexedDISI in Lucene) in case not all documents have a value in a segment. The offset either points directly to the block index for single-valued fields or the offset lookup in case of multi-valued fields. This lookup is compact compared to the value blocks being stored. However, if it were to be created for fields that should have at least one value per document, that would be a waste.

The nullability mapping attribute allows you to control whether documents are allowed to have no value. Just like the multi_value mapping attribute, the nullability attribute exists for enforcement.  Following is a mapping snippet example that requires the log.level field to have a value:

on_failure: What happens when validation fails

What happens if a document has multiple values for a field and if the multi_value mapping attribute is set to false or when a field is mapped with nullability set to false and a document doesn’t have that field? At the moment, indexing such documents will fail with a bad request error.

As part of the next minor release, the on_failure mapping attribute will be available. This allows you to indicate how to handle these validation failures, on a per-mapped field basis. This will support two values:

1. Fail: Fail indexing of the entire document with a client error. This is the current behavior in Elasticsearch 9.5.0.

2. Ignore: Ignore the validation error, mark the field as ignored, and store values for that field in a hidden field so that it can be introspected when requesting the source. 

Trying the columnar index modes

The columnar index modes are still under active development, but we encourage you to give them a test drive. As we prepare the columnar index modes for general availability (GA), we’ll add more performance and efficiency improvements. We believe that by adapting a columnar mindset, many use cases will benefit from being more cost effective or having better performance characteristics.

In Elasticsearch 9.5, Elasticsearch Query Language (ES|QL) queries no longer fail when a field isn't in the mapping. The new unmapped_fields setting lets queries load values from _source or fill with nulls, so queries keep working even when a backing index changes and a field goes missing, and can use unmapped data without reindexing. Here’s how we built that: the design choices and the edge cases (including a class of fields we nicknamed PUNKs), along with the testing strategies that gave us the confidence to ship it in general availability (GA).

Why ES|QL queries fail when a field is unmapped

You built a visualization using an ES|QL query. You refined it, and the query grew. You’re at 15 chained commands and counting, but it does just the right thing. It works, and your dashboard is useful.

Your query uses an index from a remote cluster, say my-remote:logs-foo. But actually, logs-foo is an alias, and at some point, the remote cluster makes it point to a different backing index. The new index is missing a field that’s used in your query, and your query and visualization break.

Or maybe you have an already fairly large index, and while building ES|QL queries on top of it, you realize that you’d like to use a field in the indexed documents that unfortunately never made it into the index mapping. You could reindex the data, but that would take hours.

ES|QL’s unmapped_fields setting is meant to deal with these types of situations.

If your query looks like this:

and some_field is unmapped, ES|QL’s default behavior is to fail with a verification exception.

You can use the unmapped_fields setting to instead either fill some_field with nulls or read it from the document’s _source, like so:

How ES|QL resolves queries with field caps

Before we jump into the inner workings of unmapped_fields, we have to look into how ES|QL resolves queries regularly. Let’s consider the above query:

We said that if some_field isn’t in the mapping for index, ES|QL will reject the query. How does it make that decision?

How field caps tells ES|QL which fields exist

In a typical schema-on-write fashion, Elasticsearch clusters maintain mappings with their respective indices. As a first step, ES|QL makes an internal request to the field caps endpoint to determine which fields the index has. It then passes the query, together with the field caps response, to the query planner, which consists essentially of the query analyzer (unrelated to analyzers of text fields) and query optimizer. The analyzer makes sense of raw names, like some_field, and notices that they correspond to index fields (or not). If all went well, the query is then passed on to the optimizer, which rewrites the query for efficiency, before it’s handed to the compute engine for execution.

How the analyzer resolves field names in the query plan

Let’s zoom in to the analyzer. The parsed query is represented in a tree structure, and the analyzer partially rewrites it, one command at a time, until it either has resolved all references or not.

For illustration, let’s use a somewhat more complex query and see how the analyzer would resolve it:

The parsed tree is actually a chain here, and it looks something like this:

The analyzer then moves up through the query tree to try and resolve the field names used in every command.

This is a simplified version of how we represent parse trees in tests and when debugging. The bottom of the chain corresponds to the FROM command and contains a list of all mapped fields that we know about, obtained from the field caps endpoint. (The {f} suffix marks an actually mapped field for better distinction later.)

The two EVAL nodes on top of it correspond to the remaining commands, and their fields are still unresolved, expressed by the question mark ? in front of the name. At this point, the analyzer still has to check whether they correspond to existing index fields.

For the EVAL that defines uppercased_mapped, it can see that the previous command outputs mapped_field, so the unresolved ?mapped_field marker can be replaced by a real field reference:

Next, it encounters the topmost EVAL, which defines uppercased_unmapped. The previous tree nodes produce only two fields: [mapped_field, uppercased_mapped]. The reference ?unmapped_field thus has to remain unresolved. We bail here and emit the verification exception to the user.

How unmapped_fields LOAD and NULLIFY work

Adding unmapped fields to the query plan

When using unmapped_fields=”NULLIFY” or ”LOAD”, we do something else; we act as if the field was actually in the index. The analyzer adds unmapped_field to the From node and marks it as unmapped to signal to the compute engine that this has to be read from _source or filled with nulls. Let’s express this with a {u} (for unmapped):

After amending the From, the analyzer can continue trying to resolve the topmost Eval node. It sees that the upstream nodes produce the fields [mapped_field, unmapped_field, uppercased_mapped] and thus unmapped_field can be correctly resolved:

The query plan is now fully resolved and can be passed down the regular optimization-execution pipeline. Other than the actual value extraction mechanism, everything stays the same. Schematically, the workflow looks like this:

Example: enabling unmapped fields with the SET directive

To give an example, let’s fire up a cluster and create an index with non-dynamic mappings.

We can run the example query, above:

This should result in the error message:

Unknown column [unmapped_field], did you mean [mapped_field]?

To make things work, we can prepend SET unmapped_fields=”...”; with LOAD or NULLIFY:

Inspecting the analyzer's rewrite steps

If you want to see what the query analyzer is doing to the parse tree, you can log the query rewrite steps, like so:

This will log a line containing Rule rules.ResolveUnmapped applied with change… You’ll see that unmapped_field is added to the bottom of the parse tree as described above.

Why we have to infer the schema

Of course, this isn’t the only possible method to deal with unmapped fields. Here are some alternatives:

1. We could also scan or probe the documents in index to determine that their _source actually has the unmapped_field.

2. We could disable verifications in the analyzer and make the compute engine blindly pass unmapped fields through individual computation steps.

The first alternative front-loads more work to understand the actual schema of an index and thus generally increases latency. It doesn’t scale to large, highly distributed datasets. The second alternative isn’t viable since it means a large-scale change to how ES|QL’s compute engine is built, because it passes around streams of data with fixed columns from one operator to another.

In contrast, the approach we chose is neatly compatible with ES|QL’s existing optimization pipeline.

The trade-off is that the analyzer has to correctly infer a schema based on the actual index mappings (obtained from the field caps endpoint) and additional fields used inside the query. 

This isn’t always straightforward. There were two main challenges:

1. There are many different query shapes and commands that can be used. The mechanism needs to detect unmapped fields, update the proper FROM command, and pass the new field through the halfway resolved plan correctly in all cases.

2. There are many different mappings we have to deal with, and we specifically need to make our feature work correctly when mappings change over time on top of that.

In the following, we’ll focus on LOAD, although some problems (generally many fewer) also apply to NULLIFY.

Which index to load unmapped fields from for LOOKUP JOIN and FORK

To briefly illustrate the first problem, here are some choices we needed to make:

• Which index do we load from when using lookup joins? This one?

  The unmapped_field cannot be attributed to both indices. We chose index since this is where we expect mappings to change more often than in lookup indices.

• Similarly, how do we deal with subqueries and views or the FORK command? In the following:

  one fork branch triggers loading of an unmapped field. Is it also present in the other fork branch? (Yes, it should be, but it’s not obvious and is specifically not true if the two FORKs are replaced by independent subqueries.)

Two principles to keep queries working

The second problem, diversity of mappings and their evolution over time, is a far bigger driver for complexity. We strove for two basic usability principles:

• Queries that work in the default mode should generally still work when using unmapped_fields=”NULLIFY” and ”LOAD”.

• Queries that work when all fields are mapped should generally still work with NULLIFY and LOAD when a field becomes unmapped and vice versa.

The type of unmapped fields and inadvertent type conflicts

Let’s talk about data types to see where this leads to complexity. First, when using unmapped_fields=”LOAD”, we need to assume a data type for unmapped fields. We chose KEYWORD, which allows us to avoid type conflicts when reading from _source. One document can contain ”unmapped_field”: “foo”, and another can contain ”unmapped_field”: 123.4. It’s fine because we treat both as strings.

However, this is a violation of the second principle when a non-KEYWORD field happens to go unmapped. Consider this query:

If some_field becomes unmapped, we’ll have to assume that the KEYWORD type and the query will fail with a type conflict.

Type conflicts aren’t new and can be dealt with by using explicit casts in the query, like so:

It would be great if ES|QL just inferred a useful type to cast to, but this is something for the future.

Type conflicts with partially unmapped fields, or: making PUNKs well behaved

In addition to fully unmapped fields, partially unmapped fields are everywhere and should also work with LOAD. Let’s look at a query that uses multiple indices.

Let’s say that there are indices index and index_without_some_field, containing just the following documents.

Now let’s consider the query:

and assume that some_field is unmapped in index_without_some_field. This will return:

because ES|QL doesn’t load unmapped fields per default.

Of course, when setting unmapped_fields=”LOAD”, we want to load from _source for index_without_some_field:

As with fully unmapped fields, the case is simple when some_field is mapped as KEYWORD in index. When loading from _source for index_without_some_field,  we treat the field as KEYWORD as well, so there’s no conflict.

What makes a field a PUNK

The case is less clear when some_fieldis partially unmapped and the mapped leg is of a type other than KEYWORD. Such fields caused a lot of trouble until we found the best solution, which makes their acronym quite fitting: partially unmapped non-keyword fields, or PUNKs.

Unfortunately, PUNKs are far from being esoteric. For instance, it’s very natural to filter on a PUNK:

If some_field is mapped as INTEGER in index, the type conflict looks like this:

• Mapped as an INTEGER in index.

• Unmapped in index_without_some_field and thus treated as KEYWORD.

This can again be resolved manually by providing an explicit cast:

But this is far from acceptable. Even queries that work fine without NULLIFY and LOAD typically have some PUNKs; the unmapped leg is simply treated as null then. Both guiding principles are violated if LOAD requires an explicit cast here.

Casting implicitly to the mapped type

The solution is to introduce an implicit cast to the mapped type. In this case, we know that some_field is an INTEGER in index, and thus we treat it essentially as if the user wrote:

This means that queries that work without LOAD keep working. (ES|QL may even give you more data because we load the unmapped leg of PUNKs from _source.) Queries that used to work when a field is fully mapped also keep working when it goes unmapped in some (but not all) of its indices without having to alter the query in any way.

Behavior

Default

NULLIFY

LOAD

Unmapped field in query

Query fails

Query runs

Query runs

Values returned

None

null

Read from _source

Assumed type

n/a

NULL

KEYWORD

Partially unmapped field (PUNK)

Unmapped leg is null

Unmapped leg is null

Cast to the mapped type

Pushdown optimization

Full

Full

Per-node where fully mapped

Don't throw it all away: Keeping ES|QL query optimization with unmapped fields

There's one more thing to get right; that is, to make sure that optimizations still work correctly with LOAD. Consider the previous query:

ES|QL’s optimizer aggressively pushes down such WHERE filters and turns them into Lucene queries, so the compute engine doesn’t perform unnecessary work.

For this query, evaluating the filter in the compute engine would require fetching each and every document from the index; meaning, a full scan, very slow. If some_field was mapped as an INTEGER in both indices, we would instead perform a Lucene query, which looks like this:

The compute engine then doesn’t have to load each document separately and check whether it matches the filter. Documents with some_field <= 10 are never fetched from the Lucene index, which is very efficient at this kind of filtering. Nice.

Why filter pushdown is unsafe for unmapped fields

If some_field is unmapped in index_without_some_field, however, it’s wrong to narrow the documents down using the same Lucene query, as Lucene interprets an unmapped some_field as null and thus no documents from index_without_some_field will ever match. This edge case is easy to miss, and it doesn’t help that there are several flavors of similar pushdowns. For instance, in the query:

the compute engine pushes even the counting to Lucene. Again, this is only correct if some_field is fully mapped.

This means that such optimizations can't apply to unmapped fields. It would be disappointing if a query used hundreds of indices and only one of them happened to not map some_field, causing the whole query to run unoptimized.

How the local optimizer recovers the fast path

Luckily, this problem has a solution, too. ES|QL actually has multiple optimizer runs:

1. First, a preliminary optimizer run on the node handling the _query request.

2. Then, a second, local optimizer run on every node we fan out to because we need to fetch documents from its shards.

The workflow after the initial optimization looks more like this:

If the current node happens to map some_field in all shards, the local optimizer detects this situation and treats some_field like any other fully mapped field, including performing Lucene queries to greatly narrow down the dataset to be processed. In fact, data nodes process LIMIT queries like:

in batches of shards (to avoid loading too much data too eagerly), which includes a full local optimizer run per batch. This makes it even more likely to encounter batches where some_field is fully mapped, allowing ES|QL to run a fast Lucene query.

Is it working now? Testing unmapped_fields across every ES|QL query shape

As we have seen from the optimizer issues above, problems can hide in plain sight, even for very simple queries. Because unmapped_fields=”LOAD” can affect each and every kind of query, the surface area for bugs is essentially all of ES|QL.

Accordingly, getting good test coverage was tricky and challenged us to refine our testing strategies.

Reusing spec tests with unmapped_fields

Conveniently, ES|QL has an extensive corpus of test queries, together with expected result sets; we call them spec tests because they’re written using a simple text specification language, which looks roughly like this:

This lets us create new tests out of the existing ones by introducing slight variations. For instance, any existing test that runs without SET unmapped_fields=”...” should produce the exact same results when run with SET unmapped_fields=”NULLIFY”.

It also helped find major issues early in the development process, especially for NULLIFY. The LOAD setting changes the meaning of queries much more dramatically, limiting the usefulness of this approach. However, ES|QL also uses what we call generative testing; that is, we string together random commands, run the query, and then check whether the server reports a bug. This approach cannot confirm the correctness of results, but it still helped greatly with finding query types that didn’t work properly and resulted in some kind of error. (Property-based tests would be a refinement in the future by running the queries against a reference implementation. This way, correctness of results can also be checked.)

Testing type conflicts across different mappings

In the end, one of the most important testing dimensions was using different indices with various mappings in the same query. (Recall how, above, we had to deal with type conflicts to come up with a solid approach for PUNKs? It doesn’t end there; all kinds of type conflicts are more complex with LOAD.) Since we couldn’t automatically generate correct expected results, ES|QL’s test suite had to grow by adding more than 10,000 lines of CSV spec tests. Fortunately, adding such tests is a well-suited task for an AI agent, which has cut down the effort dramatically. (Of course, the test results were still reviewed by humans.)

All testing strategies together provided us with good confidence for the GA release of unmapped_fields with Elasticsearch 9.5.

Why dynamic mapping breaks down with schemaless data

Elasticsearch wants a schema. Every field you index has a mapping that defines its type, its analyzer (for text), and how it is stored. That schema makes storage compression efficient, with fast search and cheap aggregation. It becomes a liability the moment your data stops looking like a database table.

Consider the shapes that show up in real systems:

• Log events where every service adds its own attributes. One emits labels.region, another labels.k8s.pod, and another labels.tenant_id.

• OpenTelemetry resource attributes, where the set of keys is defined by whatever agent happened to send the span.

• User-supplied metadata bags, feature flags, or tagging systems, where the keys are open-ended by design.

If you map each key as its own field, the mapping grows unbounded. This is the classic "mapping explosion." Thousands of dynamically created fields inflate the cluster state, slowing down every mapping update and eventually hitting the field limit. Each field also carries overhead in the index. You pay a structural cost for keys you didn’t plan for and may query only once.

The naive escape hatch is to store the whole object as a string and give up on querying its contents. That trades one problem for another: you keep the data but lose the ability to filter or group by anything inside it.

The flattened field type is the better path. You can index an entire JSON object under a single mapped field, keeping the keys queryable with almost none of the mapping-explosion cost. 

This post covers how a flattened field stores the data on a disk and how ES|QL reads it back.

How flattened fields index dynamic JSON keys under one mapping

Map one field as flattened:

Then write arbitrary nested JSON into it:

There is exactly one field in the mapping, labels, no matter how many keys appear across your documents. The cluster state doesn’t grow when a new key shows up. The subkeys remain individually searchable. You can reference labels.region or labels.k8s.pod in queries, even though neither was ever declared.

The catch and central tradeoff is that every leaf value is a keyword. The number 4 above is indexed as the string "4". There is no numeric typing, no date parsing, and no range math on dynamic keys. Flattened fields exchange per-field richness for schema flexibility. That fact explains almost every design decision that follows.

How Elasticsearch stores flattened field JSON keys on disk

There are two types of queries on flattened fields: an unkeyed query on the root flattened field, and a keyed query on a specific subfield. Following the previous example, a query of the form labels: "us-east-1" matches a value under any key, while the querylabels.region: "us-east-1" matches a value under the specific regionkey.

To support these two distinct query formats, the flattened mapper writes each leaf value into two distinct Lucene fields.

Take this document:

The mapper produces:

• A root field under labels, holding the bare values:

• A keyed field under labels._keyed, holding the flattened key concatenated with its value:

In the keyed field, nested objects are dot-flattened into a single key (k8s.pod), and the key is joined to its value with a reserved NULL byte (\0) as the separator. Keys that contain a NULL byte are rejected at parse time, so the separator is always unambiguous. To find the value, you split on the first NULL.

These two fields make both query shapes work:

• labels: "us-east-1" matches a value under any key, so it searches the root field.

• labels.region: "us-east-1" matches a value under a specific key. It rewrites the query to the term region\0us-east-1 and searches the keyed field.

Query restrictions on flattened field subkeys

Because every key's terms live in one sorted list, the keyed field cannot answer every query shape a plain keyword field can. Three restrictions follow:

1. No fuzzy, regexp, or wildcard on a specific subkey. Nothing about the layout makes them impossible,  but the pattern would have to be combined with the key prefix so it can’t walk past the key boundary. The mapper doesn’t do that at this writing.

2. Any range query on a subkey needs at least one bound. Elasticsearch already has a query for "this field has some value here, whatever it is": the exists query. On a flattened subkey it runs as a prefix query on key\0, which sweeps every term belonging to that key. A range with neither bound would sweep exactly the same terms. Rather than support two spellings of one scan, the mapper rejects the boundless range and asks for the exists query.

3. A range query on a subkey needs the field to be indexed. A flattened field can be mapped with index: false, which skips the inverted index and keeps only doc values, the columnar per-document storage covered in the next section. Exact-match queries survive that. With no terms to look up, Elasticsearch scans the doc values column instead, which is slower but gives the same answer. Range queries have no equivalent fallback, so a range on a subkey of an unindexed flattened field throws an Exception.

All three are limits on the Lucene query the mapper is willing to build, and where you notice them depends on how you query.

On the search API, where you name labels.region directly, they come back as errors.

In ES|QL you will not see them as errors at all. There, the same limits decide only whether a predicate is pushed into Lucene or runs in the compute engine on the extracted column. 

This is pushed to a term query on the keyed field:

This is not, so the filter runs per row on the extracted keyword:

Same answer, more work. That distinction is the subject of the second half of this post.

Under the hood: how range queries stay inside key boundaries

This part is internal. You don’t need it to use the field, but it explains where the bounds rule comes from.

For a handful of documents in one segment, the shared term list looks like this:

Each key owns a contiguous slice of that list. For example, all values for key "region" are clustered together in an ordered sublist. A range with both bounds set encodes each bound the same way a term is encoded, so a lower bound of "us-east" on the region key becomes region\0us-east and an upper bound of "us-west" becomes region\0us-west. Both endpoints already carry the key prefix, so the scan can’t leave the region slice. Nothing special is needed.

The half-open case is problematic. Handing Lucene a lower bound of region\0us-east with no upper bound would scan to the end of the term list, straight through tenant_id\0acme and every other key that sorts after region. So the mapper substitutes a sentinel for the missing side:

• A missing lower bound becomes key\0, inclusive. That’s the encoding of the empty value, and it’s the first term in the key's slice.

• A missing upper bound becomes key\1, exclusive. Byte 0x01 is the next byte after the 0x00 separator, so it sorts after every key\0value term and before the first term of any other key.

A one-sided range is therefore boxed into [key\0, key\1), which makes it exactly as safe as a closed one.

The upper sentinel also covers the case where one key is a prefix of another. If an index holds both region and regionx, then region\1 still sorts below regionx\0eu-west-1, because 0x01 is smaller than the x that follows the shared region prefix. A range on region cannot leak into regionx.

How flattened fields use the inverted index and doc values

Each leaf value can be written into two Lucene structures. Both are enabled by default, but can be disabled by the mapping configuration.

• The inverted index (when the field is indexed). Two untokenized keyword terms are indexed per value, one on the root path and one on the keyed path. This powers term, prefix, and range searches.

• Doc values (when doc_values is enabled). A columnar, document-ordered structure. This powers sorting, aggregations, and ES|QL reads.

The inverted index answers the question: "Which documents contain this term?" 

Doc values answer another question: "For this document, what are the values?" Doc values are laid out column by column so a scan touches only the bytes it needs. A flattened field uses both inverted indexes and doc values, so it can serve search and analytics from the same field.

The nature of the index means that the root field is only present in some cases. When the inverted index is disabled by the mapping, any value search requires a linear scan of the doc values for the searched value. In this case, when performing a search on the root field, there isn’t much additional overhead compared to just scanning the keyed field and ignoring the key markers. So the flattened mapper skips writing the root field, relying on the keyed field for both root and keyed queries.

Why flattened fields switched from dictionary to binary doc values

Historically, flattened field doc values used Lucene's SortedSetDocValues, which is a dictionary-compressed format. This means that every key\0value value indexed across all documents per segment is stored in one big sorted, deduplicated set of values. Each document tracks a list of ordinals into that value set.

This dictionary approach provides great compression for low-cardinality fields that tend to have repeated values. It is byte-efficient to store a value only once, and then refer to it by a single integer value. However, there is overhead associated with building and maintaining that dictionary of values, and that overhead is wasted effort when operating on high-cardinality fields that don’t repeat values.

Because flattened fields are a catch-all type, their cardinality tends to be very high. So while the dictionary approach works, it’s not the most compact or scan-friendly layout for this data, especially in time-series indices where flattened bags are common and storage pressure is real.

Recent versions switched the storage to use Lucene’s BinaryDocValues. This format just stores a literal binary blob for each document, which is compressed using Zstandard by our doc values codec when written to disk.

This new binary format provides an additional benefit: it allows us to maintain original array ordering without any overhead. Flattened fields support the mapping parameter preserve_leaf_arrays, which affects how multivalued fields are returned when using synthetic source. When configured to preserve_leaf_arrays: exact, returned values preserve the order, duplicates, and nulls from the original source value.

The dictionary encoding inherent to sorted-set doc values means the returned values are sorted, deduplicated, and de-nulled. To implement preserve_leaf_arrays, flattened fields have traditionally used an additional sidecar field, tracking the required metadata to reconstruct the original source value. However, the nature of binary doc values means that this sidecar field is no longer needed. The values are just stored and returned as indexed.

Limitations of schema on read with flattened fields

The limitations below follow from the keyword-only rule and the shared keyed field:

• No numeric, date, or Boolean typing on dynamic keys. 100 and "100" are the same term.

• No fuzzy, regexp, or wildcard on a specific subkey.

• No multi-fields (fields) or copy_to on the flattened field.

• A depth_limit (default 20) on how deeply nested the object can be.

If you need real typing for a known key, flattened now supports explicitly mapped subfields: you declare individual keys with real types under a properties block, and those keys are indexed by their own typed mapper instead of the keyed field. You get numeric ranges on labels.status_code, while everything else in labels stays dynamic and keyword-only. This is the escape hatch for the handful of keys you actually know about.

Schema on read: querying flattened field JSON keys in ES|QL

Search has supported flattened fields for years. ES|QL, the newer, piped query language built on a columnar compute engine, now reads flattened fields, too. Support began in the Technical Preview of Elasticsearch 9.5.0. It has two distinct pieces.

What ES|QL returns when you select a flattened field

ES|QL uses a dedicated data type for flattened fields, rather than folding it into keyword. When you select the root, you get the whole object back as a JSON string:

Keys come back sorted. You can carry this value through a query, count it, group by it, and run the multi-value and comparison functions on it. But an opaque JSON blob is not usually what you want to filter or aggregate on. For that, you need to reach inside it and process its contents.

How FIELD_EXTRACT reads JSON keys from flattened fields

There is no dotted-path syntax for dynamic keys in ES|QL. You cannot write labels.region for an unmapped key, because to the engine the flattened root is a single leaf value, not a set of columns. Instead you use a function:

The absence of a dotted-path syntax is a tentative limitation. The keyed field already addresses individual subkeys, so a more natural syntax for reaching into a flattened root is something we are planning to support.

FIELD_EXTRACT(field, path) takes a flattened field and a key, and returns a keyword. The rules are easier to see against a document. Take this one:

And this query:

The result is:

Four things to take from this:

• The dot is part of the key, not a navigation operator. The mapper already collapsed the nested object into the flat key k8s.pod, so "k8s.pod is a direct lookup, not a walk from k8s to pod.

• Matching is exact. "k8s" returns null because there is no leaf stored at k8s, only at k8s.pod and k8s.node. For the same reason, "host" will not find "host.name", and matching is case-sensitive, so "Region" will not find "region".

• Arrays come back multi-valued. "tags" yields a multi-valued keyword you can MV_EXPAND, count, or filter on. A missing key yields null.

• Everything is a keyword. "retries" comes back as the string "4", and a Boolean leaf comes back as "true" or "false".

JSONPath syntax is rejected outright, at parse time rather than per row. Both FIELD_EXTRACT(labels, "['k8s.pod']") and FIELD_EXTRACT(labels, "tags[0]") fail with field_extract path must be a literal flattened sub-field name.

Once extracted, the value is an ordinary keyword column. You can filter on it, group by it, sort by it, or use it as the join key in a LOOKUP JOIN:

How ES|QL pushes flattened field predicates into the columnar store

The obvious way to implement FIELD_EXTRACT would be to read the whole flattened root out of storage, render it as a JSON string, hand it to the compute engine, and parse it once per row to pull out a single key. But that means reading every key to use once and paying for a JSON parse on every document. So this obvious implementation is not performant.

ES|QL avoids this whenever possible. Between storage and the compute engine sits the block loader, the step that turns stored data into the columnar blocks the engine operates on. FIELD_EXTRACT hooks into that step instead of running after it.

When ES|QL loads a column, the flattened field type inspects the request. If the request is an extraction of a single constant key and the field has doc values, it routes straight to the keyed doc-values loader, which reads the key\0value entries for only that key out of the columnar structure. The column that arrives at the compute engine already contains only that key's values. The JSON string is never built and never parsed, and the other keys in the object are never read.

When extraction can’t be fused into the block loader, for example, because the key is computed per row or the root is the output of another function such as CASE, ES|QL falls back to the parse-per-row path. The results are identical either way. Only the cost changes.

The comparison can push down further. A predicate like FIELD_EXTRACT(labels, "region") == "us-east-1" can be pushed to Lucene as a term query against the synthetic keyed field, the same region\0us-east-1 term the search path uses. So a filter on an extracted subkey can be answered by the inverted index (if available), and the projection can be answered by doc values, exactly like a first-class field, even though the key was never in the mapping.

Ordering comparisons push down, too. The four, single-sided comparators (>, >=, <, <=) and closed BETWEEN-style ranges all become a range query on that same synthetic keyed field. This is where the key\0 / key\1 sentinels earn their keep: the single-sided forms are only pushable because the mapper can box an open bound inside the key. The pushed range is treated as a candidate, and the predicate is re-evaluated on the extracted keyword column afterwards, so multi-valued keys don’t slip through.

The values are keywords, so the ordering is lexicographic, not numeric. FIELD_EXTRACT(labels, "retries") > "10" compares strings, which means "9" is greater than "10". If you need numeric ranges on a key, map it explicitly under properties, or cast the value in ESQL.

Explicitly mapped subfields behave differently on purpose. Because they carry real types, comparison semantics diverge from the keyword path, so they are loaded and compared through their own typed mapper rather than fused into the keyed loader. And when you select a flattened root that has mapped subfields, ES|QL loads it from _source, so every leaf renders as a string and no keys are dropped silently.

When to use flattened fields vs. dynamic mapping in Elasticsearch

Use flattened fields when:

• The set of keys is open-ended or unknown ahead of time.

• You would otherwise cause a mapping explosion.

• Keyword-level filtering and grouping on the values is enough, and you don’t need numeric or date semantics on the dynamic keys.

• You have a few keys that do need real types. Map those explicitly under properties, and let the rest stay dynamic.

Avoid it, or map fields normally, when the schema is stable and you need full-text analysis, numeric aggregation, or date math across the board.

Remember that flattened is not a dumping ground for JSON you have given up on. It’s a real columnar-and-inverted store for schemaless data, and with ES|QL support, it’s now a first-class analytical citizen. You can keep the messy, unmapped parts of your data messy, and still filter, group, join, and aggregate across them as needed.

How MATCH and TO_TEXT enable full-text search on any ES|QL expression

Let's start with a query that was impossible in Elasticsearch 9.4, which uses the EVAL command:

In this example, summary has no mapping or analyzer configuration. It’s also not associated with any inverted index. It exists only for the lifetime of this query, but now you can search it anyway. Two additions make this work.

First, MATCH now accepts any expression as its first argument, not just a mapped field. That includes columns produced by EVAL and function results used inline. It also includes unmapped fields loaded directly from the original document. Furthermore, all data types normally accepted by MATCH are supported in this new use case.

The second part of this is the new TO_TEXT function, which is the first ES|QL conversion function that produces output of type text. Until now, text columns could only come from indexed mapped fields, and all strings produced by ES|QL expressions were keyword values rather than text. The distinction matters because MATCH treats the two differently: text values are analyzed, while keyword values are compared exactly, mirroring how a MATCH query on an indexed keyword field rewrites to a term query. TO_TEXT(x) is how you tell ES|QL: treat this string as full text.

This ships as a technical preview in Elasticsearch 9.5, and as such, it has some limitations:

• It’s currently filtering only. A MATCH on an expression doesn't contribute to the relevance score yet; only matches on indexed fields affect the score.

• Querying options like fuzziness and others aren't yet supported when matching an expression.

• Runtime text is analyzed with the standard analyzer. This isn’t configurable yet.

Work is underway to address these limitations.

Why use full-text search instead of LIKE or RLIKE in ES|QL?

ES|QL already had two ways to search strings without an index: LIKE (wildcard patterns) and RLIKE (regular expressions). Both work on any string expression, so it's fair to ask what MATCH adds. The answer is analysis, a more advanced form of search which uses techniques such as stemming and synonyms. It also uses stopword handling.

LIKE is simple substring matching, without any understanding of the words that comprise a string. Say, for example, you're looking for log messages about a fox:

This misses "Fox spotted near the henhouse" due to the capitalization, while matching "Outfoxed by the competition", which isn't about a fox at all. It fails in both directions, with false negatives on capitalization and false positives on substrings buried inside other words.

Regular expressions can patch the case problem, but the word-boundary problem gets ugly fast. Something like:

And even that's not right yet. It misses a fox at the end of a sentence followed by ! or ?, and it says nothing about tabs, quotes, or parentheses. Each fix makes the pattern longer, and the next person to read the query has to reverse-engineer what it’s actually doing.

MATCH makes the problem go away, because it runs both the query and the value through an analyzer, which tokenizes text into lowercase terms and then matches term against term:

This query will match values like "The quick brown fox" and "FOX spotted near the henhouse" but not "Outfoxed by the competition" or “FOXTROT protocol enabled", regardless of any punctuation surrounding the words. Of course, this all works for multi-term queries, like MATCH(TO_TEXT(message), "brown fox"), too, just the way you’d expect it to.

Work is underway to enable the use of the 36 dedicated language analyzers, with support for natural languages on data that was never indexed or mapped.

Full-text search use cases for unindexed and unmapped data

The examples above searched values computed from mapped fields. . The more interesting use cases for ES|QL MATCH on expressions involve data that was never searchable at all. Let's walk through a few.

How to search unmapped fields in ES|QL without adding a mapping

Sometimes you deliberately leave a field out of your mappings, such as a verbose stack trace or a raw request payload. You might even leave out a debug blob. Indexing one of these would cost disk and heap space on every document and wouldn’t be worth it for a field you might query once a quarter.

That decision has always been final, because unmapped fields were invisible to queries entirely. In Elasticsearch 9.5, you can use SET unmapped_fields="load" to make ES|QL load unmapped fields directly from the source document as keywords. Follow that up by wrapping it in TO_TEXT, and now you can run full-text search on it:

Here, stack_trace was never mapped. Every value is fetched from the original documents and analyzed on the fly. They’re matched row by row. That’s real work, and it will never be as fast as an inverted index lookup. But now, that field you didn't index is no longer unsearchable. You get to keep the mapping small for the everyday case and still answer the once-a-quarter question when it matters.

Full-text search on a keyword field without reindexing

Keyword fields can do a lot. They give you exact matching, fast aggregations, and sorting, which is why so many fields end up mapped that way. But mappings are decided when data arrives, and it’s easy to end up in a situation where you want to do something different with your data than you had originally intended. Maybe product_name was mapped as a keyword because the dashboards aggregate on it, and then, after receiving a year’s worth of product data, someone wants to be able to search within product_name values.

The old answer was to change the mapping to text (or add a multi-field) and reindex everything. This can be both time-consuming and costly, and in many cases, users simply won’t want to bother with it. The new answer is one function call:

TO_TEXT converts the keyword values to text on the fly, so MATCH analyzes them instead of comparing them exactly. This allows you to query a keyword field without creating a mapping or reindexing the source document. If the search becomes an everyday query, indexing the field as text is still the right long-term move, but TO_TEXT gets you an answer today, without any extra work.

Searching the same field across indices with different mappings

ES|QL can span many indices, and the same field doesn't need to always look the same in all of them. When the same field has different types in different indices, ES|QL treats it as a union type, and a conversion function resolves the conflict. Let’s consider an example in which the message field has type text in this year's index template but was a keyword in last year's:

Every value is analyzed at query time, whether it came from the text index or the keyword one. The keyword values from the older indices are tokenized and lowercased like everything else, so "connection reset" finds “Connection RESET by peer”, no matter which index it lives in.

Another interesting case is when a field is mapped in only one index but also present (and unmapped) in the other:

There's a nuance worth calling out here. If error_details is mapped in logs-2026 but not in logs-2025, Elasticsearch cannot push this query down to Lucene, because the indices where the field is unmapped would silently return no matches. Instead, the planner notices that the field is potentially unmapped and evaluates the whole MATCH row by row, wherever the rows came from. You don't have to know which of your indices have the field mapped; the query just answers the question.

How ES|QL analyzes text at query time without an inverted index

When ES|QL plans a MATCH against an expression, it analyzes the query string once, up front, into a set of terms. How each row is then evaluated depends on the expression's type:

Expression type

Processing

Matching behavior

text (via TO_TEXT)

Analyzer tokenizes value into lowercase terms

Token-against-token comparison; a row matches if any token equals any query term (OR semantics)

keyword,ip, date, numeric

No analysis; query constant converted once to the native type

Exact comparison per row

Both paths bypass Lucene entirely and evaluate values row by row. The non-text path mirrors exactly what a match query does when pushed down to Lucene against those field types, so the semantics stay consistent regardless of whether your query hits an index.

An inverted-index lookup does its work at ingest time and never touches non-matching documents at query time. A runtime MATCH does that analysis at query time, for every row that reaches it. One is fast because the work already happened; the other is flexible because the data doesn't need to have been indexed at all.

What's next for ES|QL full-text search

Everything in this post is the first installment of a larger effort to make search in ES|QL work on anything, not just on what you indexed ahead of time. The limitations called out earlier are actively being worked on, and the roadmap goes further:

• Scoring. Runtime matches will contribute to _score, so you can sort by relevance even when the data was never indexed.

• MATCH_PHRASE on expressions. Already available in Elastic Cloud Serverless, and coming to the Elastic Stack in 9.6.

• Configurable analyzers. Analyzer support for MATCH and MATCH_PHRASE on expressions, enabling language analyzers, stemming, and synonyms at query time.

• Match options. Options like fuzziness and operator for runtime matches.

• Vector search. Generating embeddings per row and running k-nearest neighbors (kNN) on runtime dense_vector expressions, bringing semantic search to unindexed data, too.

Try ES|QL full-text search on expressions today

You can try runtime search today. It's available now in Elastic Cloud Serverless, where new ES|QL capabilities land first, and it ships as a technical preview in Elasticsearch 9.5. Start with the search functions reference, and check the ES|QL limitations page for the current boundaries. It's a technical preview because we want your feedback: If you search something that was never indexed and it surprises you, either positively or negatively, we'd love to hear about it.

The palette takes shape: how multimodal search in Elasticsearch evolved from semantic_text

The semantic field is a convergence of several complementary features we've introduced over the past couple of years, bringing them together to create a cohesive multimodal search experience. Each solved an important piece of the semantic search puzzle on its own; together they enable native multimodal search.

The first brushstroke was semantic_text. Before it, running semantic search meant manually configuring mappings, wiring up ingest pipelines with an ML model, manually chunking content, and generating query-time embeddings yourself. The semantic_text field folds all of that away: it performs inference automatically at ingest time, chunks long documents for you, and simplifies the queries you write against it. Introduced in Elasticsearch 8.15 and released as GA in Elasticsearch 8.18, it has become the foundation for semantic search on the platform.

Next came the model to power multimodal search. jina-embeddings-v5-omni is our family of multimodal embedding models, capable of embedding text, images, video, audio, and PDFs into a shared vector space. Because those embeddings are semantically compatible across modalities, you can store diverse media in a single index and query across all of it at once, such as retrieving an image via a text description or matching audio against a written phrase, all without maintaining a separate pipeline for each content type. For more detailed information about how these embeddings are generated, see the model documentation.

We added the embedding query vector builder in Elasticsearch 9.4 to handle multimodal inputs at query time. Query vector builders are general-purpose tools you can use to convert input to a vector at query time as part of your request. For example, we have the text_embedding query vector builder for text-only models and input, and the lookup query vector builder for getting a vector from an existing document. The embedding query vector builder is a new type that works with multimodal models and accepts multimodal input, including text or base64-encoded binaries. This allows you to pose a query in whatever modality fits, and Elasticsearch generates the matching vector on the fly.

The final piece was multimodal ingest. The semantic_text field brought automatic embedding to text; the semantic field extends that same automatic experience to images, audio, video, and PDFs from ingest through query.

Painting the picture: creating an index with the semantic field

Let’s create an index with a semantic field. This is as simple as setting the field type to semantic and defining the inference endpoint you want to use:

In this example, we use the .jina-embeddings-v5-omni-small inference endpoint. This is our built-in jina-embeddings-v5-omni inference service, and it is available in all environments with access to the Elastic Inference Service (EIS). This includes:

• Serverless.

• Elastic Cloud Hosted (ECH).

• Self-managed with Cloud Connected Mode (CCM).

Indexing images, audio, video and PDFs

To index an image, provide an object with a type of image and a value containing the image as a base64-encoded data URL:

Arrays of objects are also accepted, allowing you to index multiple images in a single field value:

The semantic field also supports text values, just like semantic_text. You can provide such values standalone or intermix them with image values:

Text values are handled just like they are with semantic_text: long passages are chunked according to the chunking settings configured on either the inference service or field mapping. Multimodal values, such as images, are not chunked. Each multimodal value is represented as one chunk.

Other modalities are supported as well. Change the type value to match your content’s modality. Currently we support:

• image

• audio

• video

• pdf

For example, to index a video, the request would look like:

Image search and cross-modal retrieval with a text query

To find multimodal content using a text description, run a match query on the semantic field:

Just like with semantic_text, Elasticsearch automatically generates an embedding for the query text using the inference endpoint associated with the field. That query embedding is used to return semantically similar matches.

This query pattern enables easy text-to-image search. Just index an image and use a match query to retrieve it via text description! It also works for any other modality: index the multimodal input and search by description to retrieve it.

Querying with images, video, and other multimodal inputs

We can also search using a multimodal input by using the knn query with an embedding query vector builder. For example, we can search using an image:

The input object format is the same as when providing an image to index: set the type to image and value to a base64-encoded data URL.

Similar to when querying by text description, Elasticsearch automatically generates an embedding for the query image using the inference endpoint associated with the field. That query embedding is used to return semantically similar matches.

Just like with indexing, other modalities are supported, but are limited to those supported by your inference endpoint. For example, a search using a video clip would look like:

Extending the composition: highlighting, retrievers, and other semantic field features

The semantic field didn't start from a blank canvas. It's built on the same foundation as semantic_text, inheriting its behavior and its ergonomics, and extending them to multimodal content. In practice, that means nearly everything you already know about working with semantic_text carries over unchanged. If you've built with semantic_text before, the semantic field will feel immediately familiar.

Here’s a selection of the features that come along for the ride. See the documentation for a complete list.

Highlighting the best-matching chunks

If you index multiple values in a semantic field, you may want to know which value best matches the query. The semantic highlighter can be used to return the most relevant chunks as highlight fragments:

Setting order to score returns the fragments ranked by relevance, while number_of_fragments caps how many chunks come back. The response looks like:

Note how highlighted multimodal values are represented using their data URLs.

Controlling vector quantisation with index options

The semantic field stores its embeddings in an underlying vector field, and index_options lets you control how that vector field is indexed. For example, choosing a non-default quantization strategy:

Multi-field retrievers

The semantic field participates in the multi-field query format supported by the linear and rrf retrievers. Rather than hand-writing an inner retriever per field, you supply a single query and a list of fields, mixing lexical fields and semantic fields freely:

The retriever automatically separates lexical fields from semantic fields, queries each group, and normalizes the results so that each group contributes equally to the final ranking, preventing lexical matches from drowning out semantic ones.

Cross-cluster search

The semantic field supports cross-cluster search (CCS), enabling use of the field in large, multi-cluster deployments. Simply list the indices to query using the standard <cluster>:<index> format:

The fields queried across indices and clusters can use a mix of different inference endpoints that produce different query embeddings. The search request will automatically apply the proper query embedding to each individual field queried.

Off the easel, into the world: optimising multimodal embeddings for production

When you move multimodal search from experiment to production, the size of your multimodal inputs becomes a practical concern. Multimodal data is supplied as base64-encoded data URLs, and that data is stored in the index. Those strings can grow large in a hurry: a single high-resolution file can balloon into several megabytes of encoded text, which has several side effects:

• The index size on disk can increase significantly.

• Requests and responses containing multimodal data are larger, increasing transmission time and ingress/egress costs.

• Inference on larger multimodal inputs is slower.

The good news is that you don’t need that much fidelity. Multimodal embedding models reduce each input to a compact representation before generating a vector anyway, so a smaller, lower-fidelity version of a multimodal input (such as a downscaled image or a lower-bitrate audio clip) produces a very similar embedding, and similar search quality, to its full-size original. This also applies to PDF input. PDFs are generally processed visually by multimodal models, so the quality only needs to be good enough to perform operations like image embedding and OCR. Long PDFs should be broken up into chunks of smaller inputs, so the embeddings generated more accurately represent each chunk. Feeding the model small inputs keeps your documents lean, trims index and response sizes, and speeds up ingestion, all without meaningfully affecting relevance. 

Elasticsearch reinforces this practice with a guardrail: the indices.inference.max_binary_input_size cluster setting caps the size of each binary input, defaulting to 1 MB. Any individual value that exceeds the limit is rejected with a clear error, so oversized inputs surface as an actionable problem at index time rather than as silent bloat. This setting is adjustable in self-hosted and ECH through the cluster settings API. It is not adjustable in our serverless offering, where 1 MB is the hard limit for binary sizes.

When possible, it is also advised to use source filtering to exclude semantic fields from responses. For example:

This makes responses smaller, more performant, and easier to parse because multimodal data is not returned with each.

Try out the semantic field

The semantic field is available in Elasticsearch 9.5 and Serverless. Start a free trial and try it out today.