Doc values were built to make aggregations, sorting, and grouping possible on a document engine, and they do that job well. Columnar Mode changes what the columns are for, and this post walks through the five properties that separate storing columns from being a columnar database.

Five properties separate a column store from a columnar database

Doc values were an optimization on top of _source

For most of the last decade, the original JSON document was the source of truth and the columns were a derived convenience. That ordering has consequences throughout the engine.

Because the engine could always fall back to _source, per-field storage was allowed to be lossy. Text fields had no doc values at all, since text could be reread from the stored document when needed. Even synthetic source, which reconstructs a document from its fields rather than storing a copy, sometimes reads from row-shaped structures to stay faithful to the JSON that arrived, with values that exceed ignore_above and fields that arrived unmapped going into stored fields.

The result is a clear contract; whatever JSON you send, you get back, and the columns accelerate everything else. For an engine whose job is to return your documents, that’s the right way round. Columnar Mode inverts it. Every field stores itself exactly once as doc values, doc values cannot be turned off, text fields get doc values, too, and the document is reconstructed from the columns when something asks for it.

How dictionary encoding handles high-cardinality data

Sorted doc values, the default for keyword fields, store a dictionary of distinct values plus one ordinal per document pointing into it. This is an excellent trade when values repeat. A host.name field drawn from a hundred machines, or a status code that’s almost always 200, compresses beautifully and groups quickly.

It works like the index cards in a warehouse. When 50 crates hold the same product, one card and 50 pointers beats writing the product name 50 times. When 50 crates hold the same product, you can store the product name in the index with the list of 50 crate IDs. When every crate holds something unique, you might as well just put the product name on the crates; the index will help you find what crate you want, but it won't save ink.

High-cardinality fields describe a lot of real data, including URLs and trace identifiers, along with message bodies. Columnar Mode, which skips the dictionary and compresses the values in blocks instead, uses binary doc values for high-cardinality strings. Which of the two a field gets isn’t something you configure. The engine decides per field, based on the values it sees, so each column is encoded for the data it actually holds instead of one default applied to every field. Pure columnar systems have long carried cardinality in their type system, but usually as something you declare, and you own the consequences when the data shifts underneath it. Here, it’s the engine's job.

Why every field builds an inverted index by default

By default, a keyword field also builds an inverted index and a numeric field also builds a BKD tree. That happens on every field because at write time the engine doesn’t know which capability you’ll want at read time, significantly increasing the footprint of each field. Those structures also have to be rebuilt during segment merges, which costs CPU exactly when ingest is heaviest.

Our time series engine (TSDB) is proof of what happens when you stop paying for capability that the workload doesn’t use. Replacing the indices on @timestamp and dimension fields with doc value skippers, which are sparse structures holding the minimum and maximum value for each block of documents, removed 10 bytes of the original 25 bytes per OpenTelemetry (OTel) data point. There was no measurable query regression on time range and dimension filters, and indexing CPU dropped by about 10%, as a bonus.

Columnar Mode generalizes that default. Fields aren’t indexed unless something needs them to be, with only text-mapped fields keeping their inverted index for fast free-text search.

Metadata fields like _id and _routing were row-shaped

The fields you never think about followed the same document-first design. The _id field was a stored field plus an inverted index. Custom _routing was a stored field. Sequence numbers were kept for optimistic concurrency control, regardless of whether a workload ever updated a document.

TSDB deals with all three. It synthesizes _id from the _tsid and @timestamp values that already identify a data point, using a segment-level bloom filter to catch duplicates, which removes 5 bytes per data point with no loss of functionality. It trims sequence numbers once replication no longer needs them, which removes 4 bytes. Add a codec block size increase from 128 to 512 elements for another 2 bytes, and those four changes contribute across versions 9.1 through 9.4 to the 21 bytes that took OTel metrics from 25 bytes per data point down to 3.75.

Columnar Mode makes those ideas general rather than metrics-specific. By general availability (GA), all metadata fields will store themselves as doc values, while we plan to follow up and add a sort id mode synthesizing the identifier from the index sort fields, in addition to derived fields that will generalize what _tsid does for time series to any set of fields.

Columnar query execution in the ES|QL compute engine

A column store only pays off if the engine reads it as columns. Aggregations inherited the document-at-a-time shape from search, which is the natural fit for an engine built around documents. Reading columns instead lets the engine hand a whole block of values to a single instruction, and that’s where the numbers below come from.

The ES|QL compute engine changed that shape, and TSDB again shows the size of the effect:

• Vectorized execution of time series aggregations was worth up to 8x on its own.

• Decoding on-disk data straight into the primitive arrays the engine aggregates over, with no intermediate copies, was worth roughly another 10x.

• Constant blocks turned repeated values into a form of in-memory run-length encoding.

• Filter pushdown moved filters down to Lucene, where skippers can discard whole blocks unopened.

Together with the rest of the block-level query work, query latency improved by up to 160x compared to earlier versions.

That work continues. Skipper-aware operators, aggregations that group on ordinals and convert to real values as late as possible, and richer per-block summaries are all in progress, and they benefit every index mode because every mode reads doc values underneath.

What Columnar Mode changes for logs and analytical data

Storing values in columns is a storage detail. A columnar database needs five things:

1. The columns are the only copy of the data.

2. Each column is encoded for the data it actually holds.

3. Metadata is columnar, too.

4. Fields add indices only when something needs them.

5. The query engine processes blocks of values rather than records or documents.

TSDB reached all five for metrics in Elasticsearch 9.4, which is why the numbers in this post come from metrics rather than from a slide. Columnar Mode applies the same treatment to logs and security telemetry, along with analytical data. It’s in technical preview in Elasticsearch 9.5, with GA targeted for 9.7.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

The changes are about storage and performance, along with the out-of-the-box (OOTB) experience. Up until 9.5.0, Elasticsearch operated as a document-based search engine by default. It could be set up to behave like a columnar system storage-wise, but that wasn’t the OOTB experience. Columnar index modes make a number of fundamental changes that allow Elasticsearch to optimize columnar analytic and search use cases:

• Fields are stored once as doc values only and are no longer indexed by default.

• New multi-value semantics. The original ordering of multiple values per field per document (for example, in arrays) is preserved by default.

• Mappings are always flat, and object and passthrough fields in mappings are always auto-flattened.

How columnar index modes fit into Elasticsearch

Many of the columnar index mode changes originate from time series data streams (TSDSs). As part of making TSDB a competitive metrics solution, we improved doc values format on disk and only store dimensions and metric fields once as doc values. We also improved query performance. TSDB is already columnar today. Essentially, this makes TSDB’s storage mode columnar. The lessons learned from TSDB are now being applied more broadly to Elasticsearch. 

Note that columnar index modes are opt-in and columnar, and document-based indices can coexist in the same cluster. An enterprise search use case can use a document-oriented index mode, while a logging use case can use logsdb_columnar index mode and be fully columnar, all in the same cluster. In fact, there are currently seven index modes, and indices can all use them in the same cluster.

How columnar storage stays fast without an inverted index

Indexed fields, either an inverted index for string-based fields or block k-dimensional (BKD) tree for numeric fields, allow Elasticsearch to query or filter by field very efficiently. However, the cost for this is an additional expensive data structure that uses a lot of disk space and is expensive to build at index time and at merge time. With the columnar index modes, fields are no longer indexed by default, so what did we do for query performance to be still acceptable on fields that were no longer indexed?

One major change was improving doc values scanning performance. This is key and is the cornerstone that any columnar system relies on. Previously, the scanning of doc values was essentially document by document. Lucene’s doc values API only allowed for looking up one value at a time. Historically, this fit the execution model of a search engine. In our own doc value format, we build the capability to allow bulk reading of values for Elasticsearch Query Language (ES|QL) queries. Also over recent minor Lucene releases, Lucene doc values API added support for bulk reading. Without this, fast columnar scanning wouldn’t have been possible.

Secondly, we fully adopted doc value skippers, a hierarchical skiplist over doc values. Contrary to an inverted index or a BKD tree, doc values skippers are lightweight data structures. At its core, a skipper allows queries to skip over a range of documents that don’t match a query. It can do this because it stores information, like min and max values. So, for example, when a range query is executed, an interval of documents can be skipped based on the intermediate result and a doc value skipper’s min and max values. The effectiveness of doc value skippers depends on the order in which documents are laid out on disk. This is why index sorting should be enabled or altered to match the use case.

By significantly improving our columnar scanning and doubling down on doc values skippers, we’re able to avoid indexing fields by default. Note that a field can still be indexed; the index mapping attribute just defaults to false in columnar mode. The exceptions to this rule are text-based fields, which are still indexed by default. This is because text fields provide free text search, which includes text analysis, along with phrase and wildcard matching. This is different from just filtering.

How Elasticsearch handles high and low cardinality fields

When setting up a schema with string fields, an important configuration parameter is often cardinality; that is, whether many unique values or a few unique string values are expected. Many systems have dedicated field or column types that target low and high cardinality string fields.

Fields that have low cardinality are typically stored with a dictionary, containing all unique values. Then, for each row offset, the offset into the dictionary containing the term the row has is stored. This is often called an ordinal. For low cardinality fields, this works well, as storing an ordinal per document takes up much less space. Encoding techniques, like delta encoding, offset encoding, and bitpacking, work well for ordinals to compact the per-document storage to just a few bits. 

However, for high cardinality fields, the dictionary and ordinal approach can work counterintuitively. If a larger percentage of the documents have a unique value, building the dictionary becomes expensive and storage savings diminish. The dictionary then becomes another level of indirection for reading values. This is why most systems in that case store values in a columnar fashion using block-based compression. For example, values of multiple rows are stored in 128KB blocks using a sliding-window dictionary-based compression algorithm (like zstandard or lz4). This, in general, is a simple and effective method to store higher cardinality fields and avoids building and maintaining a dictionary. 

With document-based Elasticsearch, there are two ways to map a string: using either the keyword field mapping or one of the text-based field mappings. The former stores an inverted index and dictionary-based doc values. The latter only stores an inverted index. This is why, typically, a text field mapper is often used in combination with a keyword mapper as a multi-field. Also, keyword field mapper (as the name suggests) is meant for keywords or fields that have a lower cardinality and uses dictionary-based doc values implementation. However, in practice, keyword field mappers are also used for high cardinality.

In columnar mode, every field is only stored once by default. For keyword fields, this means only doc values are stored with no inverted index. Text-based fields now also store doc values and an inverted index by default. Text-based field mappers are different from the keyword field mapper, as these are not automatically used via Elasticsearch’s dynamic mapping logic for columnar indices, and therefore text fields keep storing an inverted index by default.

For both keyword- and text-based fields, we didn’t choose to expose a cardinality mapping attribute. It’s not always possible to know ahead of time whether a field is low or high cardinality. When flushing and merging segments to disk, Elasticsearch sees all values and can determine the cardinality of a field. This is why we’re choosing to automatically determine whether the usage of a dictionary and ordinal-based encoding is beneficial over block-based compression using a simple cardinality threshold. If a field is below this threshold, dictionary and ordinal-based encoding is used; otherwise block-based compression is used. This simplifies configuration of the mappings and makes it possible to automatically optimize storage as data evolves, since some segments may use dictionaries while others may use blocks for the same field. However, this is currently not ready yet and so, as part of 9.5.0, in columnar mode, both keyword- and text-based field mappers store values in doc values in a block-based compressed layout on disk.

The two approaches compare as follows:

Dictionary and ordinal encoding

Block-based compression

Suits

Low cardinality fields

High cardinality fields

What’s stored

A dictionary of unique values, plus one ordinal per document

Values for many documents compressed together in blocks

Compression

Delta encoding, offset encoding, and bitpacking reduce each ordinal to a few bits

Sliding-window dictionary compression, such as zstandard or lz4, typically over 128KB blocks

Read path

Resolve the ordinal, and then look up the value in the dictionary

Decompress the block, and then read the value directly

Cost as cardinality rises

Dictionary grows large, savings shrink, and the extra indirection stays

Stable, with no dictionary to build or maintain

Used in columnar mode tech preview

Not yet

Yes, for both keyword and text fields

Columnar mapping attributes: multi_value, nullability, on_failure

The columnar index modes provide more control over how data is stored as doc values. By default, Elasticsearch is lenient and accepts all non-malformed values (for example, nulls and multiple values per field and document). If documents have fields with multiple values per document or no value, doc values store additional data structures to deal with them and therefore implicitly increase costs. 

With columnar, new mapping attributes provide additional control. Note that these new mapping attributes are currently only available with the columnar index modes but will eventually also be available for all index modes.

Three new mapping attributes are involved:

Attribute

Default

Enforces

On violation

Available

multi_value

true

One value per document per field

Document indexing fails

9.5.0

nullability

true

Field must have a value

Document indexing fails

9.5.0

on_failure

fail

How the above failures are handled

Sets fail or ignore behavior

Next minor release

multi_value: Enforcing single-valued fields

By default, Elasticsearch accepts multiple values per document. To understand the implications of this, we first should take a look at how Elasticsearch (using Lucene’s doc values) stores a dense numeric field where all documents have a single value:

With this layout, all values are stored in blocks. The number of values per block depends on the index mode but is typically 128 values and is always the same within an index. All values in a block are encoded using various encoding techniques, like delta encoding and bit packing, so each block can have a different size, depending on how well the encoding techniques compress the values. This is why a block index is required. 

Lucene has the notion of a docid (internal numbering for a document), which is essentially a row identifier.

1. Queries produce matching docids.

2. In case of a dense field, the block id can be resolved from the docid directly.

3. The offset of a block can be resolved from the block index.

4. Once that has been looked up, the target block gets decoded and all values are available.

5. Finally, from docid, the ordinal within the decoded values array can be resolved, which produces the final value.

Now let’s have a look at how the data layout changes when documents have multiple values per document:

To determine how many values belong to a single docid, an offset lookup is required.

In this case, a docid has one or more offsets. Each offset points to a block index. Values for a single document are adjacent but can stretch over blocks. In general, compaction of values works well in blocks because values are similar. However, multi-value fields can cause the compaction of values to be less efficient if the number of values per field and document is large and values aren’t similar. This and the additional storage of offsets result in multi-value fields typically having a higher storage footprint on disk.

If a field is truly single-valued, you may want to enforce this property. The multi_value mapping attribute makes that possible now. An example is a log level field. Logs typically have one log level (such as debug, info, or error). Enforcing that this field is single-valued in your mappings can help avoid accidentally using more storage than anticipated. A mapping snippet example that disallows the field log.level to have multiple values per document:

Note that even if a field allows multiple values, this doesn’t mean an offset lookup is stored. This only happens when a Lucene segment has at least one document with two or more values. The multi_value mapping attribute exists just for enforcement.

nullability: Requiring every document to have a value

By default, Elasticsearch accepts documents with fields that have no value or null value. Just as  multiple values per document require additional accounting, documents with no value require additional accounting to identify which of them have at least one value.

Doc values store a docid to offset lookup (known as IndexedDISI in Lucene) in case not all documents have a value in a segment. The offset either points directly to the block index for single-valued fields or the offset lookup in case of multi-valued fields. This lookup is compact compared to the value blocks being stored. However, if it were to be created for fields that should have at least one value per document, that would be a waste.

The nullability mapping attribute allows you to control whether documents are allowed to have no value. Just like the multi_value mapping attribute, the nullability attribute exists for enforcement.  Following is a mapping snippet example that requires the log.level field to have a value:

on_failure: What happens when validation fails

What happens if a document has multiple values for a field and if the multi_value mapping attribute is set to false or when a field is mapped with nullability set to false and a document doesn’t have that field? At the moment, indexing such documents will fail with a bad request error.

As part of the next minor release, the on_failure mapping attribute will be available. This allows you to indicate how to handle these validation failures, on a per-mapped field basis. This will support two values:

1. Fail: Fail indexing of the entire document with a client error. This is the current behavior in Elasticsearch 9.5.0.

2. Ignore: Ignore the validation error, mark the field as ignored, and store values for that field in a hidden field so that it can be introspected when requesting the source. 

Trying the columnar index modes

The columnar index modes are still under active development, but we encourage you to give them a test drive. As we prepare the columnar index modes for general availability (GA), we’ll add more performance and efficiency improvements. We believe that by adapting a columnar mindset, many use cases will benefit from being more cost effective or having better performance characteristics.

Columnar mode is Elasticsearch's analytics-optimized columnar storage mode, built for scan-heavy workloads, like log analytics. In this mode, keyword fields don't get an inverted index by default, so term and wildcard queries scan doc values. DocValuesSkippers (zone maps) already trim how much data a scan touches, but these rewrites cut the cost of what's left. 

How Lucene's query rewrite mechanism works

In Lucene, every query has the option to implement a rewrite method that returns another query. This method returns a query with the same semantics but a different implementation. The query engine repeatedly calls the rewrite method until the returned query doesn’t change. This final query is the one that’s actually evaluated. Importantly, the rewrite can see the actual query arguments and specialize the implementation based on these.

For example, in a query looking for documents where a string field contains the value "foo", the rewrite method knows that the term we’re searching for is "foo". In theory, rewrite could replace the general query class with something specific to "foo". For example, the original query class ScanningBinaryDocValuesTermQuery could be replaced with FooQuery. Now this rule probably wouldn't be helpful, but it gives a sense for the level of specialization that’s achievable with rewrite rules.

Rewrite rules and query optimization in database systems

It's worth placing rewrite rules in the larger context of database systems. Lucene and Elasticsearch aren’t the first systems to use transformation rules to optimize queries. Most (or maybe all) database systems use some kind of rule system during query optimization. The most influential rewrite rule system was in IBM's Starburst database. This system's core contribution was extensibility; for example, it was possible to add new data types and storage methods, along with (most importantly to us) optimizer rewrite rules.

Each rule consisted of two parts:

1. A condition function: A predicate determining whether the rule applies to the current query graph.

2. An action function: The transformation that rewrites the query plan into a more optimal form.

A rule engine applied matching rules until a stopping condition was met.

Though Lucene's rewrite method is superficially different from these condition and action functions, it achieves the same goal. It checks whether certain conditions match, and if they do, it applies the rewrite by returning a new query. If conditions don’t match, the rewrite returns this, replacing the query with itself; that is, choosing not to apply the rule.

Why these rules live in Lucene, not the ES|QL query optimizer

Elasticsearch actually contains a separate rewrite rule system within the Elasticsearch Query Language (ES|QL) optimizer. This operates on the high-level structure of a query; for example, doing predicate pushdown to avoid unnecessary computation on documents that will be filtered out. But it’s still useful to have the rule system within Lucene. Since Lucene acts as the storage layer for ES|QL (and classic _search) queries, it’s easier to express rewrites that take advantage of the physical data format in Lucene rather than in a higher-level optimizer.

A query rewrite rule for wildcard queries: Simpler code, no automaton

Wildcard queries support the ? and * operators to match any character once or any character multiple times. These operators can appear any number of times in a wildcard query. As with regexes, to evaluate whether a string matches a wildcard query, we build an automaton from the query string and then use the string bytes to do state transitions through the automaton. This is relatively fast, but if you have to evaluate it for every document, the latency really adds up.

But maybe we don't always have to run an automaton. Consider a query like *foo*. How would you implement this if you were writing a simple query engine to find matching strings in a list of strings? Pretty much every programming language has the tool you want built in: a method that finds a substring within a given string. This function doesn't need a complicated automaton; it probably just consists of a couple of for loops.

Now of course we couldn't use this function to implement an arbitrary wildcard query, but we don't have to. The rule rewrite system isn't for the general form. It's for implementing special cases, and it can see the specific query. It knows that we’re looking for *foo* and realizes that this specific case doesn't require the heavyweight automaton machinery. And it can do the same for any query that starts and ends with a *, with some term in the middle.

The following pseudo-code shows the pattern. At the top, we have the generic WildcardQuery. It has two notable fields: the query string (for example, *foo*) and the automaton built for that query. The matches method checks whether the field value for a given docId is a match by using it to evaluate the state transitions of the automaton. More interestingly, its rewrite method checks whether the query matches our special case. We show this with a regex that checks whether the query string starts with a *, has any non-*characters at least once, and then ends in a *. If so, we return the special case as a ContainsQuery and pass in the inner query string (since it doesn't care about the *s). The ContainsQuery then just does a simple contains check to see whether the term bytes are somewhere within the value bytes.

Benchmarking the wildcard rewrite on ClickBench Q20

The wildcard rewrite is straightforward, but does it actually work? Yes, we can use the ClickBench benchmark, which has several queries of this form. Query 20 (Q20) is FROM hits | WHERE URL LIKE "*google*" | STATS count = COUNT(*). It's exactly the query shape that this rule matches: a string match against the wildcard query *google*. And since the query is just counting, we can see exactly how well this technique works. It turns out to be quite effective. Q20 saw a 1.75x improvement on median latency of hot query times, with no filter cache. All benchmarks in this post were run on an Intel Core i9-13900H.

Adding SIMD to the substring search: 1.75x to 2.3x

But can we do better? Yes, switching to a simple contains check opens up a new possibility. Instead of using the two for loops, we can swap scalar logic for single instruction, multiple data (SIMD) logic. Elasticsearch uses the Panama vector API (see our post on SIMD in Elasticsearch), which lets us implement the contains check in SIMD. This works particularly well for longer strings that can take advantage of the wide SIMD registers; for strings under 24 characters, we still use the scalar approach. With this change, we saw another 1.32x improvement, resulting in a total speedup of 2.3x over the automaton-based approach.

A query rewrite rule for empty strings: Less data, no decompression

One benefit of Lucene-based rules is that they’re low level and can fit to the data format. That’s the case for this rule, which applies to string data. 

How columnar storage encodes string data

In Elasticsearch's standard mode, string values are stored by document; this is a row-major format. But in columnar mode, unsurprisingly, the data is stored in columnar-major format. A column of string data is stored in chunks. Each chunk contains many string values and consists of an array of integer offsets and a (Zstd-compressed) blob of the strings' bytes. For a string at index i, offsets[i] points to the offset in the decompressed byte blob where the string starts. So the length of string i can be computed from offset[i+1]-offsets[i]. (There's a dummy extra offset at the end, so we can easily compute the length of the last string). The following diagram shows how a chunk with the strings ‘Feta’, ‘Asiago’, ‘’, ‘Stilton’, and ‘Brie’ is encoded.

Why a term query has to decompress the chunk

Now that we understand the columnar format, let's get back to query optimization. First, consider a term query for the query foo. We’re looking for documents where a given string field exactly matches the string foo. So how do we implement this on a string column in the above format? The algorithm is straightforward:

The bottleneck is the Zstd decompression step. But there's not much we can do about that; if we want to check the bytes, we have to decompress the chunks. But remember, we aren't trying to optimize the general case, we’re looking for special cases. (In reality, you don't just try to think up special cases. These optimizations came about by first running a useful query, realizing that it could be faster, and then looking for ways to improve it.)

Rewriting the empty string query as a length check

One special case we found that’s worth improving is a query for the term "". Admittedly, it's a silly term, but empty strings are all over the place. Since they're rarely useful, we usually filter them out with a query like term != "". Thankfully, this is a query we can optimize.

Consider the above algorithm for the empty string term. The line if value == term is a bit weird; we’re asking Does this value equal the empty string? We can do that, but there are no bytes to compare, so the check unwinds:

1. We only need to know whether the value has length 0.

2. If we only need the length, we don't need to look up the value in the decompressed chunk.

3. If we never look up a value, we don't need any bytes from the chunk at all.

4. If we need no bytes from the chunk, we don't need to decompress it.

All we need are the lengths, and those live in the offsets array. It's compressed, too, but with cheap integer compression rather than Zstd, which is much faster.

With this realization, we can rewrite empty string term queries. The one new operation we need is docValues.loadLength(docId), which reads directly from the offset array without touching the compressed bytes. After the previous example, this should look familiar. The most interesting part is TermEqualsQuery.rewrite; it finds the empty string special case and replaces the query with the simpler version that only checks the length.

Benchmarking the empty string rewrite: 1.6x faster

Now let's see how this stacks up. There aren't any pure-scan ClickBench queries that use this rule as directly as Q20 does for the previous rule, so we'll make our own. Consider the query: FROM hits | WHERE SearchPhrase != '' | STATS count(*). On this query, we see a 1.6x speedup, which is a great improvement for a fairly uncomplicated change. Better yet, ES|QL can take advantage of loadLength directly. Any time that ES|QL accesses a string's BYTE_LENGTH, without needing the string itself, the request uses this same specialized length loading to avoid unnecessary decompression.

What makes a good query rewrite rule

The two rules covered here follow the same shape: identify that a query is a special case, and then swap it for a cheaper implementation. But they reduce cost in different ways. 

Wildcard rule

Empty string rule

Query shape detected

*term*

field == ""

Replaced with

SIMD substring search

Length check on the offsets array

Cost reduced

Algorithmic work

Data access

Speedup

2.3x

1.6x

The underlying pattern is worth noting: finding a query that leaves performance on the table, finding a special case that can be optimized, and swapping in a cheaper implementation. The hard parts are finding queries that uncover these opportunities for optimization and then identifying the special cases. The actual fix is often relatively straightforward, as both rules here show. Our work on columnar mode has provided many opportunities to run interesting queries and hunt down exactly these kinds of wins.

That's also why extensibility in a rule system is so important. These rules can't be built into a database from the start; they're found through an incremental discovery process. Lucene's rewrite system makes that practical. As columnar mode grows to handle new workloads, rules like these will keep emerging.

To try columnar mode and the optimizations described in this article, use Elastic Cloud Serverless or Elasticsearch 9.5 or later, where columnar mode is available as a technical preview.

Why dynamic mapping breaks down with schemaless data

Elasticsearch wants a schema. Every field you index has a mapping that defines its type, its analyzer (for text), and how it is stored. That schema makes storage compression efficient, with fast search and cheap aggregation. It becomes a liability the moment your data stops looking like a database table.

Consider the shapes that show up in real systems:

• Log events where every service adds its own attributes. One emits labels.region, another labels.k8s.pod, and another labels.tenant_id.

• OpenTelemetry resource attributes, where the set of keys is defined by whatever agent happened to send the span.

• User-supplied metadata bags, feature flags, or tagging systems, where the keys are open-ended by design.

If you map each key as its own field, the mapping grows unbounded. This is the classic "mapping explosion." Thousands of dynamically created fields inflate the cluster state, slowing down every mapping update and eventually hitting the field limit. Each field also carries overhead in the index. You pay a structural cost for keys you didn’t plan for and may query only once.

The naive escape hatch is to store the whole object as a string and give up on querying its contents. That trades one problem for another: you keep the data but lose the ability to filter or group by anything inside it.

The flattened field type is the better path. You can index an entire JSON object under a single mapped field, keeping the keys queryable with almost none of the mapping-explosion cost. 

This post covers how a flattened field stores the data on a disk and how ES|QL reads it back.

How flattened fields index dynamic JSON keys under one mapping

Map one field as flattened:

Then write arbitrary nested JSON into it:

There is exactly one field in the mapping, labels, no matter how many keys appear across your documents. The cluster state doesn’t grow when a new key shows up. The subkeys remain individually searchable. You can reference labels.region or labels.k8s.pod in queries, even though neither was ever declared.

The catch and central tradeoff is that every leaf value is a keyword. The number 4 above is indexed as the string "4". There is no numeric typing, no date parsing, and no range math on dynamic keys. Flattened fields exchange per-field richness for schema flexibility. That fact explains almost every design decision that follows.

How Elasticsearch stores flattened field JSON keys on disk

There are two types of queries on flattened fields: an unkeyed query on the root flattened field, and a keyed query on a specific subfield. Following the previous example, a query of the form labels: "us-east-1" matches a value under any key, while the querylabels.region: "us-east-1" matches a value under the specific regionkey.

To support these two distinct query formats, the flattened mapper writes each leaf value into two distinct Lucene fields.

Take this document:

The mapper produces:

• A root field under labels, holding the bare values:

• A keyed field under labels._keyed, holding the flattened key concatenated with its value:

In the keyed field, nested objects are dot-flattened into a single key (k8s.pod), and the key is joined to its value with a reserved NULL byte (\0) as the separator. Keys that contain a NULL byte are rejected at parse time, so the separator is always unambiguous. To find the value, you split on the first NULL.

These two fields make both query shapes work:

• labels: "us-east-1" matches a value under any key, so it searches the root field.

• labels.region: "us-east-1" matches a value under a specific key. It rewrites the query to the term region\0us-east-1 and searches the keyed field.

Query restrictions on flattened field subkeys

Because every key's terms live in one sorted list, the keyed field cannot answer every query shape a plain keyword field can. Three restrictions follow:

1. No fuzzy, regexp, or wildcard on a specific subkey. Nothing about the layout makes them impossible,  but the pattern would have to be combined with the key prefix so it can’t walk past the key boundary. The mapper doesn’t do that at this writing.

2. Any range query on a subkey needs at least one bound. Elasticsearch already has a query for "this field has some value here, whatever it is": the exists query. On a flattened subkey it runs as a prefix query on key\0, which sweeps every term belonging to that key. A range with neither bound would sweep exactly the same terms. Rather than support two spellings of one scan, the mapper rejects the boundless range and asks for the exists query.

3. A range query on a subkey needs the field to be indexed. A flattened field can be mapped with index: false, which skips the inverted index and keeps only doc values, the columnar per-document storage covered in the next section. Exact-match queries survive that. With no terms to look up, Elasticsearch scans the doc values column instead, which is slower but gives the same answer. Range queries have no equivalent fallback, so a range on a subkey of an unindexed flattened field throws an Exception.

All three are limits on the Lucene query the mapper is willing to build, and where you notice them depends on how you query.

On the search API, where you name labels.region directly, they come back as errors.

In ES|QL you will not see them as errors at all. There, the same limits decide only whether a predicate is pushed into Lucene or runs in the compute engine on the extracted column. 

This is pushed to a term query on the keyed field:

This is not, so the filter runs per row on the extracted keyword:

Same answer, more work. That distinction is the subject of the second half of this post.

Under the hood: how range queries stay inside key boundaries

This part is internal. You don’t need it to use the field, but it explains where the bounds rule comes from.

For a handful of documents in one segment, the shared term list looks like this:

Each key owns a contiguous slice of that list. For example, all values for key "region" are clustered together in an ordered sublist. A range with both bounds set encodes each bound the same way a term is encoded, so a lower bound of "us-east" on the region key becomes region\0us-east and an upper bound of "us-west" becomes region\0us-west. Both endpoints already carry the key prefix, so the scan can’t leave the region slice. Nothing special is needed.

The half-open case is problematic. Handing Lucene a lower bound of region\0us-east with no upper bound would scan to the end of the term list, straight through tenant_id\0acme and every other key that sorts after region. So the mapper substitutes a sentinel for the missing side:

• A missing lower bound becomes key\0, inclusive. That’s the encoding of the empty value, and it’s the first term in the key's slice.

• A missing upper bound becomes key\1, exclusive. Byte 0x01 is the next byte after the 0x00 separator, so it sorts after every key\0value term and before the first term of any other key.

A one-sided range is therefore boxed into [key\0, key\1), which makes it exactly as safe as a closed one.

The upper sentinel also covers the case where one key is a prefix of another. If an index holds both region and regionx, then region\1 still sorts below regionx\0eu-west-1, because 0x01 is smaller than the x that follows the shared region prefix. A range on region cannot leak into regionx.

How flattened fields use the inverted index and doc values

Each leaf value can be written into two Lucene structures. Both are enabled by default, but can be disabled by the mapping configuration.

• The inverted index (when the field is indexed). Two untokenized keyword terms are indexed per value, one on the root path and one on the keyed path. This powers term, prefix, and range searches.

• Doc values (when doc_values is enabled). A columnar, document-ordered structure. This powers sorting, aggregations, and ES|QL reads.

The inverted index answers the question: "Which documents contain this term?" 

Doc values answer another question: "For this document, what are the values?" Doc values are laid out column by column so a scan touches only the bytes it needs. A flattened field uses both inverted indexes and doc values, so it can serve search and analytics from the same field.

The nature of the index means that the root field is only present in some cases. When the inverted index is disabled by the mapping, any value search requires a linear scan of the doc values for the searched value. In this case, when performing a search on the root field, there isn’t much additional overhead compared to just scanning the keyed field and ignoring the key markers. So the flattened mapper skips writing the root field, relying on the keyed field for both root and keyed queries.

Why flattened fields switched from dictionary to binary doc values

Historically, flattened field doc values used Lucene's SortedSetDocValues, which is a dictionary-compressed format. This means that every key\0value value indexed across all documents per segment is stored in one big sorted, deduplicated set of values. Each document tracks a list of ordinals into that value set.

This dictionary approach provides great compression for low-cardinality fields that tend to have repeated values. It is byte-efficient to store a value only once, and then refer to it by a single integer value. However, there is overhead associated with building and maintaining that dictionary of values, and that overhead is wasted effort when operating on high-cardinality fields that don’t repeat values.

Because flattened fields are a catch-all type, their cardinality tends to be very high. So while the dictionary approach works, it’s not the most compact or scan-friendly layout for this data, especially in time-series indices where flattened bags are common and storage pressure is real.

Recent versions switched the storage to use Lucene’s BinaryDocValues. This format just stores a literal binary blob for each document, which is compressed using Zstandard by our doc values codec when written to disk.

This new binary format provides an additional benefit: it allows us to maintain original array ordering without any overhead. Flattened fields support the mapping parameter preserve_leaf_arrays, which affects how multivalued fields are returned when using synthetic source. When configured to preserve_leaf_arrays: exact, returned values preserve the order, duplicates, and nulls from the original source value.

The dictionary encoding inherent to sorted-set doc values means the returned values are sorted, deduplicated, and de-nulled. To implement preserve_leaf_arrays, flattened fields have traditionally used an additional sidecar field, tracking the required metadata to reconstruct the original source value. However, the nature of binary doc values means that this sidecar field is no longer needed. The values are just stored and returned as indexed.

Limitations of schema on read with flattened fields

The limitations below follow from the keyword-only rule and the shared keyed field:

• No numeric, date, or Boolean typing on dynamic keys. 100 and "100" are the same term.

• No fuzzy, regexp, or wildcard on a specific subkey.

• No multi-fields (fields) or copy_to on the flattened field.

• A depth_limit (default 20) on how deeply nested the object can be.

If you need real typing for a known key, flattened now supports explicitly mapped subfields: you declare individual keys with real types under a properties block, and those keys are indexed by their own typed mapper instead of the keyed field. You get numeric ranges on labels.status_code, while everything else in labels stays dynamic and keyword-only. This is the escape hatch for the handful of keys you actually know about.

Schema on read: querying flattened field JSON keys in ES|QL

Search has supported flattened fields for years. ES|QL, the newer, piped query language built on a columnar compute engine, now reads flattened fields, too. Support began in the Technical Preview of Elasticsearch 9.5.0. It has two distinct pieces.

What ES|QL returns when you select a flattened field

ES|QL uses a dedicated data type for flattened fields, rather than folding it into keyword. When you select the root, you get the whole object back as a JSON string:

Keys come back sorted. You can carry this value through a query, count it, group by it, and run the multi-value and comparison functions on it. But an opaque JSON blob is not usually what you want to filter or aggregate on. For that, you need to reach inside it and process its contents.

How FIELD_EXTRACT reads JSON keys from flattened fields

There is no dotted-path syntax for dynamic keys in ES|QL. You cannot write labels.region for an unmapped key, because to the engine the flattened root is a single leaf value, not a set of columns. Instead you use a function:

The absence of a dotted-path syntax is a tentative limitation. The keyed field already addresses individual subkeys, so a more natural syntax for reaching into a flattened root is something we are planning to support.

FIELD_EXTRACT(field, path) takes a flattened field and a key, and returns a keyword. The rules are easier to see against a document. Take this one:

And this query:

The result is:

Four things to take from this:

• The dot is part of the key, not a navigation operator. The mapper already collapsed the nested object into the flat key k8s.pod, so "k8s.pod is a direct lookup, not a walk from k8s to pod.

• Matching is exact. "k8s" returns null because there is no leaf stored at k8s, only at k8s.pod and k8s.node. For the same reason, "host" will not find "host.name", and matching is case-sensitive, so "Region" will not find "region".

• Arrays come back multi-valued. "tags" yields a multi-valued keyword you can MV_EXPAND, count, or filter on. A missing key yields null.

• Everything is a keyword. "retries" comes back as the string "4", and a Boolean leaf comes back as "true" or "false".

JSONPath syntax is rejected outright, at parse time rather than per row. Both FIELD_EXTRACT(labels, "['k8s.pod']") and FIELD_EXTRACT(labels, "tags[0]") fail with field_extract path must be a literal flattened sub-field name.

Once extracted, the value is an ordinary keyword column. You can filter on it, group by it, sort by it, or use it as the join key in a LOOKUP JOIN:

How ES|QL pushes flattened field predicates into the columnar store

The obvious way to implement FIELD_EXTRACT would be to read the whole flattened root out of storage, render it as a JSON string, hand it to the compute engine, and parse it once per row to pull out a single key. But that means reading every key to use once and paying for a JSON parse on every document. So this obvious implementation is not performant.

ES|QL avoids this whenever possible. Between storage and the compute engine sits the block loader, the step that turns stored data into the columnar blocks the engine operates on. FIELD_EXTRACT hooks into that step instead of running after it.

When ES|QL loads a column, the flattened field type inspects the request. If the request is an extraction of a single constant key and the field has doc values, it routes straight to the keyed doc-values loader, which reads the key\0value entries for only that key out of the columnar structure. The column that arrives at the compute engine already contains only that key's values. The JSON string is never built and never parsed, and the other keys in the object are never read.

When extraction can’t be fused into the block loader, for example, because the key is computed per row or the root is the output of another function such as CASE, ES|QL falls back to the parse-per-row path. The results are identical either way. Only the cost changes.

The comparison can push down further. A predicate like FIELD_EXTRACT(labels, "region") == "us-east-1" can be pushed to Lucene as a term query against the synthetic keyed field, the same region\0us-east-1 term the search path uses. So a filter on an extracted subkey can be answered by the inverted index (if available), and the projection can be answered by doc values, exactly like a first-class field, even though the key was never in the mapping.

Ordering comparisons push down, too. The four, single-sided comparators (>, >=, <, <=) and closed BETWEEN-style ranges all become a range query on that same synthetic keyed field. This is where the key\0 / key\1 sentinels earn their keep: the single-sided forms are only pushable because the mapper can box an open bound inside the key. The pushed range is treated as a candidate, and the predicate is re-evaluated on the extracted keyword column afterwards, so multi-valued keys don’t slip through.

The values are keywords, so the ordering is lexicographic, not numeric. FIELD_EXTRACT(labels, "retries") > "10" compares strings, which means "9" is greater than "10". If you need numeric ranges on a key, map it explicitly under properties, or cast the value in ESQL.

Explicitly mapped subfields behave differently on purpose. Because they carry real types, comparison semantics diverge from the keyword path, so they are loaded and compared through their own typed mapper rather than fused into the keyed loader. And when you select a flattened root that has mapped subfields, ES|QL loads it from _source, so every leaf renders as a string and no keys are dropped silently.

When to use flattened fields vs. dynamic mapping in Elasticsearch

Use flattened fields when:

• The set of keys is open-ended or unknown ahead of time.

• You would otherwise cause a mapping explosion.

• Keyword-level filtering and grouping on the values is enough, and you don’t need numeric or date semantics on the dynamic keys.

• You have a few keys that do need real types. Map those explicitly under properties, and let the rest stay dynamic.

Avoid it, or map fields normally, when the schema is stable and you need full-text analysis, numeric aggregation, or date math across the board.

Remember that flattened is not a dumping ground for JSON you have given up on. It’s a real columnar-and-inverted store for schemaless data, and with ES|QL support, it’s now a first-class analytical citizen. You can keep the messy, unmapped parts of your data messy, and still filter, group, join, and aggregate across them as needed.

Vector search is revolutionizing the way we conduct similarity searches, particularly in fields like AI and machine learning. With the increasing adoption of vector embedding models, the ability to efficiently search through millions of high-dimension vectors becomes critical.

When it comes to powering vector databases, Elastic and OpenSearch have taken notably different approaches. Elastic has invested heavily in optimizing Apache Lucene together with Elasticsearch to elevate them as the top-tier choice for vector search applications. In contrast, OpenSearch has broadened its focus, integrating other vector search implementations and exploring beyond Lucene's scope. Our focus on Lucene is strategic, enabling us to provide highly integrated support in our version of Elasticsearch, resulting in an enhanced feature set where each component complements and amplifies the capabilities of the other.

This blog presents a detailed comparison between Elasticsearch 8.14 and OpenSearch 2.14 accounting for different configurations and vector engines. In this performance analysis, Elasticsearch proved to be the superior platform for vector search operations, and upcoming features will widen the differences even more significantly. When pitted against OpenSearch, it excelled in every benchmark track — offering 2x to 12x faster performance on average. This was across scenarios using varying vector amounts and dimensions including so_vector (2M vectors, 768D), openai_vector (2.5M vectors, 1536D), and dense_vector (10M vectors, 96D), all available in this repository alongside the Terraform scripts for provisioning all the required infrastructure on Google Cloud and Kubernetes manifests for running the tests.

The results detailed in this blog complement the results from a previously published and third-party validated study that shows Elasticsearch is 40%–140% faster than OpenSearch for the most common search analytics operations: Text Querying, Sort, Range, Date Histogram and Terms filtering. Now we can add another differentiator: Vector Search.

Up to 12x faster out-of-the-box

Our focused benchmarks across the four vector data sets involved both Approximate KNN and Exact KNN searches, considering different sizes, dimensions and configurations, totaling 40.189.820 uncached search requests. The results: Elasticsearch is up to 12x faster than OpenSearch for vector search and therefore requires fewer computational resources.

Figure 1: Grouped tasks for ANN and Exact KNN across different combinations in Elasticsearch and OpenSearch.

The groups like knn-10-100 means KNN search with and . In HNSW vector search, determines the number of nearest neighbors to retrieve for a query vector. It specifies how many similar vectors to find as a result. sets the number of candidate vectors to retrieve at each segment. More candidates can enhance accuracy but require greater computational resources.

We also tested with different quantization techniques and leveraged engine-specific optimizations, the detailed results for each track, task and vector engine are available below.

Exact KNN and Approximate KNN

When dealing with varying data sets and use cases, the right approach for vector search will differ. In this blog all tasks stated as knn-* like knn-10-100 use Approximate KNN and script-score-* refer to Exact KNN, but what is the difference between them, and why are they important?

In essence, if you're handling more substantial data sets, the preferred method is the Approximate K-Nearest Neighbor (ANN) due to its superior scalability. For more modest data sets that may require a filtration process, Exact KNN method is ideal.

Exact KNN uses a brute-force method, calculating the distance between one vector and every other vector in the data set. It then ranks these distances to find the nearest neighbors. While this method ensures an exact match, it suffers from scalability challenges for large, high-dimensional data sets. However, there are many cases in which Exact KNN is needed:

• Rescoring: In scenarios involving lexical or semantic searches followed by vector-based rescoring, Exact KNN is essential. For example, in a product search engine, initial search results can be filtered based on textual queries (e.g., keywords, categories), and then vectors associated with the filtered items are used for a more accurate similarity assessment.

• Personalization: When dealing with a large number of users, each represented by a relatively small number (like 1 million) of distinct vectors, sorting the index by user-specific metadata (e.g., user_id) and brute-force scoring with vectors becomes efficient. This approach allows for personalized recommendations or content delivery based on precise vector comparisons tailored to individual user preferences.

Exact KNN therefore ensures that the final ranking and recommendations based on vector similarity are precise and tailored to user preferences.

Approximate KNN (or ANN) on the other hand employs methods to make data searching faster and more efficient than Exact KNN, especially in large, high-dimensional data sets. Instead of a brute-force approach, which measures the exact nearest distance between a query and all points leading to computation and scaling challenges, ANN uses certain techniques to efficiently restructure the indexes and dimensions of searchable vectors in the data set. While this may cause a slight inaccuracy, it significantly boosts the speed of the search process, making it an effective alternative for dealing with large data sets.

In this blog all tasks stated as knn-* like knn-10-100 use Approximate KNN and script-score-* refer to Exact KNN.

Testing methodology

While Elasticsearch and OpenSearch are similar in terms of API for BM25 search operations, since the latter is a fork of the former, it is not the case for Vector Search, which was introduced after the fork. OpenSearch took a different approach than Elasticsearch when it comes to algorithms, by introducing two other engines — nmslib and faiss — apart from lucene, each with their specific configurations and limitations (e.g., nmslib in OpenSearch does not allow for filters, an essential feature for many use cases).

All three engines use the Hierarchical Navigable Small World (HNSW) algorithm, which is efficient for approximate nearest neighbor search, and especially powerful when dealing with high-dimensional data. It's important to note that faiss also supports a second algorithm, ivf, but since it requires pre-training on the data set, we are going to focus solely on HNSW. The core idea of HNSW is to organize the data into multiple layers of connected graphs, with each layer representing a different granularity of the data set. The search begins at the top layer with the coarsest view and progresses down to finer and finer layers until reaching the base level.

Both search engines were tested under identical conditions in a controlled environment to ensure fair testing grounds. The method applied is similar to this previously published performance comparison, with dedicated node pools for Elasticsearch, OpenSearch, and Rally. The terraform script is available (alongside all sources) to provision a Kubernetes cluster with:

• 1 Node pool for Elasticsearch with 3 e2-standard-32 machines (128GB RAM and 32 CPUs)

• 1 Node pool for OpenSearch with 3 e2-standard-32 machines (128GB RAM and 32 CPUs)

• 1 Node pool for Rally with 2 t2a-standard-16 machines (64GB RAM and 16 CPUs)

Each "track" (or test) ran for 10 times for each configuration, which included different engines, different configurations and different vector types. The tracks have tasks that repeat between 1000 and 10000 times, depending on the track. If one of the tasks in a track failed for instance due to a network timeout, then all tasks were discarded, so all results represent tracks that started and finished without problems. All test results are statistically validated, ensuring that improvements aren’t coincidental.

Detailed findings

Why compare using the 99th percentile and not the average latency? Consider a hypothetical example of average house prices in a certain neighborhood. The average price may indicate an expensive area, but on closer inspection, it may turn out that most homes are valued much lower, with only a few luxury properties inflating the average figure. This illustrates how the average price can fail to accurately represent the full spectrum of house values in the area. This is akin to examining response times, where the average can conceal critical issues.

Tasks

• Approximate KNN with k:10 n:50

• Approximate KNN with k:10 n:100

• Approximate KNN with k:100 n:1000

• Approximate KNN with k:10 n:50 and keyword filters

• Approximate KNN with k:10 n:100 and keyword filters

• Approximate KNN with k:100 n:1000 and keyword filters

• Approximate KNN with k:10 n:100 in conjunction with indexing

• Exact KNN (script score)

Vector engines

• lucene in Elasticsearch and OpenSearch, both on version 9.10

• faiss in OpenSearch

• nmslib in OpenSearch

Vector types

• hnsw in Elasticsearch and OpenSearch

• int8_hnsw in Elasticsearch (HNSW with automatic 8 bit quantization: link)

• sq_fp16 hnsw in OpenSearch (HNSW with automatic 16 bit quantization: link)

Out-of-the-box and Concurrent Segment Search

As you probably know, Lucene is a highly performant text search engine library written in Java that serves as the backbone for many search platforms like Elasticsearch, OpenSearch, and Solr. At its core, Lucene organizes data into segments, which are essentially self-contained indices that allow Lucene to execute searches more efficiently. So when you issue a search to any Lucene-based search engine, your search will end up being executed in those segments, either sequentially or in parallel.

OpenSearch introduced concurrent segment search as an optional flag, and does not use it by default, you must enable it using a special index setting index.search.concurrent_segment_search.enabled as detailed here, with some limitations.

Elasticsearch on the other hand searches on segments concurrently out-of-the-box, therefore the comparisons we make in this blog will take into consideration, on top of the different vector engines and vector types, also the different configurations:

• Elasticsearch ootb: Elasticsearch out-of-the-box, with concurrent segment search;

• OpenSearch ootb: without concurrent segment search enabled;

• OpenSearch css: with concurrent segment search enabled

Now, let’s dive into some detailed results for each vector data set tested:

2.5 million vectors, 1536 dimensions (openai_vector)

Starting with the simplest track, but also the largest in terms of dimensions, openai_vector - which uses the NQ data set enriched with embeddings generated using OpenAI's text-embedding-ada-002 model. It is the simplest since it tests only Approximate KNN and has only 5 tasks. It tests in standalone (without indexing) as well as alongside indexing, and using a single client and 8 simultaneous clients.

Tasks

• standalone-search-knn-10-100-multiple-clients: searching on 2.5 million vectors with 8 clients simultaneously, k: 10 and n:100

• standalone-search-knn-100-1000-multiple-clients: searching on 2.5 million vectors with 8 clients simultaneously, k: 100 and n:1000

• standalone-search-knn-10-100-single-client: searching on 2.5 million vectors with a single client, k: 10 and n:100

• standalone-search-knn-100-1000-single-client: searching on 2.5 million vectors with a single client, k: 100 and n:1000

• parallel-documents-indexing-search-knn-10-100: searching on 2.5 million vectors while also indexing additional 100000 documents, k:10 and n:100

The averaged p99 performance is outlined below:

Here we observed that Elasticsearch is between 3x-8x faster than OpenSearch when performing vector search alongside indexing (i.e. read+write) with :10 and :100 and 2x-3x faster without indexing for the same k and n. For :100 and :1000 (standalone-search-knn-100-1000-single-client and standalone-search-knn-100-1000-multiple-clients Elasticsearch is 2x to 7x faster than OpenSearch, on average.

The detailed results show the exact cases and vector engines compared:

Recall

knn-recall-10-100

knn-recall-100-1000

Elasticsearch-8.14.0@lucene-hnsw

0.969485

0.995138

Elasticsearch-8.14.0@lucene-int8_hnsw

0.781445

0.784817

OpenSearch-2.14.0@lucene-hnsw

0.96519

0.995422

OpenSearch-2.14.0@faiss

0.984154

0.98049

OpenSearch-2.14.0@faiss-sq_fp16

0.980012

0.97721

OpenSearch-2.14.0@nmslib

0.982532

0.99832

10 million vectors, 96 dimensions (dense_vector)

In dense_vector with 10M vectors and 96 dimensions. It is based on the Yandex DEEP1B image data set. The data set is created from the first 10 million vectors of the "sample data" file called learn.350M.fbin. The search operations use vectors from the "query data" file query.public.10K.fbin.

Both Elasticsearch and OpenSearch perform very well on this data set, especially after a force merge which is usually done on read-only indices and it’s similar to defragmenting the index to have a single "table" to search on.

Tasks

Each task warms up for 100 requests and then 1000 requests are measured

• knn-search-10-100: searching on 10 million vectors, k: 10 and n:100

• knn-search-100-1000: searching on 10 million vectors, k: 100 and n:1000

• knn-search-10-100-force-merge: searching on 10 million vectors after a force merge, k: 10 and n:100

• knn-search-100-1000-force-merge: searching on 10 million vectors after a force merge, k: 100 and n:1000

• knn-search-100-1000-concurrent-with-indexing: searching on 10 million vectors while also updating 5% of the data set, k: 100 and n:1000

• script-score-query: Exact KNN search of 2000 specific vectors.

Both Elasticsearch and OpenSearch performed well for Approximate KNN. When the index is merged (i.e. has just a single segment) in knn-search-100-1000-force-merge and knn-search-10-100-force-merge, OpenSearch performs better than the others when using nmslib and faiss, even though they are all around 15ms and all very close.

However, when the index has multiple segments (a typical situation where an index receives updates to its documents) in knn-search-10-100 and knn-search-100-1000, Elasticsearch keeps the latency in about ~7ms and ~16ms, while all other OpenSearch engines are slower.

Also when the index is being searched and written to at the same time (knn-search-100-1000-concurrent-with-indexing), Elasticsearch maintains the latency below 15ms (at 13.8ms), being almost 4x faster than OpenSearch out-of-the-box (49.3ms) and still faster when concurrent segment search is enabled (17.9ms), but too close to be significative.

As for Exact KNN, the difference is much larger: Elasticsearch is 6x faster than OpenSearch (~260ms vs ~1600ms).

Recall

knn-recall-10-100

knn-recall-100-1000

Elasticsearch-8.14.0@lucene-hnsw

0.969843

0.996577

Elasticsearch-8.14.0@lucene-int8_hnsw

0.775458

0.840254

OpenSearch-2.14.0@lucene-hnsw

0.971333

0.996747

OpenSearch-2.14.0@faiss

0.9704

0.914755

OpenSearch-2.14.0@faiss-sq_fp16

0.968025

0.913862

OpenSearch-2.14.0@nmslib

0.9674

0.910303

2 million vectors, 768 dimensions (so_vector)

This track, so_vector, is derived from a dump of StackOverflow posts downloaded on April, 21st 2022. It only contains question documents — all documents representing answers have been removed. Each question title was encoded into a vector using the sentence transformer model multi-qa-mpnet-base-cos-v1. This data set contains the first 2 million questions.

Unlike the previous track, each document here contains other fields besides vectors to support testing features like Approximate KNN with filtering and hybrid search. nmslib for OpenSearch is notably absent in this test since it does not support filters.

Tasks

Each task warms up for 100 requests and then 100 requests are measured. Note the tasks were grouped for sake of simplicity, since the test contains 16 search types * 2 different k values * 3 different n values.

• knn-10-50: searching on 2 million vectors without filters, k:10 and n:50

• knn-10-50-filtered: searching on 2 million vectors with filters, k:10 and n:50

• knn-10-50-after-force-merge: searching on 2 million vectors with filters and after a force merge, k:10 and n:50

• knn-10-100: searching on 2 million vectors without filters, k:10 and n:100

• knn-10-100-filtered: searching on 2 million vectors with filters, k:10 and n:100

• knn-10-100-after-force-merge: searching on 2 million vectors with filters and after a force merge, k:10 and n:100

• knn-100-1000: searching on 2 million vectors without filters, k:100 and n:1000

• knn-100-1000-filtered: searching on 2 million vectors with filters, k:100 and n:1000

• knn-100-1000-after-force-merge: searching on 2 million vectors with filters and after a force merge, k:100 and n:1000

• exact-knn: Exact KNN search with and without filters.

Elasticsearch is consistently faster than OpenSearch out-of-the-box on this test, only in two cases OpenSearch is faster, and not by much (knn-10-100 and knn-100-1000). Tasks involving knn-10-50, knn-10-100 and knn-100-1000 in combination with filters show a difference of up to 7x (112ms vs 803ms).

The performance of both solutions seems to even out after a "force merge", understandably, as evidenced by knn-10-50-after-force-merge, knn-10-100-after-force-merge and knn-100-1000-after-force-merge. On those tasks faiss is faster.

The performance for Exact KNN once again is very different, Elasticsearch being 13 times faster than OpenSearch this time (~385ms vs ~5262ms).

Recall

knn-recall-10-100

knn-recall-100-1000

knn-recall-10-50

Elasticsearch-8.14.0@lucene-hnsw

1

1

1

Elasticsearch-8.14.0@lucene-int8_hnsw

1

0.986667

1

OpenSearch-2.14.0@lucene-hnsw

1

1

1

OpenSearch-2.14.0@faiss

1

1

1

OpenSearch-2.14.0@faiss-sq_fp16

1

1

1

OpenSearch-2.14.0@nmslib

0.9674

0.910303

0.976394

Elasticsearch and Lucene as clear victors

At Elastic, we are relentlessly innovating Apache Lucene and Elasticsearch to ensure we are able to provide the premier vector database for search and retrieval use cases, including RAG (Retrieval Augmented Generation). Our recent advancements have dramatically increased performance, making vector search faster and more space efficient than before, building upon the gains from Lucene 9.10. This blog presented a study that shows when comparing up-to-date versions Elasticsearch is up to 12 times faster than OpenSearch.

It's worth noting both products use the same version of Lucene (Elasticsearch 8.14 Release Notes and OpenSearch 2.14 Release Notes).

The pace of innovation at Elastic will deliver even more not only for our on-premises and Elastic Cloud customers but those using our stateless platform. Features like support for scalar quantization to int4 will be offered with rigorous testing to ensure customers can utilize these techniques without a significant drop in recall, similar to our testing for int8.

Vector search efficiency is becoming a non-negotiable feature in modern search engines due to the proliferation of AI and machine learning applications. For organizations looking for a powerful search engine capable of keeping up with the demands of high-volume, high-complexity vector data, Elasticsearch is the definitive answer.

Whether expanding an established platform or initiating new projects, integrating Elasticsearch for vector search needs is a strategic move that will yield tangible, long-term benefits. With its proven performance advantage, Elasticsearch is poised to underpin the next wave of innovations in search.

So, why not allow non-normalized vectors in dot-product and thus enable maximum-inner-product? What's the big deal?

Negative values and Lucene optimizations

Lucene requires non-negative scores, so that matching one more clause in a disjunctive query can only make the score greater, not lower. This is actually important for dynamic pruning optimizations such as block-max WAND, whose efficiency is largely defeated if some clauses may produce negative scores. How does this requirement affect non-normalized vectors?

In the normalized case, all vectors are on a unit sphere. This allows handling negative scores to be simple scaling.

Figure 1: Two opposite, two dimensional vectors in a 2d unit sphere (e.g. a unit circle). When calculating the dot-product here, the worst it can be is -1 = [1, 0] * [-1, 0]. Lucene accounts for this by adding 1 to the result.

With vectors retaining their magnitude, the range of possible values is unknown.

Figure 2: When calculating the dot-product for these vectors [2, 2] \* [-5, -5] = -20

To allow Lucene to utilize blockMax WAND with non-normalized vectors, we must scale the scores. This is a fairly simple solution. Lucene will scale non-normalize vectors with a simple piecewise function:

Now all negative scores are between 0-1, and all positives are scaled above 1. This still ensures that higher values mean better matches and removes negative scores. Simple enough, but this is not the final hurdle.

The triangle problem

Maximum-inner-product doesn't follow the same rules as of simple euclidean spaces. The simple assumed knowledge of the triangle inequality is abandoned. Unintuitively, a vector is no longer nearest to itself. This can be troubling. Lucene’s underlying index structure for vectors is Hierarchical Navigable Small World (HNSW). This being a graph based algorithm, it might rely on euclidean space assumptions. Or would exploring the graph be too slow in non-euclidean space?

Some research has indicated that a transformation into euclidean space is required for fast search. Others have gone through the trouble of updating their vector storage enforcing transformations into euclidean space.

This caused us to pause and dig deep into some data. The key question is this: does HNSW provide good recall and latency with maximum-inner-product search? While the original HNSW paper and other published research indicate that it does, we needed to do our due diligence.

Experiments and results: Maximum-inner-product in Lucene

The experiments we ran were simple. All of the experiments are over real data sets or slightly modified real data sets. This is vital for benchmarking as modern neural networks create vectors that adhere to specific characteristics (see discussion in section 7.8 of this paper). We measured latency (in milliseconds) vs. recall over non-normalized vectors. Comparing the numbers with the same measurements but with a euclidean space transformation. In each case, the vectors were indexed into Lucene’s HNSW implementation and we measured for 1000 iterations of queries. Three individual cases were considered for each dataset: data inserted ordered by magnitude (lesser to greater), data inserted in a random order, and data inserted in reverse order (greater to lesser).

Here are some results from real datasets from Cohere:

Figure 3: Here are results for the Cohere’s Multilingual model embedding wikipedia articles. Available on HuggingFace. The first 100k documents were indexed and tested.

Figure 4: This is a mixture of Cohere’s English and Japanese embeddings over wikipedia. Both datasets are available on HuggingFace.

We also tested against some synthetic datasets to ensure our rigor. We created a data set with e5-small-v2 and scaled the vector's magnitudes by different statistical distributions. For brevity, I will only show two distributions.

Figure 5: Pareto distribution of magnitudes. A pareto distribution has a “fat tail” meaning there is a portion of the distribution with a much larger magnitude than others.

Figure 6: Gamma distribution of magnitudes. This distribution can have high variance and makes it unique in our experiments.

In all our experiments, the only time where the transformation seemed warranted was the synthetic dataset created with the gamma distribution. Even then, the vectors must be inserted in reverse order, largest magnitudes first, to justify the transformation. These are exceptional cases.

If you want to read about all the experiments, and about all the mistakes and improvements along the way, here is the Lucene Github issue with all the details (and mistakes along the way). Here’s one for open research and development!

Conclusion

This has been quite a journey requiring many investigations to make sure maximum-inner-product can be supported in Lucene. We believe the data speaks for itself. No significant transformations required or significant changes to Lucene. All this work will soon unlock maximum-inner-product support with Elasticsearch and allow models like the ones provided by Cohere to be first class citizens in the Elastic Stack.