The changes are about storage and performance, along with the out-of-the-box (OOTB) experience. Up until 9.5.0, Elasticsearch operated as a document-based search engine by default. It could be set up to behave like a columnar system storage-wise, but that wasn’t the OOTB experience. Columnar index modes make a number of fundamental changes that allow Elasticsearch to optimize columnar analytic and search use cases:

• Fields are stored once as doc values only and are no longer indexed by default.

• New multi-value semantics. The original ordering of multiple values per field per document (for example, in arrays) is preserved by default.

• Mappings are always flat, and object and passthrough fields in mappings are always auto-flattened.

How columnar index modes fit into Elasticsearch

Many of the columnar index mode changes originate from time series data streams (TSDSs). As part of making TSDB a competitive metrics solution, we improved doc values format on disk and only store dimensions and metric fields once as doc values. We also improved query performance. TSDB is already columnar today. Essentially, this makes TSDB’s storage mode columnar. The lessons learned from TSDB are now being applied more broadly to Elasticsearch. 

Note that columnar index modes are opt-in and columnar, and document-based indices can coexist in the same cluster. An enterprise search use case can use a document-oriented index mode, while a logging use case can use logsdb_columnar index mode and be fully columnar, all in the same cluster. In fact, there are currently seven index modes, and indices can all use them in the same cluster.

How columnar storage stays fast without an inverted index

Indexed fields, either an inverted index for string-based fields or block k-dimensional (BKD) tree for numeric fields, allow Elasticsearch to query or filter by field very efficiently. However, the cost for this is an additional expensive data structure that uses a lot of disk space and is expensive to build at index time and at merge time. With the columnar index modes, fields are no longer indexed by default, so what did we do for query performance to be still acceptable on fields that were no longer indexed?

One major change was improving doc values scanning performance. This is key and is the cornerstone that any columnar system relies on. Previously, the scanning of doc values was essentially document by document. Lucene’s doc values API only allowed for looking up one value at a time. Historically, this fit the execution model of a search engine. In our own doc value format, we build the capability to allow bulk reading of values for Elasticsearch Query Language (ES|QL) queries. Also over recent minor Lucene releases, Lucene doc values API added support for bulk reading. Without this, fast columnar scanning wouldn’t have been possible.

Secondly, we fully adopted doc value skippers, a hierarchical skiplist over doc values. Contrary to an inverted index or a BKD tree, doc values skippers are lightweight data structures. At its core, a skipper allows queries to skip over a range of documents that don’t match a query. It can do this because it stores information, like min and max values. So, for example, when a range query is executed, an interval of documents can be skipped based on the intermediate result and a doc value skipper’s min and max values. The effectiveness of doc value skippers depends on the order in which documents are laid out on disk. This is why index sorting should be enabled or altered to match the use case.

By significantly improving our columnar scanning and doubling down on doc values skippers, we’re able to avoid indexing fields by default. Note that a field can still be indexed; the index mapping attribute just defaults to false in columnar mode. The exceptions to this rule are text-based fields, which are still indexed by default. This is because text fields provide free text search, which includes text analysis, along with phrase and wildcard matching. This is different from just filtering.

How Elasticsearch handles high and low cardinality fields

When setting up a schema with string fields, an important configuration parameter is often cardinality; that is, whether many unique values or a few unique string values are expected. Many systems have dedicated field or column types that target low and high cardinality string fields.

Fields that have low cardinality are typically stored with a dictionary, containing all unique values. Then, for each row offset, the offset into the dictionary containing the term the row has is stored. This is often called an ordinal. For low cardinality fields, this works well, as storing an ordinal per document takes up much less space. Encoding techniques, like delta encoding, offset encoding, and bitpacking, work well for ordinals to compact the per-document storage to just a few bits. 

However, for high cardinality fields, the dictionary and ordinal approach can work counterintuitively. If a larger percentage of the documents have a unique value, building the dictionary becomes expensive and storage savings diminish. The dictionary then becomes another level of indirection for reading values. This is why most systems in that case store values in a columnar fashion using block-based compression. For example, values of multiple rows are stored in 128KB blocks using a sliding-window dictionary-based compression algorithm (like zstandard or lz4). This, in general, is a simple and effective method to store higher cardinality fields and avoids building and maintaining a dictionary. 

With document-based Elasticsearch, there are two ways to map a string: using either the keyword field mapping or one of the text-based field mappings. The former stores an inverted index and dictionary-based doc values. The latter only stores an inverted index. This is why, typically, a text field mapper is often used in combination with a keyword mapper as a multi-field. Also, keyword field mapper (as the name suggests) is meant for keywords or fields that have a lower cardinality and uses dictionary-based doc values implementation. However, in practice, keyword field mappers are also used for high cardinality.

In columnar mode, every field is only stored once by default. For keyword fields, this means only doc values are stored with no inverted index. Text-based fields now also store doc values and an inverted index by default. Text-based field mappers are different from the keyword field mapper, as these are not automatically used via Elasticsearch’s dynamic mapping logic for columnar indices, and therefore text fields keep storing an inverted index by default.

For both keyword- and text-based fields, we didn’t choose to expose a cardinality mapping attribute. It’s not always possible to know ahead of time whether a field is low or high cardinality. When flushing and merging segments to disk, Elasticsearch sees all values and can determine the cardinality of a field. This is why we’re choosing to automatically determine whether the usage of a dictionary and ordinal-based encoding is beneficial over block-based compression using a simple cardinality threshold. If a field is below this threshold, dictionary and ordinal-based encoding is used; otherwise block-based compression is used. This simplifies configuration of the mappings and makes it possible to automatically optimize storage as data evolves, since some segments may use dictionaries while others may use blocks for the same field. However, this is currently not ready yet and so, as part of 9.5.0, in columnar mode, both keyword- and text-based field mappers store values in doc values in a block-based compressed layout on disk.

The two approaches compare as follows:

Dictionary and ordinal encoding

Block-based compression

Suits

Low cardinality fields

High cardinality fields

What’s stored

A dictionary of unique values, plus one ordinal per document

Values for many documents compressed together in blocks

Compression

Delta encoding, offset encoding, and bitpacking reduce each ordinal to a few bits

Sliding-window dictionary compression, such as zstandard or lz4, typically over 128KB blocks

Read path

Resolve the ordinal, and then look up the value in the dictionary

Decompress the block, and then read the value directly

Cost as cardinality rises

Dictionary grows large, savings shrink, and the extra indirection stays

Stable, with no dictionary to build or maintain

Used in columnar mode tech preview

Not yet

Yes, for both keyword and text fields

Columnar mapping attributes: multi_value, nullability, on_failure

The columnar index modes provide more control over how data is stored as doc values. By default, Elasticsearch is lenient and accepts all non-malformed values (for example, nulls and multiple values per field and document). If documents have fields with multiple values per document or no value, doc values store additional data structures to deal with them and therefore implicitly increase costs. 

With columnar, new mapping attributes provide additional control. Note that these new mapping attributes are currently only available with the columnar index modes but will eventually also be available for all index modes.

Three new mapping attributes are involved:

Attribute

Default

Enforces

On violation

Available

multi_value

true

One value per document per field

Document indexing fails

9.5.0

nullability

true

Field must have a value

Document indexing fails

9.5.0

on_failure

fail

How the above failures are handled

Sets fail or ignore behavior

Next minor release

multi_value: Enforcing single-valued fields

By default, Elasticsearch accepts multiple values per document. To understand the implications of this, we first should take a look at how Elasticsearch (using Lucene’s doc values) stores a dense numeric field where all documents have a single value:

With this layout, all values are stored in blocks. The number of values per block depends on the index mode but is typically 128 values and is always the same within an index. All values in a block are encoded using various encoding techniques, like delta encoding and bit packing, so each block can have a different size, depending on how well the encoding techniques compress the values. This is why a block index is required. 

Lucene has the notion of a docid (internal numbering for a document), which is essentially a row identifier.

1. Queries produce matching docids.

2. In case of a dense field, the block id can be resolved from the docid directly.

3. The offset of a block can be resolved from the block index.

4. Once that has been looked up, the target block gets decoded and all values are available.

5. Finally, from docid, the ordinal within the decoded values array can be resolved, which produces the final value.

Now let’s have a look at how the data layout changes when documents have multiple values per document:

To determine how many values belong to a single docid, an offset lookup is required.

In this case, a docid has one or more offsets. Each offset points to a block index. Values for a single document are adjacent but can stretch over blocks. In general, compaction of values works well in blocks because values are similar. However, multi-value fields can cause the compaction of values to be less efficient if the number of values per field and document is large and values aren’t similar. This and the additional storage of offsets result in multi-value fields typically having a higher storage footprint on disk.

If a field is truly single-valued, you may want to enforce this property. The multi_value mapping attribute makes that possible now. An example is a log level field. Logs typically have one log level (such as debug, info, or error). Enforcing that this field is single-valued in your mappings can help avoid accidentally using more storage than anticipated. A mapping snippet example that disallows the field log.level to have multiple values per document:

Note that even if a field allows multiple values, this doesn’t mean an offset lookup is stored. This only happens when a Lucene segment has at least one document with two or more values. The multi_value mapping attribute exists just for enforcement.

nullability: Requiring every document to have a value

By default, Elasticsearch accepts documents with fields that have no value or null value. Just as  multiple values per document require additional accounting, documents with no value require additional accounting to identify which of them have at least one value.

Doc values store a docid to offset lookup (known as IndexedDISI in Lucene) in case not all documents have a value in a segment. The offset either points directly to the block index for single-valued fields or the offset lookup in case of multi-valued fields. This lookup is compact compared to the value blocks being stored. However, if it were to be created for fields that should have at least one value per document, that would be a waste.

The nullability mapping attribute allows you to control whether documents are allowed to have no value. Just like the multi_value mapping attribute, the nullability attribute exists for enforcement.  Following is a mapping snippet example that requires the log.level field to have a value:

on_failure: What happens when validation fails

What happens if a document has multiple values for a field and if the multi_value mapping attribute is set to false or when a field is mapped with nullability set to false and a document doesn’t have that field? At the moment, indexing such documents will fail with a bad request error.

As part of the next minor release, the on_failure mapping attribute will be available. This allows you to indicate how to handle these validation failures, on a per-mapped field basis. This will support two values:

1. Fail: Fail indexing of the entire document with a client error. This is the current behavior in Elasticsearch 9.5.0.

2. Ignore: Ignore the validation error, mark the field as ignored, and store values for that field in a hidden field so that it can be introspected when requesting the source. 

Trying the columnar index modes

The columnar index modes are still under active development, but we encourage you to give them a test drive. As we prepare the columnar index modes for general availability (GA), we’ll add more performance and efficiency improvements. We believe that by adapting a columnar mindset, many use cases will benefit from being more cost effective or having better performance characteristics.

Columnar mode is Elasticsearch's analytics-optimized columnar storage mode, built for scan-heavy workloads, like log analytics. In this mode, keyword fields don't get an inverted index by default, so term and wildcard queries scan doc values. DocValuesSkippers (zone maps) already trim how much data a scan touches, but these rewrites cut the cost of what's left. 

How Lucene's query rewrite mechanism works

In Lucene, every query has the option to implement a rewrite method that returns another query. This method returns a query with the same semantics but a different implementation. The query engine repeatedly calls the rewrite method until the returned query doesn’t change. This final query is the one that’s actually evaluated. Importantly, the rewrite can see the actual query arguments and specialize the implementation based on these.

For example, in a query looking for documents where a string field contains the value "foo", the rewrite method knows that the term we’re searching for is "foo". In theory, rewrite could replace the general query class with something specific to "foo". For example, the original query class ScanningBinaryDocValuesTermQuery could be replaced with FooQuery. Now this rule probably wouldn't be helpful, but it gives a sense for the level of specialization that’s achievable with rewrite rules.

Rewrite rules and query optimization in database systems

It's worth placing rewrite rules in the larger context of database systems. Lucene and Elasticsearch aren’t the first systems to use transformation rules to optimize queries. Most (or maybe all) database systems use some kind of rule system during query optimization. The most influential rewrite rule system was in IBM's Starburst database. This system's core contribution was extensibility; for example, it was possible to add new data types and storage methods, along with (most importantly to us) optimizer rewrite rules.

Each rule consisted of two parts:

1. A condition function: A predicate determining whether the rule applies to the current query graph.

2. An action function: The transformation that rewrites the query plan into a more optimal form.

A rule engine applied matching rules until a stopping condition was met.

Though Lucene's rewrite method is superficially different from these condition and action functions, it achieves the same goal. It checks whether certain conditions match, and if they do, it applies the rewrite by returning a new query. If conditions don’t match, the rewrite returns this, replacing the query with itself; that is, choosing not to apply the rule.

Why these rules live in Lucene, not the ES|QL query optimizer

Elasticsearch actually contains a separate rewrite rule system within the Elasticsearch Query Language (ES|QL) optimizer. This operates on the high-level structure of a query; for example, doing predicate pushdown to avoid unnecessary computation on documents that will be filtered out. But it’s still useful to have the rule system within Lucene. Since Lucene acts as the storage layer for ES|QL (and classic _search) queries, it’s easier to express rewrites that take advantage of the physical data format in Lucene rather than in a higher-level optimizer.

A query rewrite rule for wildcard queries: Simpler code, no automaton

Wildcard queries support the ? and * operators to match any character once or any character multiple times. These operators can appear any number of times in a wildcard query. As with regexes, to evaluate whether a string matches a wildcard query, we build an automaton from the query string and then use the string bytes to do state transitions through the automaton. This is relatively fast, but if you have to evaluate it for every document, the latency really adds up.

But maybe we don't always have to run an automaton. Consider a query like *foo*. How would you implement this if you were writing a simple query engine to find matching strings in a list of strings? Pretty much every programming language has the tool you want built in: a method that finds a substring within a given string. This function doesn't need a complicated automaton; it probably just consists of a couple of for loops.

Now of course we couldn't use this function to implement an arbitrary wildcard query, but we don't have to. The rule rewrite system isn't for the general form. It's for implementing special cases, and it can see the specific query. It knows that we’re looking for *foo* and realizes that this specific case doesn't require the heavyweight automaton machinery. And it can do the same for any query that starts and ends with a *, with some term in the middle.

The following pseudo-code shows the pattern. At the top, we have the generic WildcardQuery. It has two notable fields: the query string (for example, *foo*) and the automaton built for that query. The matches method checks whether the field value for a given docId is a match by using it to evaluate the state transitions of the automaton. More interestingly, its rewrite method checks whether the query matches our special case. We show this with a regex that checks whether the query string starts with a *, has any non-*characters at least once, and then ends in a *. If so, we return the special case as a ContainsQuery and pass in the inner query string (since it doesn't care about the *s). The ContainsQuery then just does a simple contains check to see whether the term bytes are somewhere within the value bytes.

Benchmarking the wildcard rewrite on ClickBench Q20

The wildcard rewrite is straightforward, but does it actually work? Yes, we can use the ClickBench benchmark, which has several queries of this form. Query 20 (Q20) is FROM hits | WHERE URL LIKE "*google*" | STATS count = COUNT(*). It's exactly the query shape that this rule matches: a string match against the wildcard query *google*. And since the query is just counting, we can see exactly how well this technique works. It turns out to be quite effective. Q20 saw a 1.75x improvement on median latency of hot query times, with no filter cache. All benchmarks in this post were run on an Intel Core i9-13900H.

Adding SIMD to the substring search: 1.75x to 2.3x

But can we do better? Yes, switching to a simple contains check opens up a new possibility. Instead of using the two for loops, we can swap scalar logic for single instruction, multiple data (SIMD) logic. Elasticsearch uses the Panama vector API (see our post on SIMD in Elasticsearch), which lets us implement the contains check in SIMD. This works particularly well for longer strings that can take advantage of the wide SIMD registers; for strings under 24 characters, we still use the scalar approach. With this change, we saw another 1.32x improvement, resulting in a total speedup of 2.3x over the automaton-based approach.

A query rewrite rule for empty strings: Less data, no decompression

One benefit of Lucene-based rules is that they’re low level and can fit to the data format. That’s the case for this rule, which applies to string data. 

How columnar storage encodes string data

In Elasticsearch's standard mode, string values are stored by document; this is a row-major format. But in columnar mode, unsurprisingly, the data is stored in columnar-major format. A column of string data is stored in chunks. Each chunk contains many string values and consists of an array of integer offsets and a (Zstd-compressed) blob of the strings' bytes. For a string at index i, offsets[i] points to the offset in the decompressed byte blob where the string starts. So the length of string i can be computed from offset[i+1]-offsets[i]. (There's a dummy extra offset at the end, so we can easily compute the length of the last string). The following diagram shows how a chunk with the strings ‘Feta’, ‘Asiago’, ‘’, ‘Stilton’, and ‘Brie’ is encoded.

Why a term query has to decompress the chunk

Now that we understand the columnar format, let's get back to query optimization. First, consider a term query for the query foo. We’re looking for documents where a given string field exactly matches the string foo. So how do we implement this on a string column in the above format? The algorithm is straightforward:

The bottleneck is the Zstd decompression step. But there's not much we can do about that; if we want to check the bytes, we have to decompress the chunks. But remember, we aren't trying to optimize the general case, we’re looking for special cases. (In reality, you don't just try to think up special cases. These optimizations came about by first running a useful query, realizing that it could be faster, and then looking for ways to improve it.)

Rewriting the empty string query as a length check

One special case we found that’s worth improving is a query for the term "". Admittedly, it's a silly term, but empty strings are all over the place. Since they're rarely useful, we usually filter them out with a query like term != "". Thankfully, this is a query we can optimize.

Consider the above algorithm for the empty string term. The line if value == term is a bit weird; we’re asking Does this value equal the empty string? We can do that, but there are no bytes to compare, so the check unwinds:

1. We only need to know whether the value has length 0.

2. If we only need the length, we don't need to look up the value in the decompressed chunk.

3. If we never look up a value, we don't need any bytes from the chunk at all.

4. If we need no bytes from the chunk, we don't need to decompress it.

All we need are the lengths, and those live in the offsets array. It's compressed, too, but with cheap integer compression rather than Zstd, which is much faster.

With this realization, we can rewrite empty string term queries. The one new operation we need is docValues.loadLength(docId), which reads directly from the offset array without touching the compressed bytes. After the previous example, this should look familiar. The most interesting part is TermEqualsQuery.rewrite; it finds the empty string special case and replaces the query with the simpler version that only checks the length.

Benchmarking the empty string rewrite: 1.6x faster

Now let's see how this stacks up. There aren't any pure-scan ClickBench queries that use this rule as directly as Q20 does for the previous rule, so we'll make our own. Consider the query: FROM hits | WHERE SearchPhrase != '' | STATS count(*). On this query, we see a 1.6x speedup, which is a great improvement for a fairly uncomplicated change. Better yet, ES|QL can take advantage of loadLength directly. Any time that ES|QL accesses a string's BYTE_LENGTH, without needing the string itself, the request uses this same specialized length loading to avoid unnecessary decompression.

What makes a good query rewrite rule

The two rules covered here follow the same shape: identify that a query is a special case, and then swap it for a cheaper implementation. But they reduce cost in different ways. 

Wildcard rule

Empty string rule

Query shape detected

*term*

field == ""

Replaced with

SIMD substring search

Length check on the offsets array

Cost reduced

Algorithmic work

Data access

Speedup

2.3x

1.6x

The underlying pattern is worth noting: finding a query that leaves performance on the table, finding a special case that can be optimized, and swapping in a cheaper implementation. The hard parts are finding queries that uncover these opportunities for optimization and then identifying the special cases. The actual fix is often relatively straightforward, as both rules here show. Our work on columnar mode has provided many opportunities to run interesting queries and hunt down exactly these kinds of wins.

That's also why extensibility in a rule system is so important. These rules can't be built into a database from the start; they're found through an incremental discovery process. Lucene's rewrite system makes that practical. As columnar mode grows to handle new workloads, rules like these will keep emerging.

To try columnar mode and the optimizations described in this article, use Elastic Cloud Serverless or Elasticsearch 9.5 or later, where columnar mode is available as a technical preview.