CPU-based indexing can also degrade search performance when indexing and search happen simultaneously, especially in online systems, like ecommerce platforms and ad-serving pipelines. This happens because index builds consume significant CPU resources, leaving fewer cycles available for queries and driving up search latency and latency variance. Even after indexing completes, latency can remain high due to index fragmentation. Customers often run a forced merge to consolidate segments, but it can be slow on CPUs, further delaying recovery to the expected search latency by hours to days.

Elasticsearch has introduced GPU-accelerated vector indexing powered by NVIDIA cuVS. In this blog, we show how Elasticsearch achieves up to 7x faster vector indexing throughput by offloading hierarchical navigable small world (HNSW) index construction to GPUs, indexing 138 million vectors in under 10 minutes using eight NVIDIA RTX PRO 6000 GPUs.

Indexing over 100M Vectors in under 10 minutes

Figure 1 shows vector indexing throughput on 138 million 1,024-dimensional vectors from the MS MARCO dataset, representing roughly 4 TB of multimodal documents (Figure 1). The benchmark was run with Rally, Elasticsearch’s macro-benchmarking framework. The test ran on a server with 8 NVIDIA RTX Pro 6000 GPUs with a 2 socket AMD EPYC 9555. GPUs were turned on for the GPU test and turned off for the CPU test. The result was that Elasticsearch achieved 7x higher indexing throughput, lowering time to index 138M vectors from 1 hour on CPUs to under 10 minutes on GPUs.

Does GPU vector indexing change recall?

Index acceleration isn’t helpful if indexes built on GPU have lower search quality compared to CPUs. The throughput-versus-recall curves for GPU-built and CPU-built indexes overlap, showing that GPU indexing delivers the same recall profile as Elasticsearch’s CPU-based indexing path, with minimal accuracy or search-time performance penalty (Figure 2).

Search latency under concurrent indexing load

Offloading indexing to the GPU improved search latency on the CPU by 6x when running indexing and search simultaneously (Figure 3) because indexing on GPUs frees CPU resources for faster search latency. This allows a single Elasticsearch cluster to support real-time ingestion and low-latency retrieval at the same time, without forcing a tradeoff between freshness and search performance.

Force merge time for HNSW indexes on GPU versus CPU

Additionally, GPU-accelerated indexing in Elasticsearch reduced forced merge time from roughly four hours to five minutes, helping restore low-latency search in real-time (Figure 4). Note: Force merge was used to merge each shard into four segments.

Here are the full results for GPU versus CPU vector indexing in Elasticsearch:

Metric

CPU

GPU

Change

Index build time, 138 million vectors

~1 hour

Under 10 min

7x throughput

p90 search latency under indexing load

Baseline

6x lower

6x

Force merge, four segments per shard

~4 hours

~5 min

~48x

Recall at target

95%

95%

Unchanged

What GPU vector indexing means for Elasticsearch users

Elasticsearch’s integration of NVIDIA cuVS brings GPU-accelerated vector indexing to production AI search with over 84% reduction in search latency to deliver faster ingest with a 640% increase in indexing throughput with a lower CPU overhead and limited infrastructure tradeoffs. This unlocks a new class of applications, such as faster RAG pipelines, semantic search, multimodal retrieval, and fraud detection at scale.

Get started with GPU vector indexing in Elasticsearch

Install Elasticsearch with GPU-accelerated vector indexing by following these instructions. Replicate the above benchmarks by running rally msmarco track with parameters in the GitHub issue. Learn more about GPU-accelerated vector indexing, search, and preprocessing by visiting NVIDIA cuVS. 

Acknowledgments

The authors would like to thank Chris Hegarty, Gilad Gal, and Mayya Sharipova from Elastic, as well as Nathan Stephens from NVIDIA, for their contributions to this article.

What is the Kibana chrome?

The chrome is everything that frames an application in Kibana: the global header and the navigation, along with the header of the page that you’re currently on. Every application renders inside it. Because the chrome is shared, its APIs decide how consistent the whole product feels and how hard the next redesign will be.

What inconsistent page headers looked like to users

The clearest way to see the problem is to look at what users saw:

Take something as basic as a primary action, the one button a page most wants you to click, like Create index. Before this work, primary actions lived in three different places, depending on the page:

• In the old app header.

• In the page template header.

• Somewhere in the page body itself.

Common links had the same problem, with share, feedback, and documentation links appearing in separate spots with different styling from one app to the next. Breadcrumbs were misconfigured on many pages. For example, some showed a breadcrumb for each tab within a page, and some weren't clickable. Still others repeated the title of the page you were already on.

Every one of these was the output of a well-intentioned team using a flexible API in the way it’s usually used. Users had to relearn each page's layout, and every attempt to redesign the chrome had to account for hundreds of local variations.

Goals for the page header redesign

The goal of the redesign was simple; that is, pages should express what they need (for example, a title, badge, or primary action), and the shared shell should decide how it looks and where it goes. Common links and primary actions should get one home, everywhere. If it's the primary action, it's always in the same place, styled the same way. And consistency should stop being something that each of dozens of teams has to get right by hand, forever.

Getting there wasn't a restyling exercise; it required changing the contract between applications and the platform.

How typed props replaced EuiPageHeader's open React nodes

Each page header was easy to build in isolation, and the local choices piled up into visible differences across the product: inconsistent spacing, title styling, and action placement from one page to the next.

The fix, everywhere this showed up, was the same mechanism: stricter APIs. Instead of accepting arbitrary React nodes, a control declares what it means through typed props, and the shared shell decides how it looks and where it goes. That one change buys two things at once: consistency, and governance over what the shared surface is allowed to contain.

Take the page header. With EuiPageHeader, the API exposed layout settings and accepted React nodes for nearly every visible area:

Because pageTitle, description, and rightSideItems all accept arbitrary React nodes, every team filled them in differently. One page put a badge next to the title, while another styled its own pill. One team's actions were plain buttons in one order; the next team's were a different mix in another order, some collapsing into a menu and others not. Everyone used the same shell component, yet the headers looked and behaved like they came from different products. The shared component guaranteed the wrapper, not what went inside it.

The AppHeader API closes that off. The same header is expressed as typed props:

Now there’s only one way to express each part, so every page renders it the same. Badges live in their own typed collection, separate from the title. A description carries its text and, optionally, a "Learn more" URL. Actions declare whether they’re primary or secondary, and the component decides how they look and when they collapse. Plus, richer controls fit the same shape. An editable title or a favorite toggle is a structured config rather than a bespoke React tree:

The split is consistent throughout; the application supplies state and behavior (the current name, what happens on rename, how to toggle a favorite) and the header owns presentation. It renders the controls, handles keyboard interaction, shows validation errors, and reflects pending changes. Because those roles are explicit, layout and responsive behavior stay inside the shared component, and applications don't need a coordinated update every time the header changes.

Why a shared UI shell needs a closed set of controls

The global shell had a similar problem that was one step more open-ended. Any plugin could register arbitrary content on the left or right and pick its position with a number, and no conversation with the platform team or designers was required:

That registerRight API is an open door. Any team can add a control, and the header fills up with elements no one designed together. The controls compete for space and carry their own styling, and their order is decided by whoever picked the larger number. The shell only knows that one React node follows another; it can't tell that one opens an AI assistant and another collects feedback, so it can't reason about them or keep them coherent.

The new shell replaces that open canvas with a closed set of named roles under chrome.controls and chrome.help. The old registry hasn't gone anywhere yet, because the classic header still runs alongside the new one while applications migrate, but nothing in the new shell is reachable through it:

Global chrome header versus application page header

Part of what made the old chrome hard to evolve was that "the header" was really several things tangled together. The redesign draws a hard line between two surfaces with different owners:

Chrome (global) header

Application header

Owner

The platform

The page

Scope

What's true everywhere in Kibana

What's true on this page

Contents

Navigation, project or deployment picker, search, help, AI assistant, feedback

Title, badges, description, tabs, page actions

How it's populated

Named slots under chrome.controls and chrome.help

AppHeader typed props, rendered directly or set via chrome.appHeader.set()

Because each surface has one owner and a typed contract between them, the platform can redesign the chrome without auditing hundreds of pages, and an application can evolve its header without colliding with global controls. Setting a header config returns a cleanup callback, so an application tears down its own contribution when it unmounts.

Design decisions that strict APIs force

Stricter APIs forced design decisions that flexible APIs had let everyone defer. Two are worth calling out.

Why we removed breadcrumbs from Kibana's navigation

Removing breadcrumbs from the chrome was one of the more debated calls, and the state of the data made it easier than expected. In practice, breadcrumbs across Kibana were widely misconfigured:

• Some pages generated a breadcrumb for every tab on the page.

• Some breadcrumbs weren't clickable.

• Some repeated the title of the current page.

They added visual weight without reliably adding orientation.

At the same time, the redesigned navigation already conveys hierarchy. The primary and secondary navigation show you where you are and give you a direct way back up. Keeping breadcrumbs would have meant maintaining a third (and frequently wrong) representation of the same information, so we stopped drawing the trail and let the navigation do that job.

The breadcrumb API itself didn't disappear. Applications still register breadcrumbs, and the new shell reads them to derive the back button and to fall back to a page title when an app hasn't supplied one. The data changed jobs rather than going away, which is why migrating an app to the new header rarely starts with ripping breadcrumbs out:

Before:

After:

How many action buttons a page header should show

The other recurring debate was priority. With every page's actions now flowing through one menu structure, which buttons deserve to be visible, and in what order? Priorities shift as products evolve, so this conversation is ongoing.

For launch, we made one deliberate simplification: we locked the number of buttons that can appear in the app menu. A page gets one primary action and a bounded set of secondary actions before the rest collapse into a menu. The constraint keeps any single page from dividing the user's attention across a wall of buttons, and it makes the priority conversation explicit instead of letting it be settled by whoever adds the next button.

We went through several iterations to land here. Initially, we had many visible actions. A page could have up to three buttons, plus a secondary action to the left of the primary one. That took up too much space, so we gradually simplified the layout by removing the secondary action and reducing the number of visible buttons. We also made the overflow menu more structured. Some items, such as feedback and documentation, now have a fixed place in the footer of the overflow menu.

What strict design system governance costs teams

Under the old API, a plugin could add a novel UI by choosing a side, picking an order, and mounting a React tree. Under the stricter API, a new kind of control may need a shared capability before a plugin can add it. That takes more work up front and forces a design decision that teams could previously avoid.

We accepted the cost of stricter APIs because the alternative pushed it into every later redesign. As long as arbitrary React trees mounted at arbitrary points, every change to the shell's layout or accessibility had to account for all of them.

That’s the trade we made, and it pays back on both fronts. Consistency stops being something each team has to get right by hand; there’s one way to express a control, so pages match by default. And the shared surface stays governed. Its set of controls is a deliberate design decision, not whatever accumulated at the edges. Applications describe what their controls mean, and the shell is free to decide how they look, both today and in the next redesign.

Where the redesigned Kibana chrome is available

The redesigned chrome is available now in Elastic Cloud Serverless. Open any project, and you're already using it. For Elastic Cloud Hosted and self-managed users, it ships in 9.6.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Doc values were built to make aggregations, sorting, and grouping possible on a document engine, and they do that job well. Columnar Mode changes what the columns are for, and this post walks through the five properties that separate storing columns from being a columnar database.

Five properties separate a column store from a columnar database

Doc values were an optimization on top of _source

For most of the last decade, the original JSON document was the source of truth and the columns were a derived convenience. That ordering has consequences throughout the engine.

Because the engine could always fall back to _source, per-field storage was allowed to be lossy. Text fields had no doc values at all, since text could be reread from the stored document when needed. Even synthetic source, which reconstructs a document from its fields rather than storing a copy, sometimes reads from row-shaped structures to stay faithful to the JSON that arrived, with values that exceed ignore_above and fields that arrived unmapped going into stored fields.

The result is a clear contract; whatever JSON you send, you get back, and the columns accelerate everything else. For an engine whose job is to return your documents, that’s the right way round. Columnar Mode inverts it. Every field stores itself exactly once as doc values, doc values cannot be turned off, text fields get doc values, too, and the document is reconstructed from the columns when something asks for it.

How dictionary encoding handles high-cardinality data

Sorted doc values, the default for keyword fields, store a dictionary of distinct values plus one ordinal per document pointing into it. This is an excellent trade when values repeat. A host.name field drawn from a hundred machines, or a status code that’s almost always 200, compresses beautifully and groups quickly.

It works like the index cards in a warehouse. When 50 crates hold the same product, one card and 50 pointers beats writing the product name 50 times. When 50 crates hold the same product, you can store the product name in the index with the list of 50 crate IDs. When every crate holds something unique, you might as well just put the product name on the crates; the index will help you find what crate you want, but it won't save ink.

High-cardinality fields describe a lot of real data, including URLs and trace identifiers, along with message bodies. Columnar Mode, which skips the dictionary and compresses the values in blocks instead, uses binary doc values for high-cardinality strings. Which of the two a field gets isn’t something you configure. The engine decides per field, based on the values it sees, so each column is encoded for the data it actually holds instead of one default applied to every field. Pure columnar systems have long carried cardinality in their type system, but usually as something you declare, and you own the consequences when the data shifts underneath it. Here, it’s the engine's job.

Why every field builds an inverted index by default

By default, a keyword field also builds an inverted index and a numeric field also builds a BKD tree. That happens on every field because at write time the engine doesn’t know which capability you’ll want at read time, significantly increasing the footprint of each field. Those structures also have to be rebuilt during segment merges, which costs CPU exactly when ingest is heaviest.

Our time series engine (TSDB) is proof of what happens when you stop paying for capability that the workload doesn’t use. Replacing the indices on @timestamp and dimension fields with doc value skippers, which are sparse structures holding the minimum and maximum value for each block of documents, removed 10 bytes of the original 25 bytes per OpenTelemetry (OTel) data point. There was no measurable query regression on time range and dimension filters, and indexing CPU dropped by about 10%, as a bonus.

Columnar Mode generalizes that default. Fields aren’t indexed unless something needs them to be, with only text-mapped fields keeping their inverted index for fast free-text search.

Metadata fields like _id and _routing were row-shaped

The fields you never think about followed the same document-first design. The _id field was a stored field plus an inverted index. Custom _routing was a stored field. Sequence numbers were kept for optimistic concurrency control, regardless of whether a workload ever updated a document.

TSDB deals with all three. It synthesizes _id from the _tsid and @timestamp values that already identify a data point, using a segment-level bloom filter to catch duplicates, which removes 5 bytes per data point with no loss of functionality. It trims sequence numbers once replication no longer needs them, which removes 4 bytes. Add a codec block size increase from 128 to 512 elements for another 2 bytes, and those four changes contribute across versions 9.1 through 9.4 to the 21 bytes that took OTel metrics from 25 bytes per data point down to 3.75.

Columnar Mode makes those ideas general rather than metrics-specific. By general availability (GA), all metadata fields will store themselves as doc values, while we plan to follow up and add a sort id mode synthesizing the identifier from the index sort fields, in addition to derived fields that will generalize what _tsid does for time series to any set of fields.

Columnar query execution in the ES|QL compute engine

A column store only pays off if the engine reads it as columns. Aggregations inherited the document-at-a-time shape from search, which is the natural fit for an engine built around documents. Reading columns instead lets the engine hand a whole block of values to a single instruction, and that’s where the numbers below come from.

The ES|QL compute engine changed that shape, and TSDB again shows the size of the effect:

• Vectorized execution of time series aggregations was worth up to 8x on its own.

• Decoding on-disk data straight into the primitive arrays the engine aggregates over, with no intermediate copies, was worth roughly another 10x.

• Constant blocks turned repeated values into a form of in-memory run-length encoding.

• Filter pushdown moved filters down to Lucene, where skippers can discard whole blocks unopened.

Together with the rest of the block-level query work, query latency improved by up to 160x compared to earlier versions.

That work continues. Skipper-aware operators, aggregations that group on ordinals and convert to real values as late as possible, and richer per-block summaries are all in progress, and they benefit every index mode because every mode reads doc values underneath.

What Columnar Mode changes for logs and analytical data

Storing values in columns is a storage detail. A columnar database needs five things:

1. The columns are the only copy of the data.

2. Each column is encoded for the data it actually holds.

3. Metadata is columnar, too.

4. Fields add indices only when something needs them.

5. The query engine processes blocks of values rather than records or documents.

TSDB reached all five for metrics in Elasticsearch 9.4, which is why the numbers in this post come from metrics rather than from a slide. Columnar Mode applies the same treatment to logs and security telemetry, along with analytical data. It’s in technical preview in Elasticsearch 9.5, with GA targeted for 9.7.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

PromQL compiles to the same compute engine that runs ES|QL and inherits its planner and distributed execution, along with its release process. We didn’t build a second engine for this, and there’s no plugin to install.

This post is about how we built it.

Key takeaways:

• One engine: The implementation combines Elasticsearch’s mature distributed planning, storage, and testing infrastructure with its newer compute engine, which provides a columnar execution runtime. This lets PromQL reuse proven Elasticsearch capabilities while executing through a modern, native vectorized pipeline rather than introducing a separate runtime.

• One server: Elasticsearch implements the Prometheus remote write and query APIs directly, so Prometheus-compatible ingest and queries run without any additional plugins.

• Engineered for efficiency: Supporting PromQL required new engine primitives for range-aligned evaluation grids, backward-looking windows, dynamic label grouping, pipeline result reshaping, and compact wide aggregation keys. These primitives allow PromQL queries to execute efficiently end to end, with the relevant semantics implemented directly in the compute engine rather than through external post-processing.

• Compatibility measured in real use: In addition to Prometheus compliance tests, we built a differential-testing and quality-control pipeline over 2,000 PromQL queries collected from public repositories. 

Read more:

• Query Prometheus Metrics in Elasticsearch with PromQL

• Ship Prometheus Metrics to Elasticsearch with Remote Write

• Bringing Fire to Elasticsearch: Adding Native Prometheus APIs

Why run PromQL on Elasticsearch

Many teams already store logs and traces in Elasticsearch while running metrics in Prometheus or another dedicated metrics back end.

Prometheus and its ecosystem are strong and widely adopted, but large deployments can also bring operational sprawl and scaling challenges, along with limited retention. 

So we set out to combine Elastic’s highly optimized time-series database (TSDB) with a best-in-class metrics ecosystem. The result is a smaller observability stack, with fewer systems to operate and metrics storage that scales horizontally and supports long-term retention.

One engine: PromQL and ES|QL share the same compute engine

We made an early architectural decision not to run a separate PromQL engine next to Elasticsearch.

PromQL is instead another front end to the Elasticsearch compute engine.

This puts PromQL in the normal Elasticsearch development lifecycle. It uses the same planner, distributed execution engine, testing infrastructure, and release process as Elasticsearch itself.

To learn more about Elasticsearch’s query engine, check our blog.

Like ES|QL’s time series queries that use the TS source command, PromQL is translated into a highly optimized query plan and executed across the cluster. Nodes process columnar batches through vectorized operators, while partial results move through exchanges until the final result is assembled. 

This also means that PromQL and ES|QL operate over the same execution engine and time-series data. ES|QL can additionally extend a PromQL computation with post-processing that PromQL doesn’t support, such as lookup joins and inline aggregations.

For example, assume Prometheus request counters are stored in metrics-* and keyed by the service label. A lookup index, service_registry, maps each instance to its owning team and environment and to its service tier:

This architecture requires the execution engine to support PromQL semantics natively and efficiently rather than ES|QL syntax sugar. The following sections describe the changes and new execution primitives we introduced to achieve that.

One server: Prometheus remote write and HTTP API built into Elasticsearch

Query execution is only half of the story. The Prometheus ecosystem also expects familiar ingest and query APIs.

Prometheus protocols are the de facto standard for everything metrics in almost every team’s observability stack. So we built the HTTP API directly in the Elasticsearch server, which eliminated the need for a third component and tightened integration stability and performance.

On the ingest side, we added an endpoint for the Prometheus remote write protocol. It accepts Snappy-compressed Protocol Buffer messages, maps labels to TSDS dimensions, maps the metric name/value into metric fields, infers counter versus gauge mappings, and writes directly into TSDS. The built-in template is dynamic, so users don’t have to predeclare every Prometheus label or metric.

On the query side, Elasticsearch exposes Prometheus query APIs. A request enters through the Prometheus endpoint and is executed in the compute engine.

Running PromQL efficiently in a columnar engine

Sharing an execution engine doesn’t mean treating PromQL as syntax sugar over ES|QL. PromQL has different time, grouping, and response semantics, as well as workload characteristics that matter at scale. Supporting it efficiently required extending the compute engine rather than compensating in the API layer.

PromQL time grids: Aligning evaluation steps with TSTEP

Time-series query engines are optimized for grouping over time.

Elasticsearch normally groups timestamps with TBUCKET(...), which truncates each timestamp to a fixed interval boundary. Truncation is cheap and produces deterministic bucket boundaries. It also makes intermediate results easier to reuse.

Prometheus defines evaluation points differently. For a range query, timestamps are laid out as fixed steps anchored to the query range, rather than derived by truncating each sample timestamp. Two queries with the same step but different range boundaries can therefore produce different evaluation grids.

To preserve these semantics, we introduced TSTEP(...), which derives its grouping grid from the query range and step,  rather than truncating timestamps to globally aligned boundaries.

PromQL uses TSTEP(...) internally, preserving Prometheus timestamp semantics while still lowering the operation to a native Elasticsearch execution primitive.

Some might see this as a simple problem, but the nuances matter. 

Take, for example, a query that finds a 5m rolling average of a metric: 

At evaluation time T, the result represents the average over the preceding five-minute range: 

(T - 5m, T]

When the query is executed with a five-minute step, each output value is labeled with the upper end of its corresponding five-minute window.

Elasticsearch previously lacked this semantic and supported only forward-looking window aggregation functions:

We rewrote the window-evaluation path so that both ES|QL and PromQL use a common backward-looking windowing implementation:

Together, TSTEP(...) and backward-looking windows preserve the two time semantics that matter for PromQL range evaluation.

Dynamic label grouping: How PromQL without() resolves at runtime

Time grids determine when a PromQL expression is evaluated. Aggregation determines which input series are combined and which labels identify each output series.

For most analytical query engines, that identity is known when the query is planned. The planner can allocate grouping columns and choose an aggregation strategy. It also carries a fixed key through the execution pipeline.

That’s how ES|QL works:

The output series are grouped by an explicit key (cluster, namespace).

PromQL can express the same operation in the opposite direction:

Now we know which dimensions not to use. We don’t necessarily know the full grouping key until the query is executed. This is a small language difference with significant execution consequences. 

One possible implementation is to discover every label used by the metric, subtract instance and pod, and rewrite the expression into an ordinary by(...) aggregation. That adds a discovery phase before planning. It also becomes inefficient for high-dimensional metrics where only a subset of all possible dimensions may have useful value in a particular series. Most queries need only a small subset of the available dimensions, so carrying the entire dimension universe as an aggregation key wastes memory and adds bookkeeping overhead.

We instead extended the time-series execution path with dynamic grouping columns. The engine loads dimensions as the series are read and applies the exclusions per time series. This avoids making the grouping schema a prerequisite for planning and avoids carrying a large sparse set of grouping columns through aggregation.

Dimension packing: Keeping wide PromQL grouping keys cheap

The idiomatic way of writing PromQL aggregations involves heavy use of without(...) over by(...):

Excluding labels rather than explicitly listing them makes dashboards and alerts resilient to schema evolution. If a new label is added to the metric, the query continues to preserve it unless it’s explicitly excluded.

The consequence for the engine is that effective grouping keys can be wide. Many of those labels often have low cardinality, yet each still participates in every aggregation stage.

In a columnar engine, each grouping column is normally represented as a separate vector. Ten grouping labels therefore mean 10 vectors flowing through every aggregation operator:

Dimension fields are declared in the index mapping; the planner knows the schema up front, and grouping keys stay narrow and predictable.

In the columnar engine, each grouping label is carried as a separate vector or block. A key with 10 labels therefore requires 10 vectors to be read, hashed, compared, and retained by aggregation operators. As key width grows, so does the amount of data and bookkeeping that must move through the pipeline:

To avoid paying that per-column cost for every additional label, we introduced dimension packing. Before aggregation begins, the engine encodes the full grouping key into a single compact representation. Hash and comparison operations run on the packed key rather than on each block independently:

Packing lets hashing and comparison operate on a single compact key rather than an increasing number of grouping blocks, making aggregation overhead less sensitive to key width. Because the engine is shared, ES|QL time-series queries will benefit from this optimization as well.

Building the Prometheus HTTP API response inside the pipeline

Unlike Elasticsearch's ES|QL column-oriented response format, the Prometheus response is row-oriented.  The Prometheus API returns one result row per time series, with its samples represented as timestamp-value pairs. 

To support a compatible API layer, we had to regroup in the HTTP layer converting the columnar results into boxed row objects and accumulate them in map- and list-based structures until the complete Prometheus response could be produced. 

We replaced this with the TimeSeriesCollapse compute operator. It groups rows by series and aligns samples to the query’s fixed step grid. It emits the reshaped result as ordinary columnar pages containing one row per series, with aligned multi-valued timestamp and value blocks. And it preserves compact, vectorized block representation throughout the pipeline: 

The HTTP layer can now serialize those blocks directly, avoiding the maps, lists, boxed objects, and associated allocations required by the earlier implementation.

Testing PromQL compatibility against 2,000 real queries

Prometheus compliance tests were our starting point.

Even though they gave us a strong baseline, they didn’t tell us how frequently individual PromQL features appear in real workloads. To complement that baseline, we built a second test corpus from over 2,000 PromQL queries collected from public repositories. 

We then classified those queries by the language features and expression patterns they exercise:

For each compatible query shape, we run the same query against Elasticsearch and Prometheus and compare the results. 

In addition to that, we actively rely on fuzz testing, which catches issues that unit tests alone are unlikely to expose, including differences in timestamp alignment, label retention, aggregation behavior, range-vector evaluation, and response encoding.

Which PromQL functions and APIs are supported in 9.5

Since 9.4 (technical preview), PromQL support in Elasticsearch has expanded substantially. In Elasticsearch 9.5, both the PROMQL command in ES|QL and the Prometheus HTTP APIs are generally available (GA), with more than 80% of the PromQL workflows in our real-world corpus now running without modification:

The main additions since technical preview are:

Feature

Example

Status

Prometheus remote write ingest

POST /_prometheus/api/v1/write

GA in 9.5

Range queries

/api/v1/query_range

GA in 9.5

Instant queries

/api/v1/query

GA in 9.5

Metric metadata and build info

/api/v1/metadata, /api/v1/status/buildinfo

GA in 9.5

Native histogram functions

histogram_quantile, histogram_count, histogram_sum

GA in 9.5

Per-selector offset modifiers

[5m] offset 1h

GA in 9.5

Top-level or operator

rate(a[5m]) or rate(b[5m])

GA in 9.5, up to eight operands

Prometheus remote write ingest

Elasticsearch accepts Prometheus remote write (v1) messages directly:

Snappy-compressed Protocol Buffer messages are decoded, and labels are mapped to TSDS dimensions. Metric names and values are written into the time-series index. The built-in template is dynamic, so users don’t have to predeclare every Prometheus label or metric.

Range and instant queries through the Prometheus HTTP API

Both range and instant query endpoints are available:

Range queries return matrices evaluated over a time window, and instant queries return vectors evaluated at a single timestamp. These endpoints can be used by Kibana, Grafana, or Prometheus-compatible alerting tools, and custom dashboards.

Metric metadata and build info endpoints

Elasticsearch exposes metadata about available metrics and a build-info endpoint:

The metadata endpoint returns metric types and help text, and the build-info endpoint returns the Prometheus-compatible server version. Grafana and other tools use these endpoints for feature detection and UI behavior.

Native histogram functions: histogram_quantile, count, and sum

Elasticsearch supports the main PromQL operations over native histograms:

Native histograms adapt their bucket layout to the data, providing useful precision across a wide value range without requiring users to configure every bucket boundary in advance. Classic histograms continue to work alongside native histograms.

Per-selector offset modifiers in PromQL

Offset modifiers shift a selector’s time window backward:

This returns the request rate from one hour earlier. Per-selector offsets are commonly used to compare current traffic, latency, or resource usage with an earlier baseline, such as the same period one week ago.

Top-level or operator in PromQL

Elasticsearch supports the top-level PromQL or operator:

In PromQL, or isn’t a Boolean operation. It performs a union between two sets of time series. Results from the left side are retained; a series from the right side is added only when its label set doesn’t match a series already returned by the left side. This is useful during migrations where the same logical metric may exist under an old and a new name.

The implementation follows Prometheus’s left-side precedence rules and preserves the __name__ label. Top-level chains of up to eight operands are supported.

PromQL features not yet supported in Elasticsearch

GA doesn’t mean complete PromQL compatibility. Some less common and more complex parts of PromQL remain unsupported. These gaps now define the next phase of the work: 

Feature

Example

Status

Advanced vector matching

on(instance) group_left

Planned

Sorting and ranking

topk, bottomk, limitk, sort, sort_desc

Planned

Label manipulation

label_replace, label_join

Planned

Absolute time modifier

@ 1710000000

Planned

Mixed-offset compound expressions

rate(...) - rate(... offset 1h)

Planned

Alerting and target endpoints

/api/v1/alerts, /api/v1/targets

Out of scope

Advanced vector matching with on() and group_left

Some binary operations that require Prometheus vector matching aren’t yet part of GA.

For example, this query divides per-instance request rates by a per-instance capacity metric:

The on(instance) clause specifies which labels identify matching series. group_left permits many request-rate series to match a single per-instance capacity series, while retaining the labels from the higher-cardinality left-hand side.

These expressions are common when joining a detailed metric with metadata or a lower-cardinality capacity metric. Basic binary expressions are supported where applicable, while the remaining vector-matching forms are planned work.

Sorting and ranking: topk, bottomk, and sort

Prometheus sorting and ranking functions are also not yet part of GA:

This returns the 10 services with the highest request rate. Similar queries are widely used in “top offenders” dashboards for traffic, latency, errors, and resource consumption.

The remaining functions include:

Label manipulation with label_replace and label_join

PromQL can construct or rewrite labels during query evaluation. These functions are particularly useful when dashboard variables, naming conventions, or label schemas don’t match exactly:

This creates an environment label from the cluster label.

Another common example combines existing labels into a display-oriented label:

This produces a target label, such as payments/api-7f6d9. label_replace(...) and label_join(...) aren’t yet included in GA.

Advanced time modifiers: The @ modifier and mixed offsets

Several advanced time modifiers and expression forms remain outside the GA scope.

For example, an absolute @ modifier evaluates a selector at a fixed Unix timestamp rather than at the query’s normal evaluation time:

This is useful for comparisons against a fixed historical point.

PromQL also permits expressions in which the two sides use different offsets:

This compares current traffic with traffic one hour earlier. Per-selector offset is available in GA, but not every combination of offsets and compound expressions is part of GA yet.

Prometheus API endpoints not yet implemented

In addition, the Prometheus HTTP API surface isn’t yet fully complete. Notably:

Alerting metadata through:

used by tools that inspect active alert state.

Target discovery through:

used to inspect scrape targets, health, and labels.

These endpoints concern Prometheus server and scrape-target state rather than querying metrics stored in Elasticsearch.

For the full list of limitations, see the PromQL limitations page. 

Try PromQL in Elasticsearch 9.5

To query Prometheus metrics in Elasticsearch 9.5 or Serverless, see the PromQL documentation and the Prometheus HTTP API reference.

However, with the rapid progression of coding agents, performance optimization has become a task we can tackle semiautomatically. Unlike many software engineering challenges, optimizing code offers a cheap and objective verifier. If you ask an AI model to make code faster, there’s a hard number at the end telling you exactly what happened, backed by profiling tools that explain why. This makes performance a perfect candidate for automation, provided you can actually trust the numbers.

If you simply point a coding agent at a benchmark, you typically get low signal-to-noise: wins that fall inside the variance of the environment, or variations caused by thermal throttling rather than better code. To capture optimizations that actually benefit Elasticsearch users, we had to bridge the gap between "checkable in principle" and "checked in practice." We built a highly trustworthy measurement loop: a harness that assumes the agent will be wrong a good fraction of the time but reliably catches and proves it when it’s right and then helps guide it where to look next.

Once the machinery is in place, the results speak for themselves. By letting this harness loose on the codebase, we've already begun uncovering meaningful wins across the stack. In part 2 of this post, we’ll dive into some examples it has found so far, including string conversion inefficiencies in Elasticsearch Query Language (ES|QL), an improvement to our NEON vector dot product implementation, and an upgrade opportunity for the gzip library we were using. In this part, we’ll take a look at the design choices we made and how they relate to the broader topic of effective harness development.

The AI code optimization pipeline architecture

The first step in any software engineering problem is to identify the correct high-level components. We made an architectural choice that turned out to be very helpful for this problem: separate understanding where opportunities exist from the loop making code changes. The agent starts with a real workload but only uses it to mine information about where to seek performance improvements. At this stage, it’s instructed to go broad and consider a range of performance-related signals. Once it has found and classified the hot spots, the agent reads the context of the code around them to understand the optimization opportunities. We use a separate task to condense the ranked list of hot spots into artifacts that a loop can iterate against in minutes: a microbenchmark that we prove exercises the hot path in its real operating regime. Finally, we use a proposer-verifier loop to actually make changes to the codebase to improve performance on the benchmark. This hands off to validation to assess the impact on real workloads at the end. Our CLI (atune) supplies the tools this process needs, and the rest is largely automated by a set of task-specific instructions.

For context, our high-level architecture looks like the following. Pink boxes are the humans, and teal boxes are the agent. There are three task types, one skeleton loop, and one referee.

An exploration task profiles a real workload and produces ranked opportunities; a human promotes one into an exploitation task, which iterates against an approved microbenchmark and commits each accepted experiment; a validation run on the real workload guards the result before a human reviews and opens the PR. Where no benchmark covers the hot path, a benchmark task authors one and a human approves it into a registry. A performance atlas informs every task and accumulates what each one learns.

Why performance optimization suits autonomous agents

Three properties make a task ideally suited for autonomous work, and it's worth being explicit about them because they provide a checklist you can use to evaluate automation candidates. You want:

1. An objective verdict so that the agent can be held to something other than its own opinion.

2. A dense guiding signal so that it knows where to look next instead of guessing.

3. A bounded blast radius so that being wrong is affordable.

Performance gives you all three. Benchmarks provide the verdict and profilers provide the gradient, while a rejected patch costs you wall-clock time rather than correctness. The change gets reverted, and the reason gets recorded. Life goes on. Regarding the first two, we've come to think the gradient matters more than the verdict. ″This got faster″ is binary, whereas a profile hints at what to try next. An agent can generate its next hypothesis conditioned on a rich guiding signal, the richer the better, rather than grinding through a list.

Signals are what the agent gets to see

A useful mental model is that the CLI is the agent's sensory apparatus. That changes how you design each command. Rather than exposing a capability, you design it to return a clear and concise answer to a question about the task at hand, and, when relevant, an explanation the model can reason over, instead of raw data it has to parse and interpret. These are the signals that the agent acts on, and we ended up with the following for our harness:

Signal

Question it answers

Facet-decomposed macro profile

Where in the code does real workload time go, per query type?

Allocation and lock sampling, in the same capture

Is the cost cycles, garbage, or contention?

Cost-composition classification

Is this in scope compute, other product code, GC, JIT tax, or parked threads?

Input-shape instrumentation

What does the workload actually feed this code?

Statistical verdict

Did this change help, at this measured noise floor?

Allocation-rate comparison

Did the new code end up allocating more?

Interpreted disassembly

Why did that result happen?

End-to-end A/B guard, with differential profile attribution

Did anything appear to break, and was it us?

Environment check

Is this machine even fit to measure right now?

Upstream duplicate search

Has somebody already reported or fixed this?

Four of these signals are worth dwelling on, because in each case the tool encodes a judgment that the agent would otherwise have had to keep making by hand.

Facet-decomposition is the clearest example. Blended CPU shares hide breadth: if you profile a mixed query workload, the grouping hash map insert and the percentiles sketch update can both show up as single-digit percentages of the total run time and look comparable. They aren't comparable at all, because the hash map insert is paid for by nearly every aggregation query, while the sketch update is only paid for when somebody asks for percentiles. So the profiler runs each named query facet as its own race, and every opportunity the agent records carries a breadth field (universal, broad, or narrow) and gets ranked by headroom × tractability × breadth. A universal 3% beats a narrow 10%. Putting the ranking function in the tool prevents it from having to be rediscovered on every run.

The cost-composition classification works as a router. Rather than handing the agent a flat top-N list of frame names, it buckets every sampled stack into ″in scope compute,″ ″other product compute,″ ″GC,″ ″JIT and safepoint overhead,″ ″off CPU waiting,″ and ″parked threads.″ Each of those buckets implies a different kind of investigation. If GC is above about 15%, the real target is allocation rate, and the CPU top frames will actively mislead you, because they show where objects were collected rather than where they were created. If JIT and safepoint overhead is above about 25%, you're looking at a ceiling rather than an opportunity, since no in-scope code change will move it. If threads are parked and core utilization is low, this is a concurrency problem and CPU flame graphs are the wrong instrument entirely. We wrote that mapping into the playbook as a table, so the model (even a cheap one) reads a profile the way that an experienced engineer would, rather than reaching straight for the top frame. The classification is a prior for forming a hypothesis, though, not a substitute for evidence, so the agent still has to cite specific frames when it proposes an experiment.

Interpreted disassembly is a tool we hadn't originally provided, but it most definitely earns its place. It helps answer why, the question that unblocks the next hypothesis. Flame graphs tell you where the time goes; they rarely tell you why a change made things worse. So atune asm runs the benchmark briefly with the JIT told to print the assembly for one hot method, captures both sides of the working-tree diff, reduces each to the final C2 compilation, normalizes the addresses, and diffs them. The diff alone would likely still be 4,000 lines of aarch64, so on top of it sits an interpretation layer: a per-mnemonic delta, a net instruction count, the compilation tier that was actually captured, and a vectorization signal that counts vector register references on each side and raises a warning when they're eliminated, halved, or narrowed from Advanced Vector Extensions (AVX) to Streaming SIMD Extensions (SSE) width. The playbook then maps mnemonic patterns to causes:

Pattern in the diff

Likely cause

b.eq/b.ne up, csel down

New unpredictable branches

Clusters of str/ldr against the stack pointer

The compiler ran out of registers

NEON loads replaced by scalar compares

The vector path degraded

In one experiment, the agent fused two SIMD mask extractions into one, and the benchmark regressed by 26%. The vectorization warning explained it in about 10 seconds. Without that tool, the agent has a dead end and no working model of the machine; with it, it has a corrected model and several new ideas.

The fourth signal is less a single tool than a habit; the instruments check themselves. Core utilization is derived two independent ways, from sample density and from process sampling, so the two can be compared. The classification is rejected if the unclassified bucket exceeds a budget, on the grounds that a breakdown which can't account for its own samples shouldn't be reasoned over. The disassembly capture warns when the compilation it caught isn't the steady-state one. The upstream duplicate search is restricted to read-only commands, and that restriction is enforced by a test that greps the source, so no future edit can quietly reintroduce the ability to file anything. Each of these exists because a tool that can be confidently wrong is worse than a tool that is merely absent.

One small piece of design is worth highlighting as a specific instance of good return practice. atune compare returns 0 for improved, 1 for no change, 2 for regressed, and 3 for error, and the loop branches on this code. That means no parsing and no ambiguity about what the verdict was. Plus, no tokens are spent interpreting prose.

If you take one thing away from our CLI design, it’s a broader design principle. In a general setting, the interesting thing isn't the individual signals we found useful to understand performance; it's that the CLI is capturing and packaging the judgment of an experienced performance engineer into tools that return answers rather than raw data. Structurally imposing good judgment about the problem an agent is tasked with improves outcomes. The right CLI is as much part of that story as the instructions. Furthermore, tokens are saved by tools that return decisions and digests rather than data. That means a comparison verdict instead of raw JMH output; a triage summary sitting on top of a 4,000-line disassembly diff.

From 20-second probes to hours of validation

Building a verifier that you can afford to consult is a separate problem from building one that you can trust. This covers the affordability part. Or, if you like aphorisms, real workloads are where truth lives and where iteration goes to die. A macro profile takes 45 to 60 minutes, and an end-to-end validation run takes hours. But a microbenchmark takes minutes. That's why the exploration-then-exploitation split works; you go broad on the real workload once and then hand off to a microbenchmark that you can iterate against in minutes.

The catch is that the handoff is only sound if the microbenchmark exercises the hot path in a realistic operating regime; that is, with the right cardinality and right data distribution. If you get that wrong, your fast loop spins fast but in the wrong direction. Until we finalized the handoff procedure, we saw cases where the agent accepted changes on a benchmark whose key distributions happened to flatter it, and only the end-to-end run caught the problem.

Handing off from exploration to exploitation

The handoff between the two phases is structured rather than informal. An exploration task's primary deliverable is a set of opportunity records, and each one carries the scope paths that an exploitation task would be allowed to edit, a headroom estimate, a classification (constant factor, structural,

allocation, or concurrency), and the benchmark it would gate on (or an explicit "no benchmark coverage" flag, if none exists). Each also carries a narrow test pattern so that the correctness gate stays cheap. The test pattern field exists because of a specific incident; an exploration task omitted it, and the resulting exploitation task ran a very heavy test suite on every experiment. The fix was to change the upstream artifact rather than add an instruction downstream. This is a pattern we use repeatedly; make and record decisions as early as possible rather than re-derive them each time.

Validating a new benchmark before it can gate anything

Where benchmark coverage is genuinely missing, a dedicated benchmark task authors one, and that new benchmark has to pass validity checks before anything can rely on it. Two of them are mechanical: what fraction of the benchmark’s hot self-time comes from frames that actually appear in the production profile and whether the parameters fall inside the input shapes we measured. The third is a checklist that the agent has to attest item by item. It exists to avoid the JIT getting a simpler world than production.

1. Inputs have to be reshuffled rather than fixed or sorted, so the branch predictor doesn’t get too good.

2. Results have to be consumed, or dead-code elimination deletes the thing that you meant to measure.

3. Inputs must not be compile-time constants, or they get folded away.

4. Call sites have to see roughly the product mix of types, because a monomorphic call site inlines, whereas a megamorphic one doesn’t.

A human then approves it into a hash-pinned registry. Until that happens, it's inert, because task setup refuses any task citing an unapproved benchmark. We think of that approval as the strongest gate in the system, and it's deliberately placed. An approved benchmark can decide accept or reject in every future task, so it's the one place where we ask for a human signature on an artifact rather than on a decision.

The validation ladder

Underneath all of this sits a hierarchy of feedback mechanisms with the property that each rung is cheaper and weaker than the one below it, and the cheap tiers are for rejection only.

Tier

Cost

Role

probe

~20–60 s

Directionally right? Can never accept

codegen capture

~4 min

Why did that happen?

screen

~5–15 min

Cheap statistical filter

confirm

~20–60 min

The accept decision

end-to-end

hours

Regression guard, advisory

The asymmetry between accept and reject is doing real work here. A probe is a single paired fork, whereas the accept predicate requires a full confirm run with a matching calibration record. Agents are very good at telling believable stories; indeed, they're trained on many tasks judged by both LLMs and by humans, so being convincing is actively rewarded. We don't want an agent to be able to promote a cheap signal into a decision by being persuasive about it.

For this task benchmark, wall-clock is a real cost consideration. A confirm run might take an hour. So one has to weigh carefully all the costs involved when choosing the setup. A model that lands one hypothesis in four typically beats a cheaper one landing one in 10 by a margin on end-to-end metrics. The usual instinct to down-spec the model on a long-running loop is exactly backward here. When your loop has an uncertain outcome and significant costs beyond the tokens it consumes, you may well find yourself in the same situation.

How do you know a performance improvement is real?

Is it faster? is a statistical question. So we made the accept predicate code rather than judgment and put it somewhere the agent can't bypass. There are four ideas in the accept decision, and in each case, the alternative we rejected is as informative as the choice we made.

Forks are the statistical unit

Each JMH fork collapses to its mean, and verdicts come from an exact two-sided Mann-Whitney U test at α = 0.05 plus a seeded bootstrap confidence interval over three to five fork means per side. The reason is that iterations within a fork share JIT and heap state and are therefore autocorrelated, so treating them as independent samples manufactures significance out of nothing. We rejected comparing single-run scores, which is pure noise, and iteration-level t-tests, which can be confidently wrong. Seeding the bootstrap means that a rerun reproduces the verdict exactly because the agent needs to be able to tell the difference between a result that changed and a result that was never stable.

Pair candidate and baseline in time

The screen (three forks, optionally over a subset of parameters) exists only to kill bad hypotheses in maybe 10 minutes instead of an hour. The accept decision itself comes from a confirm run that measures candidate and baseline back to back using a stash-flip. In an unstable environment, thermal and background drift only cancels if both sides ran under the same conditions, so an hours-old baseline is really a different experiment.

The noise floor is measured, not assumed

The minimum effect size that a task will accept has to clear the A/A-calibrated coefficient of variation for that specific benchmark on that specific machine, and both the confirm run and the comparison refuse to proceed without a matching calibration record. A 1–2% improvement on a laptop is indistinguishable from noise, and because the floor varies by benchmark and by JDK, any global constant you pick will be too loose somewhere and too tight somewhere else.

The accept rule is composite and deliberately conservative

A parameter combination counts as improved only if p < α, the effect clears the calibrated floor, and the confidence interval excludes zero. Overall acceptance then requires that nothing regressed (not the primary benchmarks and not the guards) and that at least one primary combination improved. The headline figure is the geometric mean of the per-combination speedup ratios, which is always positive and composes across experiments, so a task's cumulative improvement is a meaningful number rather than a sum of incomparable percentages.

What must not get slower

Guards deserve a note of their own, because they answer a different question to the primary benchmarks; not Did this get faster? but What must not get slower while it does? The task definition lists them separately for that reason, and they're typically the operations and the input regimes that the change isn't aimed at. If you're optimizing insert throughput on a hash table, iteration is a guard and so is a collision-heavy key distribution. A change that improves the common case by weakening the hash function will look good on uniformly distributed keys while catastrophically degrading more adversarial inputs. We know that because it happened; the collision distribution was missing from the matrix that accepted one of our early experiments, and the task now carries a comment telling future readers never to drop it again for a hash-quality-sensitive scope. While guards cost wall-clock on every confirm, they also surface edge-case regressions, and omitting them can be much more costly in the long run.

Runtime isn't the only thing worth guarding, either. The confirm run also captures normalized allocation rate on both sides and flags any change that buys speed with more than about 15% extra garbage. That one is advisory rather than blocking, because sometimes the trade is the right one. However, it's the kind of regression a purely time-based accept rule would happily wave through but might raise a red flag to an experienced performance engineer with better understanding of the calling context.

The end-to-end gate is one-sided and default open

The end-to-end gate is a different statistical problem: small n, high noise, and a very strong prior that we should accept based on our microbenchmark results. Our first design treated accept and reject on an equal footing, and it produced multiple clearly spurious rejections, so the redesign is one-sided and default open. An operation is flagged only when the median regression exceeds the threshold and every candidate repetition is slower than every base repetition. That full-separation criterion is the nonparametric one-sided test at this sample size, and it's robust to the single outlier repetition that would occasionally fool us otherwise. A flag then also has to be corroborated against a differential CPU flame graph, where only a rise in the task's own in-scope CPU share counts as real. Near misses get reported for transparency but don't trigger triage, and nothing is ever auto-rejected; a flag is a request for human attention rather than a verdict. We have a final backstop which is the large suite of performance tests that we already run against Elasticsearch on a daily basis.

The one-sided gate lesson generalizes beyond benchmarking. A noisy gate should be one-sided and default open where there is strong prior reason to accept. A symmetric threshold on a noisy signal doesn't just cost you real wins, it teaches the loop to distrust its own instruments, and that’s a much more expensive failure.

Exploration and exploitation need different permissions

Exploration and exploitation might look like two phases of one activity, but they have different inputs (a macro workload versus a pinned scope) and different outputs (ranked opportunities versus commits). They also have different failure modes, which means they want different permissions. We made the split a first-class property of a task, which lets us enforce it; an exploration task literally cannot commit. The baselining, comparison, checkpointing, and validation CLI all refuse exploration tasks, benchmarking allows probes only, and every probe diff is always reverted. A broad, speculative survey is safe because nothing it does can edit the code.

A nice ancillary benefit is prompt focus. Each type reads one playbook, in full, with the others explicitly not loaded. If you try to write a single document covering both "find where the headroom is" and "land a validated win inside this scope", you get something that does neither well, because the instructions for good exploration (follow the profile, widen the net, a broad survey is preferred) are close to the opposite of the instructions for good exploitation (one hypothesis at a time, minimal diff, never widen scope).

AI agent memory: Journals, knowledge bases, and postmortems

Sessions are ephemeral, but what you can learn from them isn't, so the harness accumulates three durable assets, plus one disposable view derived from them. These are a journal of the code changes we’ve tried, a knowledge base of how the code performs, postmortems of when the harness failed and, because sessions can be stopped and resumed, a session summary. What makes them work together is a clean ownership rule about which kind of fact goes where.

The journal records what we tried and measured. It's append-only, one file per task, and one record per experiment, and it's written before the code is edited. Rejections carry a forward-looking note in the form "do not retry X because Y", which is probably the highest value line, because it's what stops the next session re-deriving a dead end. Records also carry the environment and the driving model, which means that hypothesis hit rates are comparable across models.

The knowledge base records how the code works and how it performs. It's an indexed collection of per-area summaries, each stamped with the commit it was written against, and every playbook ends with an upkeep step that appends whatever durable facts the run turned up. Because the performance characteristics of the JDK also change from time to time, for example, a new Vector API might implement vector masking more efficiently on AArch64, findings from profile data are also tagged with the JVM version they apply to.

The postmortems record mistakes that the agent has made in the past, and they're indexed by symptom rather than by date. The question a session actually has when a number looks wrong is Have I seen this shape of wrongness before?, and a chronological list doesn't answer it. So the rows read like "validation fails on operations structurally unrelated to your diff", or "screen reports no matching combinations".

Keeping the journal and the knowledge base distinct sounds pedantic but isn't because without the rule, both of them turn into a diary that has a tendency to bloat the context window or miss critical information in context.

The disposable state is a session handoff, and our advice is to never write it by hand and to avoid asking a model for a session summary, if possible. Our harness regenerates a one-page digest mechanically from the journal, so a resumed session doesn’t spend time and tokens reconstructing where the task had got to. Because it’s derived rather than authored, it can’t drift from the record in the way hand-maintained content does. That’s also why it isn’t part of the audit trail; because it’s cheap to regenerate it from the journal.

Mechanical session handoffs point to two lessons, and they turn out to be the same one. Every time a person appears in the loop, it’s a source of friction and an opportunity for error. And every time you reach for a model, ask whether code can do the same job. This seems like an odd thing to advocate in a project whose primary premise is delegating to a model, but the habit is easy to fall into once you have one to hand. Judgment is expensive, wherever it happens to sit, so the person and model both have to earn their place on merit.

Two further things are critical for durable agent memory. The first is that knowledge rots, so you have to lint it. Elasticsearch's main branch moves daily, which means the knowledge base's file citations go stale, so one linter checks them. Another checks that every command, flag, and path cited in the agent-facing docs actually exists, that every postmortem is linked from the index, and that every task type has a playbook, and it runs as part of the test suite. Treating prose written for an agent as a testable artifact is the reason it stays true. The second is that loading discipline is half of memory. The instruction is to load the index and then the one or two summaries matching the task's scope; never to bulk-load the rest. Memory you can't afford to read isn't memory.

Coding agent guardrails: Containment, scope, and stop conditions

Elasticsearch is millions of lines of code, and an unscoped "make it faster" run against a codebase that size is unreviewable and unfalsifiable. It’s also expensive. The converse is that a verifier only protects what it can see, so we have to apply the same boundary to what it can change. The outcome is a task that fixes its goal, allowed paths, benchmarks, thresholds, and stop conditions before any code is edited, and none of those are things the agent may change during a run.

The human owns the safety envelope, defining scope, thresholds, and stop conditions before any code is edited, and owning every handoff that crosses a trust boundary. The agent session owns judgment: reading profiles, forming one hypothesis at a time, editing inside scope. The atune CLI owns mechanical enforcement: scope checks, correctness gates, statistics, calibration, stop conditions, environment checks, and the signals. Persisted state is the record: a worktree pinned at a base commit, an append-only journal, and generated reports.

Containment happens in two layers. The first layer is coarse and task-agnostic; it’s a single static permissions file lets the session edit the per-task worktree (which includes a local branch of Elasticsearch, all journal entries, and CLI artifacts) and the knowledge base, and it denies the pristine Elasticsearch clone, the task definitions, the harness config, and the audit trail. The second is precise and per-task: a git-level scope check that runs over tracked and untracked files before any build, so an out-of-scope edit is rejected before it can be benchmarked or committed.

Two containment layers have a nice corollary. Widening the first layer to the whole worktree costs nothing, because nothing out of scope survives the second one. Coarse containment plus precise scope beats trying to make a single mechanism do both jobs, which is what we tried first and which left us with a permissions file that needed editing for every new task.

Stop conditions are mechanical. There's a maximum number of experiments and of consecutive rejections, along with a cumulative improvement target, and proposing a new experiment is refused once one of them fires. A human can override; the agent can't. That asymmetry is what makes the gates independent of which model happens to be driving.

Tests are add-only. New test files ship with the checkpoint, and modifying or deleting an existing test is blocked by the scope check. This closes the single most tempting shortcut in the entire problem space by construction rather than by instruction, which seems like the right way to handle any shortcut you'd otherwise have to keep asking an agent not to take.

The harness lives outside the code it optimizes. The Elasticsearch clone is a separate, gitignored directory, and each task gets a worktree pinned at a base commit. The payoffs compound; the subject repo stays pristine and upstream mergeable with nothing related to the harness leaking into a PR, and the audit trail is versioned independently of a codebase that moves daily. Also, tasks are isolated from each other and from any developer checkout. Targeting a newer Elasticsearch means creating a new task rather than repointing an old one, because that task's numbers are tied to its base. It also means that the harness is retargetable in principle, since the Elasticsearch-specific parts are configuration, knowledge base, and benchmark registry rather than architecture.

The decisions the agent never makes

The rule we settled on is that the agent runs the loops and a human owns every step that crosses a trust boundary. That is creating work, blessing a measurement instrument, publishing a branch, or acting outside the repo. None of those are in the agent's allowlist, and each has a reason worth stating:

• Humans have to sign off on the task because the thing being constrained can't set its own limits.

• Calibrating the noise floor needs to be done once per benchmark, per machine, and by default is measured rather than assumed. However, it's rather expensive and we allow a human to override if they know the environment well.

• Deciding what to optimize next by promoting an opportunity is a judgment and a new scope.

• Since benchmarks go on to gate other tasks, we consider reviewing this artifact part of the correctness safety net.

• We leave outward-facing actions, such as pushing a branch or filing an issue, to a human until we're confident in the process.

• We allow actions to be forced, but the override has to sit outside the thing being overridden.

How the human actions get surfaced in the workflow matters. The generated report and the session handoff both print the human actions currently due, at the moment they become due, rather than leaving them to be inferred from the playbook.

We're deliberately not taking a position on how permanent the manual processes are. The right amount of supervision for a new technology is an empirical question, and we'd rather measure it than argue about it. We’ve started with a relatively high degree of supervision because that's the cheap direction in which to be wrong (a gate you never needed is easier to remove than a regression you shipped) and because the harness makes the question answerable. Every gate is a named, logged transition, so over time we can see which of them ever changed an outcome and which only ever cost friction. In summary, measure first, and then refine.

Building the harness is the same kind of loop

A lot of the harness design didn't fall out of an initial design document. The signal set, the ranking function, the shape of a task, and the exact wording of a playbook rule each came from watching a run go wrong. If there's one piece of advice here that generalizes, it's to use the thing before it's ready and to instrument your own disappointment.

The clearest example is a rule we now call distrust surprising results. A validation run reported that every operation had regressed, the worst of them by 14.8%. It was wrong twice over. A target operation pattern had overmatched a completely different code path, and a stale output directory from an earlier run was being read alongside the new one. Offered a coherent story, the agent took it and reverted a change that was actually good.

What went into the playbook after that incident is not "be careful." It's a three-step check to run before acting on a surprising verdict:

1. Trace the code path, and confirm that the thing which moved can even reach your diff.

2. Read the raw per-repetition data rather than the summary, and recompute one headline number by hand.

3. Compare the report's shape against a known good run, because a structurally different report implicates the pipeline rather than the code.

Alongside that, there’s another important rule, which is if the agent concludes that the harness is buggy, it must not fix it mid-run, because a mid-run harness change makes every result in that run incomparable. It should journal the evidence and stop.

Improving the harness is itself a loop worth describing. Asking the model to review its own transcripts and the harness documents, and to propose the rule itself, usually works well. It's good at spotting where its own instructions were ambiguous, in a way that's hard to reproduce by rereading the instructions yourself. What makes that output useful is having somewhere for it to land: a terse rule in the playbook, the narrative in a dated postmortem, a symptom keyed index row, and a linter that keeps the citations honest.

Restraint turns out to be part of the same discipline. The design document carries an explicit list of extension points that we've deliberately not built, because the need for them is still speculative. That's the same "don't guess, wait for evidence" rule we impose on the optimization loop, applied to ourselves.

How this applies beyond performance optimization

A few of these themes aren't specific to performance work or to Elasticsearch.

Verifiable work is the current frontier. The same insight drives reinforcement learning with verifiable rewards, and it’s what AlphaEvolve is built around. The tasks agents are consistently good at are the ones that come with a cheap oracle (tests, compilers, benchmarks), and the interesting move isn't finding more such domains but manufacturing oracles for domains that lack them. Performance is an instructive case precisely because the oracle is, in some senses, obvious, and yet building it still took most of the engineering.

The referee pattern generalizes, too. Separating a fallible optimizer from mechanical enforcement is the same shape as sandboxed execution and policy engines: judgment in the model, invariants in code. The practical consequence is that the system's safety doesn't depend on which model drives it. A weaker model wastes benchmark time, but it can't corrupt the code or accept a bogus win.

Goodhart is a standing adversary for any optimization task that uses an agent, and it has a formal treatment worth reading. An agent optimizes exactly what you tell it to, so the two-tier benchmark structure, add-only tests, benchmark approval registry, adversarial guard distributions, and a marker file that makes timing an instrumented build mechanically impossible are all one design theme wearing different clothes.

Tools are context engineering. The consensus is drifting away from "expose everything" and toward a few well-shaped tools that return digests: the principles of progressive disclosure, self-documenting interfaces, and verdicts rather than payloads.

Memory is becoming architecture. A rules file, per-task playbooks, a durable knowledge base, and an append-only journal form a hierarchy with different lifetimes, owners, and loading rules, and the hard part is eviction and staleness rather than storage; hence, the linters.

Finally, human-in-the-loop is a dial rather than a switch, so where it should sit is something to measure per domain rather than assert.

What's in part 2 of this post

The harness design is a set of hypotheses about what autonomous performance work needs, and the harness was built so that we could test them. Part 2 is that test: the first four PRs we raised using it, their gains, how many hypotheses it took to get each one, which gates actually caught something, and where the harness got in its own way. That includes the changes which didn't survive end-to-end validation, since as usual, the rejections are as informative.

Why snapshots pin primary shards in stateful Elasticsearch

In traditional stateful Elasticsearch, snapshots lock primary shards to their active nodes, completely preventing relocation. That works fine when cluster topology is stable and nodes stay online between maintenance windows, but stateless Elasticsearch works differently. Index data lives in an external object store, with local disk as a cache, and the cluster scales automatically based on CPU, memory, and data size, both vertically (upsizing nodes) and horizontally (adding nodes).

During vertical scale-up, existing nodes must vacate all shards and shut down before new hardware takes over. Since Elasticsearch version 8.13, shard snapshots can pause during node shutdowns and resume after relocation, so a long-running snapshot doesn't block an infrastructure update.

Horizontal scale-out is a different story: No nodes shut down, so pause logic never triggers. New nodes sit idle while existing nodes finish their snapshots, and as soon as a snapshot is queued, the primary shard is pinned to its node, significantly delaying relocation.

Clusters typically scale out because they're already under heavy load. Blocking shard relocations at that moment limits the cluster's ability to reduce pressure, which shows up as degraded indexing throughput and higher latency. The resource imbalance can also trigger unexpected autoscaling behavior. And even when overall topology stays the same, shard locking disrupts hotspot mitigation and workload distribution. These failures used to surface as server-side warnings: "undesired allocation due to snapshot."

Stateful Elasticsearch

Stateless Elasticsearch

Snapshot reads from

Local shard data on the node holding the primary

The object store, using file locations recorded in the commit

Primary shard during snapshot

Pinned to its node until the snapshot completes

Free to relocate at any time

Effect on horizontal scale-out

New nodes wait for in-flight snapshots before taking shards

New nodes take shards immediately, regardless of snapshot state

How stateless snapshots read directly from the object store

A shard snapshot pins primary shards because it needs to read local shard data. In stateless Elasticsearch, that data already lives in the object store, so reading from local disk is unnecessary. Letting snapshots read directly from the object store removes the requirement to lock primary shards. They can relocate freely, and backup is decoupled from cluster balancing.

Stateless commits include location information for each data file in the object store, so snapshots can read and stream directly to the snapshot repository (a separate object store bucket). In the future, we plan to look at server-side ranged copies, which object stores support natively, to skip the local copy step entirely.

Tracking commits when shards relocate mid-snapshot

A snapshot is bound to a specific commit point that determines which files to back up, and those files must remain accessible for the full duration of the operation. In stateful clusters, this is simple: The snapshotting node and the data node are the same, so the commit is managed locally and held until completion.

In a stateless model, the snapshotting node and the data node can be entirely separate, or they can diverge if a shard relocates mid-snapshot. To handle this, we added a transport action that acquires commits on remote data nodes over the network. The data node tracks which commit belongs to which snapshot and releases it once cluster state signals completion.

There's a wrinkle during relocation. A stationary shard relies on its commit point to preserve files. A relocating shard must release its commit so its local store can close cleanly. To keep files accessible through that transition, a newly recovered primary temporarily preserves all existing data files in the object store until notified of snapshot completion via cluster state. This handles both graceful relocations and ungraceful recovery from node or engine failures.

No more allocation delays and improved cache stats

After stateless snapshots shipped, the server-side "undesired allocation due to snapshot" warnings stopped. The chart below shows the before and after, with the release marked by the red arrow. Hotspot mitigation became more responsive because shard relocations no longer had to wait for backup operations.

Cache use is also improved. Snapshots that bypass local shard data stop competing with indexing for cache space. After the release (also marked in the chart), we observed the following two positive changes in cache metrics:

1. The median cache population throughput, defined as bytes per second for filling the local disk cache from the object store, increased about 50%.

2. Cache misses, where data must be retrieved from the object store to fill local disk cache, have dropped more than 60%. 

What comes after stateless snapshots

Object-store-native architectures are increasingly the standard for cloud-native data systems, and stateless snapshots are a step toward fully exploiting that model across Elasticsearch operations. Backups read from the object store, and shards move freely. Neither process waits on the other. Removing the local shard dependency is a step toward further modularizing the stateless architecture.

How Elasticsearch Shard Balancing Works

Elasticsearch uses a weights-based algorithm to create a Desired Balance, an assignment of shards to data nodes. The Balancer determines the target allocation of shards across a cluster of nodes using four key metrics weighted in a linear algorithm. A total weight is calculated per node, and the shard balancer aims to equalize the total weights across cluster nodes. A final Desired Balance shard allocation is precomputed based on the latest cluster state information, and then the elected master node initiates incremental shard moves to reach the desired shard allocation.

The four metrics are:

• Write Load: the total write threadpool activity per node, using the sum of threadpool indexing activity per data-stream shard.

• Disk Usage: the total disk usage of shards per node, using the sum of disk space used per shard.

• Shard Count: the total number of shards assigned to a node.

• Index Balance (shard anti-affinity): per index, how many shards in the index are assigned to the node.

The total weight of a node is calculated using a linear algorithm that finds the deviation from the node-level cluster average for each individual metric, applies a different weight factor multiplier to each, and then takes the sum of all resultant values. The weight factor multipliers attempt to equalize the relative magnitude of each metric so that metrics with large values do not eclipse metrics with naturally small values. Write load tends to be a small value, related to thread usage, and thus gets multiplied by a relatively larger weight factor of 10; whereas disk usage in bytes is a very large number and therefore gets multiplied by a tiny weight factor of 2e-11.

The following are the cluster settings with default values, representing the different weight factors:

cluster.routing.allocation.balance.shard: 0.45

cluster.routing.allocation.balance.index: 0.55

cluster.routing.allocation.balance.disk_usage: 2e-11

cluster.routing.allocation.balance.write_load: 10.0

The linear algorithm looks something like this:

Shard movements are triggered to ensure that the difference in total node weight across cluster nodes remains below the cluster.routing.allocation.balance.threshold with a default value of 1: whenever the threshold is exceeded, shards are moved from the most heavily weighted nodes to the least heavily weighted nodes until the difference between the most heavily weighted and least heavily weighted node is at or below the threshold. Whenever cluster activity occurs that changes shard allocation (e.g., create/delete index, add/remove node, or the disk usage grows), the Balancer rechecks the weights across nodes and triggers shard rebalancing if the delta between the most and least heavily weighted nodes exceeds the configured threshold. The threshold-based approach attempts to balance the trade-off between keeping the cluster perfectly balanced and minimizing shard movements. Large Elasticsearch deployments that use nodes with greater resources typically benefit from raising the threshold setting: a larger weight delta between nodes reduces shard rebalancing.

Shard movement is also constrained by strict shard assignment rules that prohibit certain node assignments according to cluster and index level settings. Examples include: not assigning copies of the same shard to the same node or host; not allowing further assignment of shards to a node that does not have spare disk space; and excluding node(s) as host for a particular index. More on this below.

How a Balanced Cluster Looks (Based on Weights)

Using the linear algorithm and cluster setting defaults previously described, the following is an example of what the Balancer considers balanced. Notably, it can sometimes allow considerable deviation across nodes in any one particular metric. For simplicity, index balance is not included.

Weight Node1 = 0.45 (5 - 6) + 10 (0.3 - 0.33) + 2e-11 (1e+11 - 8e+10) =   - 0.35

Weight Node2 = 0.45 (7 - 6) + 10 (0.4 - 0.33) + 2e-11 (4e+10 - 8e+10) =   0.35

Weight Node3 = 0.45 (6 - 6) + 10 (0.3 - 0.33) + 2e-11 (1e+11 - 8e+10) =   0.10

Limitations of Weights-Based Shard Allocation

Elasticsearch Serverless deployments are managed by Elastic and could run into many edge cases that, without additional configuration, the weights-based shard allocation handles poorly. In self-managed Elasticsearch deployments, it is possible to work around many of these issues by configuring the cluster to suit the user’s workload. However, Elasticsearch Serverless is configured once and must work across all customer use cases. Issues experienced by some Elasticsearch customers (early adopters of Serverless among them) include:

• Continuous rebalancing background noise in active clusters. This could be because the threshold setting needs tuning or because the cluster is very busy.

• The Balancer’s behavior cannot be tuned in a predictable manner. Adjusting the Balancer settings (individual weight factors) can lead to unpredictable outcomes due to the linear algorithm. For example, decreasing the shard count weight factor relative to the other weight factors can lead to data node OOM events when shard count balancing is deprioritized and too many shards pile up on a single node.

• No explanation of why the Balancer is making shard moves. The linear algorithm is difficult to understand without relevant node metrics.

• The linear algorithm allows a high value in one metric to cancel out a low value in another metric. For example, a node can have a higher than average (across cluster nodes) write load, but counterbalance with a lower than average shard count (or vice versa), and the linear algorithm cancels out the spikes: no shards are moved to address the write load hotspot.

• Index-level hotspots can occur when a disproportionate number of index shards are assigned to the same node, rather than spreading out across nodes, despite the index balance weight in the linear algorithm. Index balance weight can, in some situations, be little compared to the other weight factors. It can also get skewed and counterbalanced by another non-average individual weight in the linear algorithm, as described in a previous bullet.

• No search load balancing.

• Regular indices do not have write load estimate support, leaving some write load hotspots unaddressed. Only data stream indices have write load estimates.

• Write load hotspots can be missed. Write load estimates are only refreshed at rollover time, which can be infrequent in some configurations, causing new load to be ignored for some time. The write load is also the average write load activity over a potentially large window of time between index rollover events, so temporary write load increases can disappear when averaged with inactive write periods.

The above issues persist in some Elasticsearch deployments and require monitoring and workload tuning to manage when they do occur. Shard allocation balancing in Elasticsearch Serverless aims to address these issues and avoid any manual intervention requirements using a new approach that is explained in subsequent sections of this article.

Elasticsearch Serverless Shard Allocation 

Elasticsearch Serverless considers node resources individually: shards are rebalanced away from a node when any resource usage on that node grows to threaten performance, and shard movements to a node are declined when the assignment could threaten that node’s performance.

The Elasticsearch single combined score per node is replaced in Elasticsearch Serverless with independent per-resource decisions:

Elasticsearch Weights-Based Balancing

Elasticsearch Serverless Resource-Aware Deciders

Decision Basis

Single weighted sum across four metrics

Each resource evaluated independently

Metric Interaction

A high value can offset a low one

No offsetting; each decider acts separately

Decision Types

YES / NO

YES / NO / NOT_PREFERRED

Rebalancing Trigger

Weight delta across nodes exceeds threshold

Individually configurable safe limits per resource

Explainability

Can only make an educated guess

Each move traces to a named decider

The Elasticsearch Balancer has three phases, in order of priority, for shard movement decisions. The first phase is to assign unassigned shards. Assignment of unassigned shards is the top priority for data availability reasons. The second phase is to move shards that can no longer remain where they are assigned due to cluster configuration changes. Internally, AllocationDecider implementations enforce cluster settings, like index-level shard allocation filtering, shard allocation awareness, disk usage thresholds, or moving shards off of a node before shutdown. The third phase rebalances shards when the cluster.routing.allocation.balance.threshold is exceeded, using the previously described weights algorithm.

The new Serverless balancing approach adds additional logic to the Balancer’s first and second phases, leveraging the existing AllocationDecider logic, and eliminates the third phase. Previously, each AllocationDecider had simple responses of YES and NO. Now, the decision type of NOT_PREFERRED has been added, along with several new AllocationDecider implementations. An AllocationDecider will return NOT_PREFERRED when it observes that performance might suffer from a shard’s assignment to a particular cluster node. The Serverless Balancer will prefer a node assignment for the shard where all AllocationDecider implementations reply YES.

A NOT_PREFERRED shard allocation may be left uncorrected if all other node assignments return NO or NOT_PREFERRED. Such responses mean that the shard cannot be assigned elsewhere without either violating a cluster/index rule or potentially degrading the performance of another cluster node. Serverless Autoscaling activates before all cluster nodes hotspot: even one unaddressable hotspot leads to a scale-up event. New AllocationDecider implementations have also been added for important finite resources, like available heap memory (further discussion below), using only the original YES and NO decisions: exceeding certain categories of resources can lead to node unavailability.

The individual weight metrics in the Balancer’s linear algorithm have been replaced by resource-aware AllocationDecider implementations, and new AllocationDecider implementations are being built for additional resources: Serverless Search Tier load-balancing improvements are currently in development. Each shard migration will have a clear purpose to address a potential resource usage bottleneck.

Internal stats have shown far fewer shard movements in general, without any noticeable accompanying node performance degradations – one workload showed a 50% reduction in shard movements with the same write throughput. Fewer shard movements has the benefit of: avoiding momentary read/write latencies from warming up local caches; and saving on cloud infrastructure costs moving data between servers.

Serverless IndexBalanceDecider: Avoid Colocation of Index Shards

The IndexBalanceDecider ensures index shard anti-affinity much more strictly than the original weights-based linear algorithm could achieve. Colocation of index shards in excess of the index’s average shards per available node is avoided, except in the case of a strict NO assignment (essentially non-existent right now in Serverless except for shutting down nodes and rolling upgrade incompatible version checks) or NOT_PREFERRED assignment due to temporary node hotspotting.

The IndexBalanceDecider is a very effective means of pre-balancing both write load and search load before user workloads begin to generate load statistics: each index begins life with its shards distributed across as many nodes as possible.

IndexBalanceDecider Results: Even Write Load Distribution Across Cluster Nodes

Write load across data nodes became much more evenly distributed after the IndexBalanceDecider was enabled in the Serverless Production environment. Projects fleet-wide generally show even ingest load (counted in saturated WRITE threadpool threads), combining the release of the IndexBalanceDecider and many other prior improvements:

A reproducible workload demonstrates a clear before and after view of the impact of the new IndexBalanceDecider when an ingest workload was run with and without it enabled:

The IndexBalanceDecider also serves in the Serverless Search Tier to distribute shards of the same index as much as allowed, similarly limited only by the tier’s node count and the number of shards in each index.

Serverless HeapUsageDecider: Assign Shards by Available Heap

The HeapUsageDecider limits shard count on a node based on available heap to hold in-memory shard metadata and run associated write/read operations, removing the dependency on shard count limits per node. The HeapUsageDecider returns a strict YES or NO decision, rather than using the new NOT_PREFERRED decision type, because a data node risks an OOM event if the estimated available heap memory is exceeded.

HeapUsageDecider Results: Reduced Data Node OOMs

Data node OOMs in the serverless index tier decreased significantly as the HeapUsageDecider rolled out to the Serverless production environment.

Index tier OOM errors still occur from time to time in the Serverless Index Tier, though at a much reduced rate, as miscellaneous runaway memory usage edge cases are surfaced. The remaining OOM errors are being progressively resolved as they are identified, through a combination of memory usage improvements in the code, adding component level limits, and updating the internal Elasticsearch Serverless memory model service to more completely account for memory usage.

The HeapUsageDecider is not yet turned on in the Serverless Search Tier, due to the need for additional and different metrics, but that work is in active development.

Serverless WriteLoadDecider: Prevent and Correct Write Load Hotspots

The WriteLoadDecider receives periodically refreshed (every 30 seconds by default) per shard and per node write load stats and uses the data to correct and avoid write load hotspots. The master node retrieves stats directly from each data node’s write threadpool: an Elasticsearch node tracks the total time that its WRITE threadpools is in use, and each individual Elasticsearch shard instance tracks how much time it spent using its node’s WRITE threadpool.

A write load hotspot is identified at the node level. The criteria for a hotspot is the presence of WRITE threadpool queue latency above a configured threshold and sufficiently high, and sustained, WRITE threadpool thread saturation. Once that situation is detected, the Balancer is signaled to select shards to move away from a hotspotting node, until fresh non-hotspotting write load stats are received from the node. The Balancer will do nothing if all nodes are hotspotting at once, expecting the Autoscaler to solve the problem by introducing more, or bigger, data nodes to the cluster.

The WriteLoadDecider uses a heuristic to choose shards to move away from a hotspotting node that aims to minimize ingest disruptions while still effectively reducing a node’s write load. A shard write load threshold is identified on a hotspotting node: the threshold is currently calculated as ½ the ingest load of the hottest shard on that node. Shards that can be moved are then prioritized in the following order:

threshold = ½ * maxWriteLoadShardOnNode

1. Shards with write load in the range [threshold, maxWriteLoadShardOnNode), the shard at or closest to threshold preferred.

2. Shards with write load in the range (threshold, 0], the shard closest to threshold preferred.

3. Shards with write load equal to maxWriteLoadShardOnNode.

4. Shards with zero write load.

The heuristic prefers to avoid disruption to the highest ingest shards and instead chooses middlingly loaded shards. Movement of the hottest shard will cause the most latency disruption; and movement of the coldest shards will be the least effective in resolving a hotspot.

The Balancer limits write load hotspot correction shard moves to one move per hotspotting node per stats refresh period, in order to see the effect of a move in real-time node-level write load, before attempting any further corrections. This was a simple initial design that proved effective. Furthermore, the Balancer will not move a shard whose write load alone is sufficient to meet the node-level hotspot criteria: this would just relocate a hotspot to another data node, not actually resolve the hotspot. The Serverless Autoscaler and Serverless Autosharding components are relied upon to resolve hotspots that reallocation of shards cannot.

The WriteLoadDecider returns NOT_PREFERRED when acceptance of a shard could cause a node to start experiencing WRITE threadpool queue latency and create a hotspot. A shard will still be relocated to a NOT_PREFERRED node, however, and risk some performance degradation, as a better option than, say, risking a data node OOM from keeping a shard on a data node where the HeapUsageDecider returns NO.

WriteLoadDecider Results: Hotspots are Quickly Corrected 

Hotspot stats showed general improvement as the WriteLoadDecider was rolled out to Serverless production, in particular the fleet-wide time to correct a hotspot decreased greatly:

Since these graphs were collected, additional work has been released incrementally to better prevent and correct hotspots, and improvements are still in progress.

Serverless Autoscaling, Autobalancing, and Autosharding

Elasticsearch Serverless relies on both new autobalancing logic and new autoscaling logic. The Serverless Balancer must sufficiently distribute shard resource usage across nodes in order to fully saturate the cluster’s resources. The Serverless Autoscaler will trigger a scale-up event when it receives a report that a certain percentage of the total cluster resources are in use and more resources are needed. The Autoscaler will not scale up the cluster if one node is hotspotting and another node has an excess of available resources because the resources are summed across nodes. Therefore, the Balancer must first do a good job on load distribution, and then the Autoscaler will activate as needed.

Autosharding based on write load is also in progress and coming soon to address shard hotspots. Elasticsearch Serverless projects have a default number of shards per index based on the project type. These defaults generally work, but do not account for all possible workloads. Hotspots can occur when an index has too few shards, as well as too many. Too few index shards leads to the Balancer being unable to further distribute an index’s write load across available data nodes, and then the Autoscaler will not see a problem because the cluster-level resources are not fully consumed. Conversely, indices cannot by default have too many shards, since that could degrade search performance for small indices and potentially strain cluster metadata operations if the total number of shards in a cluster grew too large.

Production Example: 708 TB Data Set, 37 Index Tier Nodes (not counting Search Tier), 4,100 Indices, 30,000 Shards

The following graphs cover a period when the Index Tier, in an Elasticsearch Serverless project, scales up from 10 to 37 indexing nodes and then back down to 10 after a write load spike dissipated.

Graph of the Ingest Load Per Index Node

This graph shows fairly even distribution of load, though a little less even temporarily during scale-up. There are nearly 250 fully saturated write threads at peak load. 

Graph of CPU Saturation Per Index Node

CPU usage remains within safe bounds. Usage is mostly below 60%, except for momentary outliers that reach into the 90% range as write load rises before nodes are added to the cluster.

Graph of WRITE Threadpool Queue Latency Per Node 

When a node’s WRITE threadpool is fully saturated, tasks are placed in the threadpool’s queue. Queuing can happen with few tasks, if active write tasks are long-running, or there may simply be a lot of tasks.

This graph’s time window is zoomed in further than the others. One node reaches 75 seconds of queue latency during the scale-up spike. There are 29 nodes when the queue latency spike occurs at 19h25m, before autoscaling calls for 37 nodes at 19h28m.

Graphs of Total Cluster Ingest per Second, in Documents and MBs

Ingest rate peaks at 190,000 documents / second and 54.40MB / second. The document ingest rate is respectable at 4000-5000 docs/sec per node. The MBs ingest rate, however, is very low in this case: this can happen when indexing operations involve heavy computation. Document ingestion rate can also vary depending on the size of the documents.

What’s Next for Elasticsearch Serverless Balancing

The team is currently working on shard balancing improvements for the Serverless Search Tier, focusing on creating metrics and AllocationDecider implementations for search performance. The team is excited to share these improvements soon!

Why upgrade to Redux Toolkit v2?

RTK v2 was released in late 2023. That's nearly three years of running on a major version behind in one of the most widely used state management libraries in the JavaScript ecosystem. It reflects how hard this upgrade is in a codebase of Kibana's size. A previous attempt tried the big-bang approach and stalled when the real scope became clearer. 

So what does v2 actually bring? It ships alongside Redux core 5.0, React-Redux 9.0, Reselect 5.0, and Redux Thunk 3.0. React-Redux 9.0 requires React 18 and drops the useSyncExternalStore shim that v8 carried for React 16/17. Since Kibana already runs React 18, upgrading sheds legacy compatibility code and keeps Kibana on the actively maintained Redux majors.

RTK v2 also brings genuinely useful new features, including inline selectors in createSlice and opt-in inline async thunks through a customized buildCreateSlice setup, along with a combineSlices API with slice reducer injection for code splitting. That last one is particularly interesting for Kibana's plugin architecture where lazy-loading is the norm.

How Redux is used across the Kibana monorepo

Before diving into the solution, it's worth understanding just how varied Redux usage is across Kibana. A full audit of the codebase (tracked in #239863) revealed several distinct camps:

Pattern

Plugins and packages

What the migration needs

Redux Toolkit v1

Discover, Lens, Synthetics, Security Solution

Full v1 to v2 migration

Plain Redux v4

Canvas, Maps, Index Management, Cross-Cluster Replication

redux-v4 alias only, no RTK migration

Kea

Enterprise Search (150+ files), Content Connectors

react-redux-v7 alias, no RTK migration

redux-saga

Synthetics, Graph, Uptime

Store setup only, saga is version-independent

typescript-fsa

Security Solution data-table package

Out of scope

Types and single imports

Expressions, Monitoring

Alias swap only

Plugins such as Discover, Lens, Synthetics, and Security Solution use RTK v1 APIs,including createSlice, configureStore, createAsyncThunk, and createSelector. These are the ones that actually need the v1 to v2 migration. But even here, complexity varies wildly. Lens uses stand-alone getDefaultMiddleware (removed in v2) and PreloadedState (also removed). Security Solution is the largest consumer at 300+ files, mixing modern RTK with legacy plain Redux patterns.

Canvas, Maps, Index Management, Cross-Cluster Replication, and several others still use plain Redux v4via createStore, combineReducers, applyMiddleware, and connect, which are classic patterns from the pre-RTK era. These don't need RTK migration at all since they're not using it  in the first place, but they do need the redux-v4 alias since the default redux package is now v5.

Enterprise Search and Content Connectors use kea, a Redux abstraction layer with its own logic builders (kea(), useValues, useActions). There are more than 150 files in Enterprise Search alone. RTK migration isn't applicable here, since kea is its own world. But it does depend on react-redux v7 under the hood, which is where the bundler tricks come in.

Synthetics, Graph, and Uptime use redux-saga for side effects. Saga integration is actually independent of the RTK version, but these plugins need their store setup migrated.

The Security Solution data-table package uses typescript-fsa and typescript-fsa-reducers instead of RTK entirely, with its reducer embedded into Security Solution's main store, and isn’t part of the RTK migration at all.

The Expressions plugin only imports shallowEqual from react-redux, and Monitoring only imports types. These just need an alias swap.

Asking every team to migrate simultaneously was a nonstarter. The breaking changes in RTK v2 include stricter type checking and removed APIs, like enableES5() from immer, getDefaultMiddleware and PreloadedState gone entirely, AnyAction replaced by UnknownAction, and behavioral changes in how middleware is configured.

Running Redux Toolkit v1 and v2 side by side

The solution was to flip the typical migration pattern on its head. Instead of keeping the default imports on v1 and introducing v2 under aliases, the default package names (for example, @reduxjs/toolkit, react-redux, and redux) now point to v2. The old versions live under versioned aliases:

• redux-toolkit-v1

• react-redux-v7

• redux-v4

• immer-v9

• reselect-v4

• redux-thunk-v2

This is npm's alias syntax. "react-redux-v7": "npm:react-redux@7.2.8" installs the old version under a different name. Both versions coexist in node_modules without conflicts.

The insight here is that all existing code in this pull request (PR) was moved to v1 aliases. Every import { useSelector } from 'react-redux' became import { useSelector } from 'react-redux-v7'. That's ~1,100 files touched, but the vast majority (~1,000) are mechanical one-liner import swaps. When a team is ready to migrate to v2, they switch back to the default import names. Once all v1 aliases disappear from the codebase, the old packages can be removed entirely.

This avoids the alternative, where v2 imports would end up under nonstandard names permanently, leaving nonstandard imports in the codebase for the long term.

Serving both versions through the bundler

Getting two versions of the same library to coexist at runtime is where things got interesting. Kibana uses kbn-ui-shared-deps-npm to bundle common dependencies as shared webpack externals. This needed to serve both the new v2 packages and the v1 aliases so that both are available at runtime.

Pinning @elastic/charts with yarn resolutions

Then there's @elastic/charts. It depends on RTK v1 internally and can't just be upgraded independently since it's an upstream package. Yarn resolutions pin its nested dependencies to v1 versions:

A NormalModuleReplacementPlugin in the shared deps webpack config detects when an import of immer, @reduxjs/toolkit, redux, react-redux, or reselect originates from within @elastic/charts and redirects resolution to the nested v1 copies. This ensures that @elastic/charts resolves to its compatible v1 dependency set.

Keeping Kea on React Redux v7 with webpack externals

The kea library was another fun case. It declares react-redux as a peer dependency (>= 7), so without special handling its imports resolve to Kibana's default v9 package. The migration keeps Kea consumers on react-redux-v7, so Kea must use that same React context. The fix uses function-based webpack/rspack externals that skip externalizing react-redux when the import comes from node_modules/kea, combined with a NormalModuleReplacementPlugin that rewrites it to react-redux-v7. This ensures that kea uses the v7 React context that matches the <Provider> wrapping its consumers.

Both the webpack (kbn-optimizer) and the rspack (kbn-rspack-optimizer) configs needed these changes, with a shared isKeaReactReduxImport helper extracted to keep the logic consistent.

Using an ESLint rule to prevent cross-version imports

With two versions available, accidental cross-version imports are the biggest risk. A new @kbn/imports/no_redux_toolkit_v2_imports ESLint rule catches any import of the v2 default packages (such as @reduxjs/toolkit, react-redux, or redux, among others) in code that hasn't been migrated yet. It even auto-fixes them to the v1 aliases for file imports and Jest mock paths.

The rule is scoped via an override in .eslintrc.js to the ~36 plugin and package paths currently using v1. When a team migrates, they simply remove their path from the override list. This clean, self-service approach requires no coordination.

Why mixing React Redux v7 and v9 breaks the context

This is worth calling out because it's an easy failure mode to miss during an upgrade. react-redux v9 and v7 create separate React contexts. If a component tree has a v9 <Provider> at the top but a child component calls useSelector from v7 (or vice versa), React-Redux cannot find the matching context. In development, it throws an error explaining that the component must be wrapped in a matching <Provider>; in production, the missing context causes a runtime error when the hook accesses the store.

This means that each plugin needs to be explicitly pinned to one version. Shared packages that use react-redux can only be consumed by code on the same version, since mixing isn't possible. This is a constraint that makes the migration inherently per plugin rather than per file.

Migration batches: What can move independently

The dual-version setup gives every team a clear path forward, and the dependency graph analysis from the tracking issue identified natural migration batches:

• Batch 1: Independent, self-contained stores. Packages like kbn-coloring, transform, timelines, and expandable-flyout have fully internal Redux stores with no types leaking through their public APIs. These can be migrated independently by their owning teams, with minimal risk.

• Batch 2: Coupled packages. Some packages share RTK types across boundaries and must migrate together. The machine learning (ML)/artificial intelligence for IT operations (AIOps) chain is one example: @kbn/ml-response-stream exports a streamSlice (a createSlice return value) that @kbn/aiops-log-rate-analysis embeds directly into its configureStore. Migrating one without the other causes type mismatches between v1 and v2 slice types. Similar coupling exists across the Lens ecosystem. The Lens plugin depends on @kbn/coloring (which has its own RTK store), @kbn/lens-embeddable-utils, and @kbn/lens-common, while itself being consumed by 40+ packages and plugins across chart expressions, visualizations, Maps, Canvas, and observability plugins. Whether Redux types leak through a package's public API determines if it can be migrated independently or needs coordination. kbn-coloring's store is internal to its React components so it's safe to migrate alone, but other coupling points need careful analysis.

• Batch 3+: The big ones. Discover, Security Solution, and Lens each have their own migration timelines. Security Solution's 300+ files and mix of RTK with plain Redux v4 and typescript-fsa make it the largest effort, but the different patterns can be addressed independently. Lens has the trickiest v2 breaking changes around middleware configuration; stand-alone getDefaultMiddleware and PreloadedState are both removed in v2, and it has four custom middleware files with complex typing.

Beyond the batched migrations:

• Deprecated features can stay on v1 aliases. When the feature is removed, the v1 imports disappear through code deletion, without any migration work.

• Plain Redux v4 plugins (Canvas, Maps, and others) are entirely out of scope for RTK migration. They'd benefit from modernization, but that's a separate initiative.

• Kea plugins need react-redux-v7 to react-redux alias updates eventually, but no RTK migration. The longer-term question (whether to keep Kea or migrate to RTK v2) is a separate decision.

• The dual-version approach adds measurable bundle overhead during the transition. It’s a trade-off but is acceptable for the migration period.

Lessons for other large monorepo upgrades

The ESLint rule turned out to be the linchpin. Without automated enforcement, aliased imports would drift back to default names within weeks. With it, the migration state is visible in the paths listed in the override. As of the initial PR, zero files import from @reduxjs/toolkit v2. Every RTK usage goes through the redux-toolkit-v1 alias. That's the starting line.

The preparation work also reached beyond import paths. Jest mocks referencing react-redux needed updating to react-redux-v7, as did Storybook previews, test helpers, and ambient type declarations. Multiple rounds of node scripts/eslint_all_files --no-cache --fix caught the mechanical cases; the remaining cases needed manual fixes.

If you're facing a similar major dependency upgrade in a large monorepo, the pattern of giving the new version the default name and the old version an explicit alias is worth considering. New code naturally uses the current version, while older usage stays visible and trackable until it reaches zero.

In Elasticsearch 9.5, Elasticsearch Query Language (ES|QL) queries no longer fail when a field isn't in the mapping. The new unmapped_fields setting lets queries load values from _source or fill with nulls, so queries keep working even when a backing index changes and a field goes missing, and can use unmapped data without reindexing. Here’s how we built that: the design choices and the edge cases (including a class of fields we nicknamed PUNKs), along with the testing strategies that gave us the confidence to ship it in general availability (GA).

Why ES|QL queries fail when a field is unmapped

You built a visualization using an ES|QL query. You refined it, and the query grew. You’re at 15 chained commands and counting, but it does just the right thing. It works, and your dashboard is useful.

Your query uses an index from a remote cluster, say my-remote:logs-foo. But actually, logs-foo is an alias, and at some point, the remote cluster makes it point to a different backing index. The new index is missing a field that’s used in your query, and your query and visualization break.

Or maybe you have an already fairly large index, and while building ES|QL queries on top of it, you realize that you’d like to use a field in the indexed documents that unfortunately never made it into the index mapping. You could reindex the data, but that would take hours.

ES|QL’s unmapped_fields setting is meant to deal with these types of situations.

If your query looks like this:

and some_field is unmapped, ES|QL’s default behavior is to fail with a verification exception.

You can use the unmapped_fields setting to instead either fill some_field with nulls or read it from the document’s _source, like so:

How ES|QL resolves queries with field caps

Before we jump into the inner workings of unmapped_fields, we have to look into how ES|QL resolves queries regularly. Let’s consider the above query:

We said that if some_field isn’t in the mapping for index, ES|QL will reject the query. How does it make that decision?

How field caps tells ES|QL which fields exist

In a typical schema-on-write fashion, Elasticsearch clusters maintain mappings with their respective indices. As a first step, ES|QL makes an internal request to the field caps endpoint to determine which fields the index has. It then passes the query, together with the field caps response, to the query planner, which consists essentially of the query analyzer (unrelated to analyzers of text fields) and query optimizer. The analyzer makes sense of raw names, like some_field, and notices that they correspond to index fields (or not). If all went well, the query is then passed on to the optimizer, which rewrites the query for efficiency, before it’s handed to the compute engine for execution.

How the analyzer resolves field names in the query plan

Let’s zoom in to the analyzer. The parsed query is represented in a tree structure, and the analyzer partially rewrites it, one command at a time, until it either has resolved all references or not.

For illustration, let’s use a somewhat more complex query and see how the analyzer would resolve it:

The parsed tree is actually a chain here, and it looks something like this:

The analyzer then moves up through the query tree to try and resolve the field names used in every command.

This is a simplified version of how we represent parse trees in tests and when debugging. The bottom of the chain corresponds to the FROM command and contains a list of all mapped fields that we know about, obtained from the field caps endpoint. (The {f} suffix marks an actually mapped field for better distinction later.)

The two EVAL nodes on top of it correspond to the remaining commands, and their fields are still unresolved, expressed by the question mark ? in front of the name. At this point, the analyzer still has to check whether they correspond to existing index fields.

For the EVAL that defines uppercased_mapped, it can see that the previous command outputs mapped_field, so the unresolved ?mapped_field marker can be replaced by a real field reference:

Next, it encounters the topmost EVAL, which defines uppercased_unmapped. The previous tree nodes produce only two fields: [mapped_field, uppercased_mapped]. The reference ?unmapped_field thus has to remain unresolved. We bail here and emit the verification exception to the user.

How unmapped_fields LOAD and NULLIFY work

Adding unmapped fields to the query plan

When using unmapped_fields=”NULLIFY” or ”LOAD”, we do something else; we act as if the field was actually in the index. The analyzer adds unmapped_field to the From node and marks it as unmapped to signal to the compute engine that this has to be read from _source or filled with nulls. Let’s express this with a {u} (for unmapped):

After amending the From, the analyzer can continue trying to resolve the topmost Eval node. It sees that the upstream nodes produce the fields [mapped_field, unmapped_field, uppercased_mapped] and thus unmapped_field can be correctly resolved:

The query plan is now fully resolved and can be passed down the regular optimization-execution pipeline. Other than the actual value extraction mechanism, everything stays the same. Schematically, the workflow looks like this:

Example: enabling unmapped fields with the SET directive

To give an example, let’s fire up a cluster and create an index with non-dynamic mappings.

We can run the example query, above:

This should result in the error message:

Unknown column [unmapped_field], did you mean [mapped_field]?

To make things work, we can prepend SET unmapped_fields=”...”; with LOAD or NULLIFY:

Inspecting the analyzer's rewrite steps

If you want to see what the query analyzer is doing to the parse tree, you can log the query rewrite steps, like so:

This will log a line containing Rule rules.ResolveUnmapped applied with change… You’ll see that unmapped_field is added to the bottom of the parse tree as described above.

Why we have to infer the schema

Of course, this isn’t the only possible method to deal with unmapped fields. Here are some alternatives:

1. We could also scan or probe the documents in index to determine that their _source actually has the unmapped_field.

2. We could disable verifications in the analyzer and make the compute engine blindly pass unmapped fields through individual computation steps.

The first alternative front-loads more work to understand the actual schema of an index and thus generally increases latency. It doesn’t scale to large, highly distributed datasets. The second alternative isn’t viable since it means a large-scale change to how ES|QL’s compute engine is built, because it passes around streams of data with fixed columns from one operator to another.

In contrast, the approach we chose is neatly compatible with ES|QL’s existing optimization pipeline.

The trade-off is that the analyzer has to correctly infer a schema based on the actual index mappings (obtained from the field caps endpoint) and additional fields used inside the query. 

This isn’t always straightforward. There were two main challenges:

1. There are many different query shapes and commands that can be used. The mechanism needs to detect unmapped fields, update the proper FROM command, and pass the new field through the halfway resolved plan correctly in all cases.

2. There are many different mappings we have to deal with, and we specifically need to make our feature work correctly when mappings change over time on top of that.

In the following, we’ll focus on LOAD, although some problems (generally many fewer) also apply to NULLIFY.

Which index to load unmapped fields from for LOOKUP JOIN and FORK

To briefly illustrate the first problem, here are some choices we needed to make:

• Which index do we load from when using lookup joins? This one?

  The unmapped_field cannot be attributed to both indices. We chose index since this is where we expect mappings to change more often than in lookup indices.

• Similarly, how do we deal with subqueries and views or the FORK command? In the following:

  one fork branch triggers loading of an unmapped field. Is it also present in the other fork branch? (Yes, it should be, but it’s not obvious and is specifically not true if the two FORKs are replaced by independent subqueries.)

Two principles to keep queries working

The second problem, diversity of mappings and their evolution over time, is a far bigger driver for complexity. We strove for two basic usability principles:

• Queries that work in the default mode should generally still work when using unmapped_fields=”NULLIFY” and ”LOAD”.

• Queries that work when all fields are mapped should generally still work with NULLIFY and LOAD when a field becomes unmapped and vice versa.

The type of unmapped fields and inadvertent type conflicts

Let’s talk about data types to see where this leads to complexity. First, when using unmapped_fields=”LOAD”, we need to assume a data type for unmapped fields. We chose KEYWORD, which allows us to avoid type conflicts when reading from _source. One document can contain ”unmapped_field”: “foo”, and another can contain ”unmapped_field”: 123.4. It’s fine because we treat both as strings.

However, this is a violation of the second principle when a non-KEYWORD field happens to go unmapped. Consider this query:

If some_field becomes unmapped, we’ll have to assume that the KEYWORD type and the query will fail with a type conflict.

Type conflicts aren’t new and can be dealt with by using explicit casts in the query, like so:

It would be great if ES|QL just inferred a useful type to cast to, but this is something for the future.

Type conflicts with partially unmapped fields, or: making PUNKs well behaved

In addition to fully unmapped fields, partially unmapped fields are everywhere and should also work with LOAD. Let’s look at a query that uses multiple indices.

Let’s say that there are indices index and index_without_some_field, containing just the following documents.

Now let’s consider the query:

and assume that some_field is unmapped in index_without_some_field. This will return:

because ES|QL doesn’t load unmapped fields per default.

Of course, when setting unmapped_fields=”LOAD”, we want to load from _source for index_without_some_field:

As with fully unmapped fields, the case is simple when some_field is mapped as KEYWORD in index. When loading from _source for index_without_some_field,  we treat the field as KEYWORD as well, so there’s no conflict.

What makes a field a PUNK

The case is less clear when some_fieldis partially unmapped and the mapped leg is of a type other than KEYWORD. Such fields caused a lot of trouble until we found the best solution, which makes their acronym quite fitting: partially unmapped non-keyword fields, or PUNKs.

Unfortunately, PUNKs are far from being esoteric. For instance, it’s very natural to filter on a PUNK:

If some_field is mapped as INTEGER in index, the type conflict looks like this:

• Mapped as an INTEGER in index.

• Unmapped in index_without_some_field and thus treated as KEYWORD.

This can again be resolved manually by providing an explicit cast:

But this is far from acceptable. Even queries that work fine without NULLIFY and LOAD typically have some PUNKs; the unmapped leg is simply treated as null then. Both guiding principles are violated if LOAD requires an explicit cast here.

Casting implicitly to the mapped type

The solution is to introduce an implicit cast to the mapped type. In this case, we know that some_field is an INTEGER in index, and thus we treat it essentially as if the user wrote:

This means that queries that work without LOAD keep working. (ES|QL may even give you more data because we load the unmapped leg of PUNKs from _source.) Queries that used to work when a field is fully mapped also keep working when it goes unmapped in some (but not all) of its indices without having to alter the query in any way.

Behavior

Default

NULLIFY

LOAD

Unmapped field in query

Query fails

Query runs

Query runs

Values returned

None

null

Read from _source

Assumed type

n/a

NULL

KEYWORD

Partially unmapped field (PUNK)

Unmapped leg is null

Unmapped leg is null

Cast to the mapped type

Pushdown optimization

Full

Full

Per-node where fully mapped

Don't throw it all away: Keeping ES|QL query optimization with unmapped fields

There's one more thing to get right; that is, to make sure that optimizations still work correctly with LOAD. Consider the previous query:

ES|QL’s optimizer aggressively pushes down such WHERE filters and turns them into Lucene queries, so the compute engine doesn’t perform unnecessary work.

For this query, evaluating the filter in the compute engine would require fetching each and every document from the index; meaning, a full scan, very slow. If some_field was mapped as an INTEGER in both indices, we would instead perform a Lucene query, which looks like this:

The compute engine then doesn’t have to load each document separately and check whether it matches the filter. Documents with some_field <= 10 are never fetched from the Lucene index, which is very efficient at this kind of filtering. Nice.

Why filter pushdown is unsafe for unmapped fields

If some_field is unmapped in index_without_some_field, however, it’s wrong to narrow the documents down using the same Lucene query, as Lucene interprets an unmapped some_field as null and thus no documents from index_without_some_field will ever match. This edge case is easy to miss, and it doesn’t help that there are several flavors of similar pushdowns. For instance, in the query:

the compute engine pushes even the counting to Lucene. Again, this is only correct if some_field is fully mapped.

This means that such optimizations can't apply to unmapped fields. It would be disappointing if a query used hundreds of indices and only one of them happened to not map some_field, causing the whole query to run unoptimized.

How the local optimizer recovers the fast path

Luckily, this problem has a solution, too. ES|QL actually has multiple optimizer runs:

1. First, a preliminary optimizer run on the node handling the _query request.

2. Then, a second, local optimizer run on every node we fan out to because we need to fetch documents from its shards.

The workflow after the initial optimization looks more like this:

If the current node happens to map some_field in all shards, the local optimizer detects this situation and treats some_field like any other fully mapped field, including performing Lucene queries to greatly narrow down the dataset to be processed. In fact, data nodes process LIMIT queries like:

in batches of shards (to avoid loading too much data too eagerly), which includes a full local optimizer run per batch. This makes it even more likely to encounter batches where some_field is fully mapped, allowing ES|QL to run a fast Lucene query.

Is it working now? Testing unmapped_fields across every ES|QL query shape

As we have seen from the optimizer issues above, problems can hide in plain sight, even for very simple queries. Because unmapped_fields=”LOAD” can affect each and every kind of query, the surface area for bugs is essentially all of ES|QL.

Accordingly, getting good test coverage was tricky and challenged us to refine our testing strategies.

Reusing spec tests with unmapped_fields

Conveniently, ES|QL has an extensive corpus of test queries, together with expected result sets; we call them spec tests because they’re written using a simple text specification language, which looks roughly like this:

This lets us create new tests out of the existing ones by introducing slight variations. For instance, any existing test that runs without SET unmapped_fields=”...” should produce the exact same results when run with SET unmapped_fields=”NULLIFY”.

It also helped find major issues early in the development process, especially for NULLIFY. The LOAD setting changes the meaning of queries much more dramatically, limiting the usefulness of this approach. However, ES|QL also uses what we call generative testing; that is, we string together random commands, run the query, and then check whether the server reports a bug. This approach cannot confirm the correctness of results, but it still helped greatly with finding query types that didn’t work properly and resulted in some kind of error. (Property-based tests would be a refinement in the future by running the queries against a reference implementation. This way, correctness of results can also be checked.)

Testing type conflicts across different mappings

In the end, one of the most important testing dimensions was using different indices with various mappings in the same query. (Recall how, above, we had to deal with type conflicts to come up with a solid approach for PUNKs? It doesn’t end there; all kinds of type conflicts are more complex with LOAD.) Since we couldn’t automatically generate correct expected results, ES|QL’s test suite had to grow by adding more than 10,000 lines of CSV spec tests. Fortunately, adding such tests is a well-suited task for an AI agent, which has cut down the effort dramatically. (Of course, the test results were still reviewed by humans.)

All testing strategies together provided us with good confidence for the GA release of unmapped_fields with Elasticsearch 9.5.

Observability workloads are storage-intensive by nature, and the composition of that storage determines both cost and query performance. ES95 introduces adaptive compression: rather than applying the same encoding to every numeric field, it automatically selects the encoding that best matches each field's structure. The result is a 33.6% reduction in total doc-values storage, 19% to 74% reduction on floating-point gauge metrics and a 92% reduction in @timestamp. No configuration or migration required.

Observability data is storage-intensive

Data processing systems are rarely limited by how fast they can compute. They’re limited by how fast they can move bytes: off disk, across the network, and through the memory hierarchy. Compression is how a storage engine trades CPU time for memory bandwidth, spending comparatively cheap CPU cycles so fewer bytes have to travel through the parts of the system that are usually constrained. In a read-heavy system like Elasticsearch, that trade-off pays back every time data is queried, often long after it was written.

Storage size and query performance move together; fewer bytes on disk means fewer bytes to read on every range query, every aggregation and every dashboard load. Compression is not just about saving storage. Every byte that is never written is also a byte that never has to be read.

The right encoding depends on the structure of the values themselves, and the largest wins come from exploiting the structure already present in the data rather than squeezing an opaque stream of bytes. Few workloads expose that structure more clearly than observability metrics.

A single host reports hundreds of metrics every few seconds, including CPU utilization, memory ratios, request latencies, and network throughput. Multiply that by thousands of hosts across weeks of retention, and the bytes accumulate fast. Most of that volume is structured but not uniform: timestamps arrive at near-constant intervals from thousands of concurrent series, counters increase monotonically, while gauges like 23.47 or1.15 are short decimal measurements.

A fixed compression approach cannot adapt to that variety. A timestamp column and a floating-point gauge column compress through fundamentally different techniques, but a codec that applies the same approach to both will necessarily handle one of them poorly. For most of Elasticsearch's time-series codec history, gauges were on the losing end of that trade-off.

The structure the old codec was not built to exploit

Elasticsearch stores time-series numeric values in doc values: a column-oriented structure where all values for the same field sit adjacent on disk. That adjacency makes compression possible: the codec compares consecutive values of the same field, finds patterns, and exploits them.

The time-series codec before ES95 applied the same fixed encoding to every numeric field: delta encoding followed by normalization, GCD (greatest common divisor) reduction, and bit-packing. Each encoding technique activated where it helped and skipped where it would not. For timestamps and integer counters, this approach was remarkably effective. For floating-point gauges, it could find almost nothing to work with.

The reason is how they’re stored. To support range queries, Elasticsearch stores floating-point values as integers that preserve numeric ordering. A change of 0.01 in a CPU percentage reading translates to a jump of trillions in that integer space. The codec sees those large jumps and has no strategy to further reduce their footprint. Storage stays near the original eight bytes per value.

The codec was doing the right thing with the representation it had, but the latter was chosen for querying, not compression, and the goals conflict at the bit level.

The cost of a fixed format

A compression stage for floating-point values was already on the roadmap, so the interesting part wasn’t the algorithm. The obstacle was architectural.

The previous codec baked its compression approach into the storage format. Adding a new encoding meant changing the meaning of existing bytes on disk, which forced a format migration, a rollout that can last weeks or months in large production clusters. Over time, that migration burden constrains codec development itself. The question stops being Is this a good compression idea? and becomes Is it worth another format migration? That rigidity limits the cadence of codec evolution and leads to missed compression improvements.

The right encoding without configuration

ES95 solves this at the architecture level for time-series indices. Each field's encoding is no longer baked into the format. It is selected automatically at write time, based on what the field mapping already declares: the field's name, its data type, and its metric role. Timestamps are encoded differently than counters. Counters are encoded differently than gauges. ES95 encodes all of them, and it chooses the right strategy for each.

Users already tell Elasticsearch everything the codec needs to know. The mapping describes the data; the codec chooses the compression strategy.

Compression strategy is a codec concern, not a user concern.

The alternative would have been to expose per-field encoding selection as a configuration parameter, letting users opt in to better compression for specific fields. That would shift the burden of knowing which encoding fits which data type onto those least equipped to make that call and would guarantee that most deployments never see the benefit. ES95 keeps that decision inside the codec, where it belongs. This matters most in managed and serverless deployments, where users expect the system to automatically make optimal storage decisions.

The timestamp result nobody planned for

With the adaptive architecture in place, the team set out to ship the planned float-compression algorithm. Before it arrived, the architecture proved itself by substantially improving compression for timestamps.

A time-series index is sorted first by its time-series identifier (_tsid constructed by the metric’s dimensions) and then by timestamp within each series. Timestamps on disk aren’t one smooth sequence; there are many smooth sequences laid end to end, one per series, with a large jump at every boundary where one series ends and the next begins.

The codec compresses data in fixed-size blocks without regard to those series boundaries. A block straddling a series boundary holds timestamps from two different series. The jump between them breaks monotonicity, reducing delta encoding effectiveness on blocks spanning different time series. A block that would otherwise compress to near-zero bits per value ended up needing nine or more, because bit-packing encodes every value in a block using the same fixed number of bits, so one large jump sets the cost for all of them.

In the ideal case, every block belongs to a single series: timestamps increase at near-constant intervals, delta encoding captures the regularity, and bit-packing compresses the result to near-zero bits per value. With few series, boundary blocks are rare and the overhead barely registers. On an observability cluster ingesting millions of documents across thousands of series, that changes. The cost scales along two dimensions: series count and data density. More series means more boundary events; sparser series means multiple jumps packed into single blocks. In high-churn environments, both compound, and boundary blocks accumulate into a standing tax on the most-read field in any time-series workload. It’s why @timestamp storage grew faster than the data that produced it.

The fix was to detect series boundaries and treat each run as its own independent sequence. Instead of trying to encode across the jump between two series, which forces every value in the block to pay the storage cost of that one large jump, each run is compressed on its own terms. The boundary simply becomes a seam: the jump is never seen by the encoder on either side.

That encoding is called SplitDelta. @timestamp and monotonic long counters now use it by default. No format change. No migration. Existing segments retain legacy encoding.

On the high-cardinality Elasticsearch Rally benchmark, that single unplanned encoding cut counters storage by 20%–30% and @timestamp storage by 92.4%, from 1.03 GB to 79 MB. Gigabytes to megabytes, and no, that isn’t a typo.

The pluggable pipeline had already paid for itself. SplitDelta, which wasn’t part of the original plan, slotted in without a format change or migration before ALP even shipped.

ALP: recovering the decimal that was always there

Most floating-point metrics can be expressed as short decimals with no loss of accuracy: CPU utilization at 23.47, load average at 1.15. ALP (Adaptive Lossless floating-Point compression) recovers that decimal structure from the floating-point representation, converting values into integers that the existing pipeline already handles well. ES95 feeds ALP's output into the same mature integer compression pipeline used for timestamps and counters, extracting additional savings rather than treating ALP as a standalone encoding. Values that don’t fit ALP's model (such as irregular high-precision floats or special values) fall back to direct bit-packing or the original representation without degrading the rest of the block.

ALP lets Elasticsearch treat floating-point metrics according to the structure they actually contain rather than the binary representation they happen to use. It’s applied automatically to double-valued gauge fields, selected by field type and metric role, through exactly the door the architecture had built for it.

What the numbers say

Here’s what the Elasticsearch Rally benchmark looks like on a high-cardinality workload containing 2.26 billion data points. Results are from an internal tsdb-metricsgen benchmark.

Field or metric

Storage reduction (%)

@timestamp

−92.4%

cpu.load_average.5m

−74.3%

system.cpu.utilization

−63%

memory.utilization

-19%

Total doc values

−33.6%

That overall 33.6% reduction deserves context.

A time-series index contains more than metrics. Every data point also carries the labels that identify the series: host names, IP addresses, regions, container IDs. Those dimension fields are stored as keywords. ES95 doesn’t target dimensions.

On this benchmark, two dimension fields, host.ip and host.mac, accounted for 44% of doc-values storage after ES95 ran. The 33.6% total reflects that mix. The per-field breakdown is the honest picture. Compression for dimension fields is an active area of work.

The per-field variation is the most convincing result. Some gauges shrank by nearly three quarters, while others moved by less than a fifth. That spread is direct evidence that ES95 matches compression to the structure actually present in each field. A fixed encoding treats every field identically and misses most of those wins.

Better compression without extra configuration

The storage reductions from SplitDelta and ALP are the most visible results of ES95. The more consequential result is the architecture that produced them.

Before ES95, every new compression technique required a format evolution. That reality shaped which ideas were practical to pursue. Today, new encodings become implementation decisions inside the codec rather than migration projects. Existing data never needs to move, and users gain better compression on newly written data simply by upgrading Elasticsearch. SplitDelta and ALP are the first encodings to benefit from this architecture. They will not be the last.

Asking users to choose compression algorithms would only duplicate information Elasticsearch already has. There are no per-field compression parameters to tune, and no expert knowledge is required to get good storage efficiency. Different fields get different strategies because ES95 understands what kind of data each field contains, not because a user configured it. As the codec evolves, those decisions evolve with it. The API does not.

In Elasticsearch Serverless, good defaults are part of the product. Users expect the system, not configuration, to make storage decisions. ES95 is designed to honor that expectation: encoding that starts right and gets better over time.

The compression was always there

ES95 establishes a new standard for how time-series codec evolution works. New encodings become implementation decisions, not migration projects. Users get better compression on newly written data with every Elasticsearch upgrade.

The compression was already in the data. ES95 just removed what was hiding it.

How a search is executed in Elasticsearch

Let’s start with some Elasticsearch basics. When a search request lands on an Elasticsearch node, that node is called the coordinating nodefor the search. By default, any Elasticsearch node can act as a coordinating node. The coordinating node determines which shards need to be searched based on the index or indices specified by the search. The nodes that those shards live on are called data nodes. A coordinating node can also be a data node, as it may host shards that are relevant to the search.

Elasticsearch performs the search in two primary phases: the query phase (also known as the scatter phase) and the fetch phase(also known as the gather phase). The query phase is responsible for going to the data nodes and executing the query on each shard. Each shard responds with a set of document IDs (just the IDs, no data) and an associated score for each. These results are reduced. Next, the fetch phase goes back out to the data nodes to fetch the document _source (the data). 

What exactly is a reductionin Elasticsearch? A reduction turns per-shard results from the query phase into a single merged result for the client. Suppose a search asks for the top five hits in a three-shard index, according to some relevance score. Elasticsearch must then get the top five hits from each targeted shard. Why? Because it’s possible that one shard contains the global top five, or, more likely, that the top five docs are spread across shards. 

If three shards are being searched, the coordinating node will have 15 (docID, score) pairs after the query phase. These results are reduced: The documents with the top five scores are kept and the rest thrown away. Then the fetch phase reaches back out to the data nodes to get the documents’ _source data, which Elasticsearch then responds with.

shard1_results = [(id: 231, score: 0.871), (id: 445, score: 0.812), (id: 88, score: 0.754), (id: 312, score: 0.701), (id: 567, score: 0.643)]

shard2_results = [(id: 847, score: 0.921), (id: 76,  score: 0.843), (id: 125, score: 0.783), (id: 438, score: 0.729), (id: 590, score: 0.668)]

shard3_results = [(id: 512, score: 0.887), (id: 289, score: 0.798), (id: 74,  score: 0.741), (id: 631, score: 0.682), (id: 405, score: 0.619)]

// Take the top five scores from above (that’s the “reduction”) reduced_result = [(id: 847, score: 0.921), (id: 512, score: 0.887), (id: 231, score: 0.871), (id: 76, score: 0.843), (id: 445, score: 0.812)]

What is the batched query phase?

Without batching (shard fan-out)

To understand the batched query phase, we must first understand how the query phase worked without batching. The following diagram represents a three-node Elasticsearch cluster with 12 index shards. Suppose a client search request lands on Node 1. That makes Node 1 the coordinating node for the search.

The coordinating node first fans out to all shards, performing the client’s query on each shard. Each shard query is facilitated by a transport request, which is a protocol used for communication between Elasticsearch nodes. Notice that in this diagram, each shard gets its own transport request. Node 1 holds shards itself, but no network request is needed to query those shards.

After querying the shards, the coordinating node must reduce them, as described in the previous section. At this point, the coordinating node is holding 12 shard results. Once it reduces them all, it can proceed with the fetch phase and then respond back to the client.

With batching

So what does the batched query phase change? The batched query phase first takes effect before dispatching the shard query transport requests. Now the coordinating node batches the shards it needs to query for each data node and requests them all at once.

Notice that only one transport request was made to Node 2 and one to Node 3. This is the first change in the batched query phase. Each transport request induces some overhead, so this change alone buys us our first performance improvement, reducing latency and CPU cycles spent on overhead.

The second change in the batched query phase has to do with reducing shard results. Now partial reductions occur on the data nodes themselves. For instance, after querying shards 2, 5, 8, and 11, Node 2 then reduces those results. The coordinating node, upon receiving partially reduced results from Node 2 and Node 3, must then perform afinal reduction. This is the second major enhancement we get from the batched query phase: the spreading out of reduction work across the data nodes. This reduces memory pressure on the coordinating node, which we’ll see measured later in the benchmarking section.

The benefits of batching

Fan-out (no batching) is how search worked for a very long time. It has advantages: each shard is a separate request that returns and can be retried independently from all the others. With many shards involved, though, each shard request causes overhead due to many round trips going between the coordinating node and the data nodes.

Also, the coordinating node doesn’t have enough information to be able to determine the pace at which to send requests to each data node. It sends a maximum of five concurrent requests per data node by default, where five is a bit of a magic number which allows for some parallelism, while at the same time preventing a single query from taking over an entire data node. At the same time, data nodes also don't make distinctions between the different shard requests they receive, for instance based on what parent search request they belong to. In reality, if all shard requests involving a search request are presented in one batch to each data node, the data node can then look at its internal state and adapt its pace dynamically, removing the five concurrent shard requests artifical limit mentioned above.

The batched query phase results in several benefits, including:

• Increased efficiency, thanks to fewer round trips: less CPU spent on transport overhead and fewer bytes going through the transport layer.

• Spreading out the load of reductions: the coordinating node was previously the bottleneck for reductions, and now data nodes share the work.

• Better resource usage: we may be able to better max out the data node’s CPUs.

The average search against many shards can now be served much quicker, with lower latency and higher throughput.

Batched query phase benchmarks: Latency and memory usage

To measure the benefits of the batched query phase, we ran a couple of benchmarks. The first displays the benefit of reduced transport overhead. This benchmark was built off the “many-shards-quantitative” nightly benchmark. It runs on a three-node Elasticsearch cluster, running batches of searches targeting 1,000, 5,000, and 20,000 shards. The queries are match_all queries with size: 0. That means querying the shards themselves is effectively a no-op. This benchmark is meant to isolate the work of coordinating the search across the cluster.

We ran the same benchmark with and without the batched query phase (by toggling the cluster setting search.batched_query_phase). At 5,000 shards, the batched query phase makes searches twice as fast. By 20,000 shards, it’s 2.2x faster. These results are a ceiling for what a realistic workload can expect to benefit; any real query work at the shard level will dilute the overall result. However, the gains are real, and the coordination work of your queries will benefit as shown. Note that this first benchmark did not attempt to measure any gain brought by spreading reductions across data nodes, as opposed to performing them only on the coordinating node.

Next, we benchmarked the benefits of the batched query phase on large reductions. In our benchmark, we ran a large terms aggregation over a data set called http_logs (which can be found in our rally-tracks repo). This data set has 247 million documents, which we indexed in seven indices each with 100 shards, again in a three-node Elasticsearch cluster. We ran a single terms aggregation query for the clientip field with size: 1000 and shard_size: 50000. That means we’re asking for the top 1,000 terms, although each shard will return 50,000 buckets to be reduced to that 1,000. Requesting so many terms from each shard increases precision, but has a cost in terms of memory usage and reduction overhead, which helps highlight the gain provided by spreading incremental reductions across data nodes.

The memory used is measured by something called a circuit breaker, which accounts for memory usage in Elasticsearch. This field can be found in the /_nodes/stats response under nodes.<node_id>.breakers.request.estimated_size_in_bytes. It estimates the memory in use for processing in-flight search requests.

The memory benchmark shows how the work of reductions is now spread across the cluster. In blue is the memory used with search.batched_query_phase: false. We can see that the node elasticsearch-0 is the coordinating node, as it bears the entire reduction load itself. This effect is no longer the case with search.batched_query_phase: true in red. The node elasticsearch-1 is the coordinating node, but it uses far less memory. That’s because the data nodes elasticsearch-0 and elasticsearch-2 do reductions themselves, reducing the results before they’re returned to the coordinating node.

Summary

Batching shard queries per data node reduces transport overhead and allows each data node to perform partial results reduction locally. The result is lower latency for searches targeting many shards and better distribution of work across the cluster for reduction-heavy workloads. At the same time, the new model works well for all scenarios and opens the door for potential future enhancements which we’re excited to pursue, such as:

• Optimize out-of-the-box resource usage by introspecting data nodes' activity and adjusting the pace accordingly, which would replace the five concurrent shard requests per data node artifical limit.

• Assign priorities to search requests and let each data node process shard requests accordingly.

• Reduce the number of roundtrips further by folding the `can_match` phase into the query phase. Can match is a separate "batched" roundtrip used to shortcut the query against shards that can't possibly match based on index statistics. Now that the query phase is batched, both rounds can be executed in one go.

• Discontinue support for minimize roundtrips in cross-cluster searches in favour of batched query execution; in hindsight, minimize roundtrips achieves a similar goal but is specific to cross-cluster execution, while batched execution is applicable to every search. 

Columnar Mode targets the workloads where data is written in volume, queried analytically, and retained for a long time: logs and observability, security telemetry, metrics and traces, business analytics on operational data, and AI retrieval at scale. For these workloads, it means meaningfully smaller storage footprints on day one, and the foundation for faster ingest, faster analytical queries, and longer retention as the surrounding work matures.

Columnar Mode ships alongside the existing modes, not in place of them. There are no changes to APIs, dashboards, applications, or downstream integrations. It's a new index mode that you can adopt where it helps, on the data where it fits.

The result is that Elasticsearch becomes a world-class search engine and columnar analytics engine, on the same data, in the same cluster, under the same operational umbrella. This is how Elastic stays the most useful place to put operational data as the economics of that data are rewritten by columnar architectures.

The rest of this post explains why we're doing it, along with what it changes and what it doesn't.

Why Elasticsearch is adding a columnar mode

Almost every general-purpose data system worth caring about has, over the last 15 years, made the same architectural decision: When the job is to read, aggregate, and reason over large volumes of data, you organize that data by column, not by row or document.

Elasticsearch is now making that decision, too

In addition to its document model, search heritage, and place at the heart of observability, security, and search applications across the industry, it’s now adding a second way to think about data inside the same platform. We're calling it Columnar Mode. It positions Elasticsearch to be the only major platform that does search, retrieval, and analytics at the same level, on the same data, with one operational story.

To get there we need to point out three things: One, how Elasticsearch came to be what it is today and why the document model was the right call. Two, how the rest of the data world, in parallel, converged on a completely different shape and why that movement has been so successful. And three, why these two worlds are now converging inside our users' systems and why putting columnar inside Elasticsearch is the right answer.

How Elasticsearch became a document database (and why that was the right call)

To understand where we're going, you have to understand where we started.

Sixteen years ago, when Elastic’s founder Shay Banon wrote the first version of Elasticsearch, the data world was in the middle of a quiet revolution. JSON was eating the wire. Web applications were sending and receiving data not as carefully typed rows in clearly defined tables, but as flexible, semi-structured documents (a customer order, a chat message, a product listing, a log line, a recipe), each a self-contained little world of fields, sub-fields, arrays, and nested objects.

The relational databases of the time were built around a very different assumption. Define a strict schema up front. Decompose your data into normalized tables. Use joins at query time to reassemble it. This worked beautifully for the use cases it was designed for (transactional systems, banking, enterprise resource planning systems), but it created enormous friction for the new generation of web and mobile applications. Adding a field meant a migration, and storing variable-shaped data meant either a brittle schema or a tangle of nullable columns.

In response, a new category of system emerged: document databases. The bet was simple. Take the data as it arrives, and store it in something close to its native shape. Don't make developers fight the database to express their domain. MongoDB became the flag-bearer for this idea on the operational side, and Elasticsearch carried the same banner on the search side.

For Elasticsearch, the document model was not just a developer-experience choice. It was a deep architectural commitment. The engine was designed around the idea that a record is a document, a document has fields, fields have types, and you should be able to put almost anything in and get useful behavior out (full-text search, structured filtering, sorting, aggregations, ranking) without having to model your data perfectly in advance.

This is what we mean when we talk about Elasticsearch's "document behavior":

• The engine remembers what you sent it. The original record is stored and can always be returned exactly as it was.

• Every field is, by default, made queryable in multiple ways: optimized for fast exact lookups, fast text search, fast range queries, fast sorting, and fast grouping.

• Schemas are flexible. New fields are absorbed at write time. The cost of getting your data model "wrong" is low.

• The unit of work is the document. Indexing, retrieval, and most APIs are framed in terms of "given a document, do X" or "given a query, return documents."

This is a model born from search. Apache Lucene, the storage library Elasticsearch is built on, was designed to make one workload remarkably fast: Take a query, find the small number of documents that match it, rank them by relevance, and return them. To do that well, you build inverted indexes: structures that turn the question around, so instead of asking "What's in this document?" you can ask "Which documents contain this term?" and get an answer in milliseconds. You also keep the original document around, because once you've found it, you usually want to show it or retrieve more information from it.

For the workloads Elasticsearch grew up serving (site search, log search, application search, security event lookup, "find this in a haystack") this was, and still is, the right shape. There's a reason Elasticsearch is used in production at hundreds of thousands of organizations worldwide.

But the document model has a built-in cost, and that cost is the starting point for everything that follows.

To keep its promise of fast search, flexible schemas, and faithful storage of the original record, the engine ends up storing data multiple times, in multiple shapes, each optimized for a different question. The original document is stored, and the text in each field is indexed for search. The values in each field are also stored separately, in a per-field columnar store known as doc values, so that they can be aggregated, sorted, and grouped. The system maintains all of these in parallel, by default, on every field, because at write time it doesn't know which of these capabilities you'll need at read time.

For the kinds of datasets Elasticsearch was originally designed for (relatively rich, relatively low-volume documents where every read is valuable), this trade-off is excellent. You pay a modest storage and ingestion tax for an enormous capability surface.

For the kinds of datasets Elasticsearch increasingly stores today (billions of log lines, trillions of metric points, oceans of telemetry, most of it written once and read rarely), that trade-off starts to look very different.

That’s the friction we’re setting out to resolve.

The other story: How the data world went columnar

While Elasticsearch was perfecting the document model, a parallel revolution was happening in analytics. It started in academia, became commercial, and is now the default architecture for almost every system built to read and reason over large datasets. To position Columnar Mode clearly, we have to tell that story.

The bet behind columnar storage is so simple it almost sounds like a trick. Instead of storing your data row by row (record one, record two, record three…), store it column by column (all values of timestamp, then all values of host_name, then all values of status_code…).

That single decision changes everything.

It started in the early 1990s, with research systems like MonetDB out of the Centrum Wiskunde & Informatica in Amsterdam. It was sharpened in the mid-2000s with C-Store, a project led by Michael Stonebraker at MIT, which became the basis for Vertica. Sybase IQ carried the same idea into the early commercial market. By the early 2010s, every serious analytics vendor on earth had a columnar story. Today, the lineage runs through Amazon Redshift, Google BigQuery, and Snowflake in the cloud warehouse world; through open file formats, like Apache Parquet and Apache ORC that have become the lingua franca of the data lake; through in-memory standards like Apache Arrow; and through fast operational columnar engines, like ClickHouse and DuckDB.

Why has this approach won so completely?

Because when your job is to read and reason about a lot of data, columnar storage turns nearly every dimension of cost and performance in your favor.

You only read what you need. Real analytical queries almost never want every field; they want three fields out of 50, or aggregate one field out of 200. A row-shaped store has to walk past all the other fields on the way to the ones you care about. A columnar store reads only the columns the query touched. On wide datasets, this routinely cuts the I/O of a query by 90% or more.

You compress dramatically better. When you put all the values of one field next to each other, those values are, by definition, of the same type, often with repetition and similar structure; for example, a column of timestamps from the same hour, a column of status codes that are almost always 200, or a column of hostnames drawn from a small set. Compression algorithms thrive on this kind of homogeneity. Where row-shaped stores typically achieve 1.5–3x compression, columnar systems routinely achieve 5–10x, while on low-cardinality fields, it's common to see 20–30x. That’s not just a tuning improvement; that’s a different economic regime.

You can skip vast amounts of data without reading it. Because data is organized in blocks of values from the same column, the engine can carry tiny pieces of metadata for each block (minimum, maximum, count, distinct values) and use them to prune entire blocks at query time. Looking for errors in the last hour? Skip every block whose timestamp range is older. Looking for a specific hostname? Skip every block where it doesn't appear. The system avoids work it doesn't need to do by knowing in advance that the work is pointless.

You can fully leverage the power of modern CPUs. Modern processors are designed to operate on long, regular, predictable arrays of values; that's where the cache, pipeline, and vector units do their best work. A column is exactly such an array. Columnar engines run queries by passing batches of values through tight, vectorized operators, instead of looking up one record at a time. The speed gains aren’t incremental: Across academic benchmarks reaching back to Abadi et al.'s seminal Column-Stores vs. Row-Stores (SIGMOD 2008) and the comprehensive surveys that followed, columnar engines have consistently shown one to three orders of magnitude advantage over row-shaped systems on analytical workloads.

You delay materializing the original record for as long as possible. Because the data is already organized by what queries actually do (read this column, filter on that column, aggregate this column), there's no need to reconstruct a full row until you absolutely have to. Most of the time, you don't have to at all, especially if you run analytical queries that only care about aggregating a few columns.

The cumulative effect of these properties is the reason every cloud data warehouse is columnar and why every modern data lake stores its files in Parquet or ORC. It’s also the reason new operational analytics engines have, almost without exception, chosen this shape. It’s the structural answer to the question "How do you make queries over large data cheap and fast?" and the answer turns out to be the same almost everywhere.

We should also note what columnar systems give up. They aren’t designed for "Give me one specific record and update it in place" or for transactional consistency at the level of individual rows. In general, they aren’t designed to do full-text relevance ranking the way a search engine does. They’re optimized for a different shape of workload, and that's the point. Different workloads, different defaults.

For a long time, the world worked because users could pick one tool for one workload and another tool for the other. That world is ending.

Why search and analytics are now converging

Three things have happened in the last five years that change the calculus.

First, the volume of data users generate has exploded. Driven by microservices, containers, OpenTelemetry, AI workloads, and security telemetry, industry estimates put observability and log data ingest at large enterprises in the multiple-terabytes-per-day range, with annual growth rates that have run well into triple digits for several years.

Second, the cost of storing and querying that data has become a top-line concern. Industry surveys and practitioner reports consistently flag observability and telemetry as one of the largest and fastest-growing line items in modern infrastructure budgets, and a meaningful share of that spend goes toward data that’s written, retained, and then never queried.

These first two pressures have already produced their answer in the market: Modern columnar systems built for analytics (most visibly ClickHouse on web logs and the cloud warehouses on broader analytics) have redefined what users expect to pay per terabyte stored and per query run. Elasticsearch is now meeting that bar without giving up what makes it Elasticsearch.

Third, the boundary between search workloads and analytical workloads has dissolved. Virtually every modern operational use case wants both: find me this specific event, and then aggregate everything around it; show me this log line, and then chart the trend that produced it; retrieve this document, and then group what else matches.

The users who put their logs, metrics, traces, security events, and search corpus into Elasticsearch aren’t asking us to be only a search engine or only an analytics engine. They’re asking us to be one engine that does both, on the same data, with one operational story, at the cost basis they expect from a modern columnar system.

That’s a high bar, and meeting it requires us to change something fundamental about how Elasticsearch organizes data.

For a long time, Elasticsearch has had columnar storage underneath. We’ve stored per-column data in our engine since 2013, but we’ve always treated it as a secondary structure layered on top of a document model. That made sense when the document model was the source of truth and the columnar layer was an optimization. It makes less sense when, for an enormous and growing share of our users' data, the columnar shape is the source of truth and the document model is the optimization most don’t need.

This is what Columnar Mode is for.

Columnar Mode: Elasticsearch with a second way of thinking about data

The decision we’re making is deliberately conservative in its surface area and ambitious in its impact.

We aren’t building a new product, asking users to migrate, or changing the API, the query language, the management surface, the integrations, the visualization layer, or anything else that touches their operational reality. Elasticsearch is still Elasticsearch.

What we are doing is introducing a new mode that users can apply, index by index, that turns Elasticsearch into a first-class columnar system for the data where that's the right shape.

In Columnar Mode, the engine flips its defaults:

• Data is stored once, in our columnar store (doc values). No parallel copy of the original document and no secondary structures built by default. Each field is responsible for its own storage, and the engine doesn't pay for capabilities the workload doesn't need.

• Search indexes are built only where they earn their keep. The message field of a log is still indexed by default to enable free-text search. A numeric field used only in aggregations means no index, extra storage, or additional write-time cost. The engine becomes leaner by default and lets you opt in to capability where it's worth paying for.

• The original record can be regenerated on demand from the column store, instead of being kept as a redundant copy. Users who want to keep the stored copy for query convenience can; and users operating at petabyte scale can opt to drop it entirely for further storage savings.

• The data model is genuinely columnar. Fields are flat key/value pairs, not nested object trees. Multi-valued fields, cardinality, and nullability are first-class mapping concepts, the same primitives that make pure columnar systems efficient.

• Specialized profiles ship on top. The first one is Columnar Logs, a logs-oriented variant of the mode with indexing enabled on log messages and the right defaults for time-ordered data. We’ll follow with profiles for other workloads, including, eventually, a columnar profile for vector retrieval, using the same building blocks.

All of this is new for Elasticsearch though not conceptually new in the industry.

The reason this matters is that we’re doing this without leaving anything behind. The same query that runs against a Columnar Mode index runs, unchanged, against a document-mode index. Plus, the same dashboards, agents and integrations, Service Level Objectives (SLOs), alerts, rules, and machine learning jobs work as usual. Users who want the document behavior can always keep it on the indexes where it makes sense. Users who want columnar efficiency on the indexes where that makes sense (typically, the largest indexes in their cluster) get it.

This is the difference between us and a pure columnar engine. A pure columnar engine does one of these jobs and asks you to bring another system for the other. Elasticsearch is one platform that does both.

What Columnar Mode is for

Columnar Mode is built for workloads where data is written in volume, queried analytically, and retained for a long time, rather than a universal default. These work categories include:

Logs and observability at scale. Users running tens or hundreds of terabytes of logs per day spend most of their observability budget on storage and on the aggregation queries that power dashboards, SLOs, and rate calculations. Columnar Logs is the first specialized profile precisely because this is where the impact lands hardest. At petabyte scale, Columnar Mode changes what is economically possible, that is, longer retention, more raw fidelity, and fewer compromises forced by cost.

Security event stores and threat hunting. Security telemetry has the same shape as logs, but with a heavier emphasis on faceted exploration, ad hoc correlation, and deep historical lookups for indicators of compromise. Columnar Mode keeps full-fidelity storage affordable while preserving the search-relevance behavior security analysts need for lookup and pivoting, using the same engine to handle both jobs.

Metrics, traces, and the unified observability substrate. Time-series data is already columnar in TSDB, Elasticsearch’s time series database. A future columnar profile will share the same storage building blocks across logs, metrics, and traces, letting all three sit in one engine on the same substrate without any of them compromising on workload-specific behavior. The unified observability platform becomes economically practical, not just architecturally elegant.

Operational and business analytics on application data. Beyond observability, this means dashboards over millions of orders, transactions, user events, and Internet of Things (IoT) readings. Many of the workloads that have historically pushed users to ship the same data into a separate analytical warehouse, purely to make the queries affordable, can, with Columnar Mode, stay where the data already is.

AI retrieval at scale. Vector workloads are inherently columnar. A future columnar profile for vector retrieval will combine the dense storage efficiency of the general mode with the indexing structures retrieval needs, putting retrieval augmented generation (RAG) and semantic-search workloads on the same cost basis as every other shape of data in the cluster.

The thread running through all of these is that Columnar Mode is the right answer when data is append-mostly, queries are analytical, and access patterns are per-column. For the workloads where those conditions don't hold, like search-first applications where the document is the unit of value, transactional flows that update individual records, or anything that depends on rich document structure as user-facing semantics, the document-oriented modes remain the right default, and they’ll continue to be invested in.

From the start, the principle has been that different workloads require different defaults, and Columnar Mode is what makes that real.

What changes for Elasticsearch users with Columnar Mode?

For Elasticsearch users, here’s what changes:

• Costs go down meaningfully. Columnar Mode only builds inverted indexes where they earn their keep and stops storing data multiple times on every field. The result is a smaller storage footprint for the same workloads. Users can choose between two shapes of the mode: Columnar Logs keeps an inverted index on the message field, where the bulk of log queries actually look; the pure Columnar Mode variant drops that default entirely on string-type fields. A follow-up technical deep dive will publish the benchmarks and exact storage savings. The savings compound at scale and show up in operational reality immediately on self-managed clusters in the disks and machines the cluster no longer needs, and on Elastic Cloud in the bill.

• The architecture pays off on reads and writes, too. The same choices that make Columnar Mode storage-efficient (data stored once, indexes only where they earn their keep, and vectorized execution over columns) are the ones that shape its behavior on the read and write path, as well. The workloads that benefit most are the ones that dominate observability, security, business intelligence, and AI retrieval.

• The model gets simpler. Columnar Mode is opinionated by default. The right behavior for analytical workloads is what you get out of the box, like what we’ve done for metrics with TSDB, but for all types of data. The configuration surface contracts in places it should have contracted years ago.

• The path is nondisruptive. Users can adopt Columnar Mode index by index. The cluster, applications built on top of Elasticsearch, and dashboards don't change. The model is additive: a new way of behaving that’s available where it helps, alongside the behavior they already trust.

The result is an Elasticsearch that’s leaner and faster, with no changes where it has been working all along.

What doesn't change when you turn on Columnar Mode

• The document-oriented modes aren’t deprecated. Every existing mode remains available, supported, and invested in. Users who depend on document behavior (typical search applications, application search, security workflows, and anything where the original record's structure is the point) keep that behavior.

• Existing modes get faster, not slower. The work behind Columnar Mode is, at its core, work on doc values, Elasticsearch's columnar store, which every mode uses under the hood. Compression, encoding, query-execution, and vectorization improvements that ship with Columnar Mode benefit document-mode indexes alongside columnar ones for fields that reside on doc values. LogsDB, TSDB, and standard indexes get meaningfully faster analytical queries as a side effect of this work, with no user action or migration required. Investing in Columnar Mode is investing in every mode.

• The APIs don’t change. Every interface a user or partner integrates with (the REST APIs, query languages, management UI, data collection agents, integrations, and SDKs) keeps working exactly as it does today. Columnar Mode is an index setting, not a fork in the product, and downstream components don’t need to learn anything new.

• Search relevance, vector search, and semantic retrieval aren’t affected. These remain first-class capabilities of the engine, and they benefit from many of the same query and indexing performance investments that Columnar Mode rides on.

• There’s no migration cliff. Existing indexes keep working, and new indexes can be created in whichever mode fits their workload. Users move at the pace that makes sense for their organization.

• Elastic Cloud Serverless gets columnar the same way. On Serverless, where users manage projects rather than indices, Columnar Mode becomes part of the project type configuration. Observability and log-heavy project types will move to columnar defaults as the mode matures, with no user-facing change in how projects are created or managed.

This matters because the temptation, when a company makes a big architectural bet, is to oversell it as the new gospel and undersell the engine's existing strengths. We don't have to make that trade. Elasticsearch's existing modes aren't legacy; they’re a mature, production-hardened document-oriented search engine that has been refined by more than a decade of use at scale. Columnar Mode is what they've been missing for the workloads they were never designed for.

Why columnar storage matters for the future of Elasticsearch

For most of the last decade, the dominant pattern has been fragmentation; for example, a search engine for search, a warehouse for analytics, a time-series database for metrics. This also includes a log aggregator for logs, a vector database for AI retrieval, and a graph database for relationships. Each one has its own ingestion pipeline, query language, operational story, and bill. The complexity tax this imposes on users is enormous, and the spend it generates for the industry is even larger.

The pattern that’s starting to replace it is convergence, the realization that the right architecture for a modern data engine is one that takes data once, stores it efficiently, and exposes it to whatever workload the application happens to need, whether that’s search, retrieval, aggregation, ranking, or vector similarity. That convergence is the most important trend in the data infrastructure industry, and it’s the trend on which Elasticsearch's future depends.

Columnar Mode is our central move in that direction, along with the work happening alongside it on indexing throughput, Elasticsearch Query Language (ES|QL, Elasticsearch’s analytical query language), vector retrieval, and observability and security as integrated solutions. It’s the architectural decision that unlocks the others, because without a first-class columnar layer, Elasticsearch's economics simply don’t compete on the workloads where convergence is most valuable.

With Columnar Mode, Elasticsearch becomes the only major platform in the industry that can credibly be the best search engine you can run and a competitive columnar analytics engine on the same data, in the same cluster, under the same operational umbrella. Although there will always be a specialized system that wins a specialized benchmark for individual workloads, Elasticsearch is where operational data belongs when it spans more than one shape and workload.

That’s the bet, and we’re confident it’s the right one.

The bottom line on Elasticsearch as a columnar database

Sixteen years ago, the right call for a search engine was to behave like a document store. We made that call and built one of the most widely used data platforms in the industry.

Today, the right call for the next decade of operational data (observability, security, AI retrieval, application search, business analytics) is to give that same engine a columnar way of thinking as a first-class mode users can choose, on the data where it fits, without giving up any of the things they already rely on us for.

That’s what Columnar Mode is. It’s the most important architectural change we’ll make to Elasticsearch this decade, and it’s the move that makes Elasticsearch the engine our users will still be reaching for when their current tool of choice has been displaced.

We’re going columnar. The reasoning is sound, and the path is nondisruptive. Columnar Mode reaches technical preview in Elasticsearch 9.5 and general availability (GA) in Elasticsearch 9.6, and every product team at Elastic is building on top of this foundation. We aren’t waiting for the future of data systems to arrive; we’re building it.

FIPS 140-3 support for Elasticsearch and Kibana is generally available in Elastic 9.4 for self-managed deployments. NIST has stopped accepting new FIPS 140-2 submissions, with existing certificates winding down in September 2026. Every layer of your infrastructure needs to catch up, and your search and analytics platform is no exception. Federal programs, defense integrators and regulated enterprises are actively moving procurement requirements to 140-3. With Elastic 9.4, your stack can answer "yes."

Two problems, one deadline: why FIPS 140-2 is no longer enough

If you've been running Elasticsearch in FIPS 140-2 mode, you already know the value of having a compliant Elastic Stack. But two pressures are converging:

• The standard is sunsetting. FIPS 140-2 certificates are being phased out. Procurement officers, auditors, and authorization bodies are shifting their requirements to 140-3. A stack that only supports the old standard is a stack with an expiration date on its compliance story.
• Auditors don't accept "close enough." It's not sufficient to run FIPS-approved algorithms. Your application layer needs to be explicitly configured for FIPS mode, with nonapproved algorithms disabled and cryptographic boundaries clearly documented. Partial compliance is noncompliance.

Can't I just put Elasticsearch behind a FIPS-compliant load balancer or run it on a FIPS-hardened OS? No, and here's exactly why: FedRAMP's network security requirements apply to cryptographic operations at the application layer, not just the network boundary. Intranode communication, keystore encryption, and credential hashing all need to happen inside a validated module. A compliant perimeter around a noncompliant application layer isn't a FIPS deployment; it's an audit finding waiting to happen.

What FIPS 140-3 mode does in Elasticsearch 9.4

When you flip FIPS mode on, Elasticsearch and Kibana restrict every cryptographic operation to FIPS-approved algorithms and delegate all of it to the validated module in your runtime. Here's what that covers and how.

• High-grade security Transport Layer Security (TLS) everywhere, no exceptions. Node-to-node transport, the REST API over HTTPS, Kibana talking to Elasticsearch: All of it uses only FIPS-approved cipher suites. Noncompliant suites aren't deprioritized. They're rejected.
• PBKDF2 replaces bcrypt for password hashing. Bcrypt isn't FIPS-approved, so in FIPS mode, Elasticsearch switches to PBKDF2 for the native realm, file realm, and any stored credentials. Your users and service accounts stay protected with an algorithm that the auditor won't flag.
• Keystore encryption stays inside the boundary. Secrets in the Elasticsearch keystore, API keys, repository credentials, and encryption keys for Kibana's saved objects are wrapped with FIPS-approved key derivation and encryption. No gaps between the cluster is compliant and the secrets are compliant.
• You supply the cryptographic module. FIPS 140-3 support in Elasticsearch is built on the Bouncy Castle FIPS Java API 2.0, a FIPS 140-3 validated cryptographic module that runs inside the Java Virtual Machine (JVM). Elasticsearch delegates all cryptographic operations to Bouncy Castle; it doesn't implement its own crypto functions. Elasticsearch uses your own FIPS 140-3 JVM and Bouncy Castle FIPS provider, giving you full control over your cryptographic boundary and module versioning.

Kibana takes a different path, running in a Node.js environment configured with a FIPS-compliant OpenSSL 3 provider. Together, both components operate within clearly defined cryptographic boundaries clean enough to diagram for an auditor.

Who needs FIPS 140-3 support in Elasticsearch

• Federal and defense teams operating Elastic inside FedRAMP boundaries, Cybersecurity Maturity Model Certification–scoped (CMMC-scoped) environments, or Defense Information Systems Agency (DISA) Security Technical Implementation Guide–hardened (STIG-hardened) infrastructure. You can now upgrade to 9.x without punching a hole in your authorization documentation. Your Authorization to Operate (ATO) package references FIPS 140-3, not a soon-to-expire 140-2 certificate.
• Financial services and healthcare organizations where Payment Card Industry Data Security Standard (PCI DSS), Sarbanes‑Oxley Act (SOX), or Health Insurance Portability and Accountability Act (HIPAA) audits ask how your search infrastructure handles cryptography. FIPS mode gives your compliance team a one-word answer instead of a three-paragraph explanation.
• Anyone fielding Do you support FIPS 140-3? in a procurement questionnaire. That question is showing up in enterprise requests for proposal (RFPs), partner security assessments, and insurance underwriting checklists. With 9.4, the answer is yes.

Migrating from FIPS 140-2 to FIPS 140-3

If you're running FIPS 140-2 on Elastic 8.x today, you don't have to jump to 9.x on day one. Elastic 8.19 continues to support FIPS 140-2, and FIPS 140-3 support is available starting in 8.19.15. That gives you two paths:

• Stay on 8.19.15+, and upgrade the standard in place. If you're not ready to move to 9.x, you can switch from FIPS 140-2 to 140-3 on your current major version. Your cluster stays put, and your compliance posture moves forward. FIPS 140-2 certificates remain valid through September 2026, so you have a window. But the earlier you transition, the less you're depending on a sunset timeline, and you can also benefit from new features!
• Move to 9.4, and land on 140-3 directly. If you're planning a major version upgrade anyway, 9.4 gives you FIPS 140-3 from the start, along with everything in the 9.x line: Elasticsearch Query Language (ES|QL) latest functionalities, improved ingest, updated security detections, and performance improvements. No compliance trade-off required.

Stay on 8.19.15+

Upgrade to 9.4

FIPS standard

140-3 (from 8.19.15)

140-3 (from the start)

Major version change

No

Yes

New 9.x features

No

Yes

FIPS 140-2 certificates valid until

September 2026

N/A (lands on 140-3 directly)

Kibana covered

Yes

Yes

Either way, the configuration model will feel familiar. You enable FIPS mode in your YAML config, point to a FIPS-validated JVM, and the stack handles the rest. And Kibana is covered on both paths: Your visualization and analytics layer operates within the same compliant boundary as Elasticsearch.

How to enable FIPS 140-3 in Elasticsearch 9.4

FIPS 140-3 support is available now in Elastic 9.4 for self-managed deployments with a Platinum or Enterprise subscription, the same licensing tier as FIPS 140-2. For setup instructions, supported JVM versions, configuration details, and known limitations, see the FIPS compliance documentation.

Download Elastic 9.4, and give your compliance team some good news.

Questions? Your Elastic account team can help scope a FIPS deployment, or drop into the Elastic community forums to compare notes with other operators running in regulated environments.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Frequently asked questions

Does Elasticsearch support FIPS 140-3?

Yes. FIPS 140-3 support for Elasticsearch and Kibana is generally available in Elastic 9.4 for self-managed deployments. All cryptographic operations are delegated to the Bouncy Castle FIPS Java API 2.0, a FIPS 140-3 validated module. A Platinum or Enterprise subscription is required.

When does FIPS 140-2 expire?

NIST stopped accepting new FIPS 140-2 module submissions. Existing FIPS 140-2 certificates remain valid through September 2026. Organizations running Elasticsearch in FIPS 140-2 mode should plan their migration before that deadline to avoid gaps in their compliance documentation.

Can I migrate from FIPS 140-2 to FIPS 140-3 without upgrading to Elasticsearch 9.x?

Yes. FIPS 140-3 support is available in Elastic 8.19.15 and later. You can switch from FIPS 140-2 to FIPS 140-3 on your current major version without moving to 9.x. Alternatively, upgrading directly to Elastic 9.4 lands you on FIPS 140-3 from the start.

What cryptographic changes does FIPS mode make in Elasticsearch?

When FIPS mode is enabled, Elasticsearch restricts all operations to FIPS-approved cipher suites, replaces bcrypt with PBKDF2 for password hashing, and delegates all cryptographic functions to the Bouncy Castle FIPS provider. Non-approved cipher suites are rejected, not just deprioritized.

Does Kibana support FIPS 140-3?

Yes. Kibana operates in a Node.js environment configured with a FIPS-compliant OpenSSL 3 provider. Both Elasticsearch and Kibana run within clearly defined cryptographic boundaries when FIPS mode is enabled.

Why isn't a FIPS-hardened OS or load balancer sufficient for FedRAMP compliance?

FedRAMP's network security requirements apply to cryptographic operations at the application layer, not just at the network boundary. Intranode communication, keystore encryption and credential hashing must all occur inside a validated cryptographic module. A compliant perimeter around a non-compliant application layer is an audit finding, not a FIPS deployment.

Is Elasticsearch FIPS 140-3 support available on Elastic Cloud?

Elastic 9.4 FIPS 140-3 support covers self-managed deployments. Cloud deployment availability is not covered in this announcement. Contact your Elastic account team or check the Elastic documentation for cloud FIPS roadmap details.

Vector Search on Elasticsearch Serverless

Elasticsearch Serverless is built on Stateless Elasticsearch, a fully decoupled compute and storage architecture where index data lives in remote object storage and search nodes maintain only a local cache. For vector search to be fast on this architecture, the scoring engine needs to work directly with the local cache, not copy it to the heap first.

Elasticsearch simdvec is the engine behind every vector distance computation in Elasticsearch. It provides hand-tuned AVX-512 and NEON kernels, bulk scoring with explicit prefetching, and off-heap memory access that keeps data flowing from storage straight to CPU registers. On stateful Elasticsearch, simdvec has always had a direct fuel line: Memory-mapped files feed native pointers straight into SIMD intrinsics. On serverless, the data was sitting right there in the blob cache's memory-mapped regions, in exactly the right form, but there was no path connecting it to the scoring engine.

We've now built that path. simdvec runs on Serverless with the same off-heap, native SIMD scoring as stateful. And because serverless gives us control over the entire storage layer, this is just the beginning.

Premium fuel only: why simdvec requires off-heap memory for vector scoring

simdvec's speed comes from working directly with off-heap memory. It takes a native pointer to memory-mapped data and passes it straight to C++ SIMD intrinsics. No intermediate copies, no heap allocations. The data flows from storage straight to CPU registers. This matters more than it sounds: simdvec's kernels process vectors faster than the data can be copied, so any copy in the path becomes the bottleneck, not the scoring itself.

On stateful Elasticsearch, this just works. Lucene memory-maps index files from local disk, and the scorer extracts a native pointer directly from the mapped region. This is the path that delivers the benchmark numbers we've published, and it's what we wanted to bring to serverless. To see how, we first need to understand how serverless stores and accesses data.

The serverless blob cache: how Elasticsearch stores vector data

In the stateless architecture, the primary copy of all index data lives in remote object storage, such as S3. Each search node maintains a local cache (called the blob cache) that keeps recently and frequently accessed portions of the index data on local SSD. The frozen tier on stateful Elasticsearch uses the same architecture: Searchable snapshots are backed by a similar blob cache that memory-maps regions from remote storage onto local disk. When a search hits cached data, it's served from fast local storage. When it misses, the blob cache fetches the data from the remote store and caches it for future queries.

The blob cache is organized into fixed-size memory-mapped regions, 16MB by default. It manages its own lifecycle: tracking which regions are in use, applying a least-frequently-used eviction policy when the cache is full, and reference counting to ensure regions aren't evicted while being read. The regions are still memory-mapped through the OS, but the blob cache controls which regions exist, which are populated, and when they're reclaimed. On stateful, those decisions are left entirely to the OS.

Crucially, because each region is memory-mapped, the blob cache already holds vector data in exactly the form simdvec needs. But before we built the connection, there was no way to get at it. Every vector comparison was copied into a heap array and handed to a slower scorer. No direct memory pointers, no SIMD, and garbage collection pressure on every call.

Unified scoring: one SIMD path for all storage tiers

We introduced a new abstraction that lets the scorer safely borrow direct memory from whatever storage layer is underneath, just long enough to run the SIMD computation. If the data is available as direct memory, simdvec's native kernels run. If not (data not yet cached or spanning a region boundary), the scorer falls back to a heap copy. In practice, the fallback is rare.

This gave us a single scoring entry point across all tiers:

1. Stateful (local disk): The scorer extracts a native pointer from the OS memory map.

2. Blob cache (serverless, frozen tier): The scorer borrows a direct memory slice from a cache region.

3. Fallback: The scorer copies bytes to the heap. Rare in practice.

The scorer doesn't know which tier it's running on, and it doesn't need to. It also means we no longer maintain separate scoring implementations; previously, there was a fast native path for stateful and a slower path for everything else. Now every improvement to simdvec benefits all tiers automatically, including its most powerful capability: bulk scoring.

Bulk vector scoring across blob cache regions

A single query may score thousands of candidate vectors. simdvec's bulk scoring processes these in batches with multi-accumulator inner loops, query amortization, and cache-line prefetching, up to 4x faster than single-vector alternatives when data exceeds CPU cache.

Search over an Inverted file (IVF) index is where bulk scoring has the most impact. The query selects a set of candidate posting lists and sweeps through the quantized vectors, scoring them in large batches against the query vector. On stateful, those vectors live in one contiguous memory-mapped file, so bulk scoring resolves them with straightforward pointer arithmetic and scores a batch in a single native call.

On serverless, a sweep through a posting list may cross blob cache region boundaries. We extended the direct memory abstraction with a bulk access method that resolves multiple vector offsets to their respective cache regions in a single call. If all vectors in the batch are cached and none cross a region boundary, the scorer gets a direct memory slice and passes the whole batch to simdvec's native bulk kernel with the same prefetching and pipelining as stateful. When a vector does cross a boundary, the system falls back to per-vector scoring: still zero-copy, just without the batching benefit. With 16MB regions and 1024-byte vectors, that happens roughly once every 16,000 vectors.

simdvec's bulk scoring architecture, the key differentiator highlighted in the simdvec benchmarks, now operates on serverless with the same characteristics that make it fast on stateful. So how does it perform in practice?

simdvec on Elasticsearch Serverless: vector search lap times

We benchmarked with an 18 million vector MSMARCO dataset at 1024 dimensions, using IVF with Better Binary Quantization (BBQ) 1-bit quantization. All results are on a warm blob cache with the full dataset resident in local cache regions, so we're measuring the scoring path rather than remote fetch latency.

Throughput. Under concurrent load, search throughput nearly doubled, jumping from 398 to 739 ops/s. Single-client gains were 23-39%, but the real difference shows up under concurrency: The improvement was 2-3x larger because eliminating heap copies removes the GC pressure and allocation contention that previously throttled concurrent scoring.

Tail latency. The direct memory path transformed tail latency under load:

• p99.9 dropped from 237 ms to 30 ms (87% reduction).

• p99.99 dropped from 9.1 seconds to 55 ms (99.4% reduction).

p100 dropped from 11.4 seconds to under 100 ms.

The worst-case outliers that previously took seconds now complete in tens of milliseconds. The heap-copy-induced queueing that caused latency spikes is gone.

Recall is identical. The same vectors are scored, producing the same results. And we're just getting started.

Beyond parity: what Elasticsearch Serverless can do for vector search that stateful can't

Reaching parity with stateful was the goal. But the more interesting realization is what the stateless architecture lets us do that stateful can’t.

On stateful, the OS controls memory-mapped file behavior: which pages stay resident, when to evict, how aggressively to read ahead. The application can offer hints, but they apply to entire file mappings, and the kernel may ignore them. Worse, search and indexing happen concurrently on the same node, so a hint that benefits one access pattern can hurt another. In practice, to balance different needs, you have to be conservative.

On serverless, two things are fundamentally different. The blob cache manages its own memory-mapped regions with full application-level control. And serverless separates indexing and search onto dedicated tiers: Search nodes never merge, indexing nodes never serve queries. No conflicting access patterns means we can be aggressive with memory advice. Here’s what we’re working on:

• Per-region memory advice. The blob cache knows what type of data each region holds. It can issue random-access hints for rescoring regions, where raw float32 vectors are read in unpredictable order and the kernel’s default readahead would waste memory on pages that will never be used. It can apply sequential readahead for scans through quantized vectors. On the indexing tier, merges read data sequentially, so aggressive readahead brings pages in before they're needed, with no risk of harming concurrent random reads that simply aren't happening on that node.

• Cache-aware prefetching. simdvec already prefetches at the CPU cache-line level. On serverless, we can coordinate this with the blob cache's knowledge of region residency, prefetching at multiple levels: remote store to cache, OS pages to RAM, and cache lines to CPU. The blob cache can tell the scorer which regions are resident before scoring begins, avoiding work on data that would trigger a remote fetch.

• Workload-aware eviction. The blob cache can prioritize retaining data that vector search depends on: IVF centroid indexes that are checked on every query or quantized vectors that are scored in bulk, over data that's accessed infrequently. The OS page cache evicts based on generic heuristics with no understanding of what the data represents. On serverless, eviction policy can be tuned to the workload.

The blob cache gives us a level of control over the memory hierarchy that the OS page cache simply can’t. This is why we see serverless as the most promising platform for the next generation of vector search performance work. Not just matching stateful, but surpassing it. And vectors are just the beginning.

Vector search on Elasticsearch Serverless: what we shipped and what's next

simdvec now runs everywhere Elasticsearch runs (stateful, serverless, and frozen tier) with the same native SIMD scoring, the same bulk scoring, and the same off-heap efficiency. The abstraction we built is general-purpose and already wired through every layer in the storage chain, so the same approach could benefit term lookups, aggregations, sorting, and stored field retrieval in the future.

Elasticsearch Serverless is where we're investing most heavily in vector search performance. Every improvement to simdvec, every optimization to the blob cache, and every new storage-level improvement lands here first. If you're choosing where to run your vector workloads, serverless is the platform that keeps getting faster. You can get started with a free Elastic Cloud trial.

This post is a deep dive into the implementation. For context on how synthetic _id fits into the broader metrics performance story, see How we rebuilt Elasticsearch as a leading columnar metrics datastore to achieve up to 6.6x improvement in storage efficiency and 50% improvement in indexing throughput for OpenTelemetry metrics.

We'll start by explaining why the _id field is expensive for time-series workloads. We'll then describe how synthetic _id works and how it uses a bloom filter to optimize document deduplications instead of maintaining a traditional inverted index. Finally, we'll share the performance results from our benchmarks and serverless production deployments.

The hidden cost of _id in time-series indices

Time-series indices are a specialized index mode optimized for metrics, logs, traces, and other timestamped data. They store sequences of data points (like CPU usage, stock prices, or sensor readings) that track changes to specific entities over time. In Elasticsearch, each of these data points is indexed as a document with a unique identifier called _id. This identifier is used to look up, update, or delete specific documents. When a document is indexed in Elasticsearch, the system checks whether a document with the same _id already exists. Depending on the operation type (op_type), an existing document is either replaced (index) or the new document is rejected (create); the latter is the most common path for metrics ingestion.

To perform this lookup efficiently, Elasticsearch builds an inverted index for the _id field. This inverted index maps each _id value to its location in the index, enabling fast document lookups. Until version 8.11, the _id value was also stored separately in order to be returned in search results and other APIs. From 8.11 and onwards, we optimized Elasticsearch to only store this value temporarily for document replication purposes, the value being quickly merged away and reconstructed on demand.

For many use cases, building the inverted index and storing it is an acceptable overhead. But for time-series data, like metrics or traces, the cost can add up quickly. Our experiments showed that building the inverted index for the field _id adds 6% CPU overhead compared to indexing without it. In some extreme cases, we benchmarked that it could reduce indexing throughput by 25%.

This overhead is especially painful for time-series workloads where data points are typically small (often just a timestamp and a few numeric values) and compress extremely well. The _id field, however, doesn't benefit from the same compression. As a result, the inverted index for _id can represent a disproportionate share of the total storage. In our benchmarks with OpenTelemetry (OTel) metrics, the _id inverted index alone consumed around 5 bytes of the total 25 bytes per data point.

We considered several approaches to eliminate this overhead:

• Stop indexing _id and checking for duplicates: This would be the simplest solution, but without deduplication, duplicate data points could corrupt aggregations. A gauge average, for instance, would be skewed by repeated values.

• Accept duplicates during indexing, deduplicate at query time: This preserves correctness but adds overhead to every query, degrading dashboard responsiveness.

• Deduplicate during segment merges: Duplicates would eventually be removed, but queries on unmerged segments would still return results with duplicates.

• Synthetic _id: Compute the document identifier on the fly from fields that already uniquely identify each data point, and use a lightweight bloom filter for deduplication instead of a full inverted index.

We chose synthetic _id because it maintains correctness at ingest time while eliminating the storage and CPU overhead of the traditional approach. And we decided to implement it for time-series indices because they’re very well suited for this optimization.

In time-series indices, the _id isn’t arbitrary. Each document has a time series identifier (_tsid) and a timestamp (@timestamp). The _tsid is generated from the dimensions fields of the document (like host.name, pod.name, or sensor_id), while the @timestamp marks the point in time of the document. Together, these two fields uniquely identify the document: There can only be one data point for a given time series at a given moment in time. This means we can derive the _id from the _tsid and @timestamp field values, rather than storing it separately.

How does synthetic _id work in Elasticsearch?

With synthetic _id, Elasticsearch computes the document identifier on the fly as the combination of the _tsid and @timestamp fields. This computed value is used wherever _id would normally be used: in API responses, for document lookups, and for deduplication. However, it’s never stored in an inverted index nor is it stored on disk for later retrieval.

The challenge is deduplication. When a new document arrives, Elasticsearch must verify that no document with the same _id already exists. Without an inverted index on _id, how can we perform this check efficiently?

How synthetic _id simulates an inverted index without building one

Our Elastic Lucene experts suggested a clever idea: Since _tsid and @timestamp are already stored as doc values, we could expose our own custom Lucene postings format that simulates an inverted index without actually building one.

This means that when Elasticsearch needs to look up a document by its _id, it uses the same code path as usual: It queries the underlying Lucene index to look up the _id term. But instead of hitting a real inverted index, our custom postings format intercepts the call, extracts the _tsid and @timestamp encoded in the synthetic _id, and uses their doc values to locate the document. Because time-series indices are sorted by these fields, documents belonging to the same time series are stored contiguously. This allows Elasticsearch to skip large subsets of nonmatching documents (sometimes entire segments) to find the target document(s) quickly.

While this process is efficient, it can involve several random-access reads: looking up the _tsid value, scanning for matching documents, and reading timestamps. For the common case in time-series indices where we don’t expect the document to already exist, we wanted to fail fast without touching doc values at all.

Bloom filters for fast membership testing

We solve this problem using a bloom filter, a probabilistic data structure that can quickly answer the question Could this element be in the set? with a small risk of false positives but no risk of false negatives. In other words, a bloom filter might occasionally say yes when the answer is actually no, but it will never say no when the answer is yes.

When a document is indexed, its synthetic _id is added to the bloom filter. When a new document arrives, we first check the bloom filter. If the bloom filter says no, we know for certain that no document with this _id exists and we can proceed with indexing immediately. If the bloom filter says maybe yes, we fall back to the more expensive verification using the _tsid and @timestamp doc values.

Synthetic _id indexing workflow: step by step

Let's walk through what happens when a document is indexed into a time-series index with synthetic _id enabled:

1. Compute the synthetic _id: Elasticsearch calculates _id as a combination of _tsid || @timestamp.

2. Check the live version map: Like today, we first check an in-memory map of recently indexed documents. If the document is present in this map, we can handle the duplicate immediately.

3. Filter segments by timestamp: Time-series indices are sorted by _tsid and @timestamp. We can skip any segment whose timestamp range does not overlap with the incoming document's timestamp.

4. Check the bloom filter: For each candidate segment, we test whether the _id might exist using the bloom filter.

5. Verify if needed: If the bloom filter returns a positive result, we look up the document using the _tsid and @timestamp doc values. Since documents are sorted by these fields, this lookup is efficient.

6. Index the document: If no existing version is found, the document is indexed. The _id is added to the segment's bloom filter, but no inverted index is built and the field value is never stored.

In the common case where new data arrives with recent timestamps, step 3 eliminates most segments from consideration, and step 4 quickly confirms that the document is new. The expensive verification in step 5 only happens on bloom filter false positives, which are expected to be rare.

Bloom filter false positive rate: how Elasticsearch keeps it low

One challenge with bloom-filter-based deduplication is controlling the false positive rate without sacrificing the storage efficiency we were after. To size bloom filters effectively, we consider the number of data points in each segment and target both a low false positive rate and a bit set saturation below 50%.

The saturation target serves a specific purpose: When segments are merged, we OR the bit sets rather than rebuilding bloom filters from scratch. This makes merges fast but means the false positive rate converges toward 100% as segments are merged repeatedly. Keeping saturation below 50% before merging buys headroom, delaying that convergence.

The low false positive rate target is justified by access patterns: Recent segments are checked far more often than older ones, since we prune the search space based on data point timestamps. Older, heavily merged segments with degraded bloom filters are unlikely to be checked.

Synthetic _id performance benchmarks: indexing and storage

We ran extensive benchmarks to validate our implementation.

Indexing throughput

A core goal of this effort was to match or improve on existing indexing throughput. In principle, the new approach does less work: Building an inverted index for _id requires hashing each value, building and maintaining complex data structures in memory, and flushing them to disk. These structures must also be reconstructed during segment merges, adding CPU and I/O overhead in high-throughput use cases.

Building a bloom filter isn't free (we still hash each value), but the memory footprint is smaller and there are no complex data structures to maintain or flush. The bloom filter is also cheap to merge: When possible, we simply OR the bit sets together rather than rebuilding from scratch.

The main cost of synthetic _id comes from verifying potential duplicates using doc values. However, this cost is mitigated by two factors: First, bloom filter false positives are rare, so most documents skip this step entirely. Second, time-series indices are sorted by _tsid and @timestamp, which means doc value lookups can skip large blocks of nonmatching documents efficiently.

In practice, that's exactly what we observed. Even accounting for the extra seeks needed to verify matches against the tsid and timestamp when a bloom filter returns a positive, throughput came out comparable or better than before. The savings from not building and merging the inverted index outweigh the occasional cost of a false positive check, as confirmed by our nightly benchmarks:

Storage savings

In our benchmarks with OTel metrics, synthetic _id reduced storage by approximately 5 bytes per data point. For a dataset where documents average 25 bytes per data point, this represents a 20% reduction in storage from this single optimization alone.

These results were soon confirmed by our nightly benchmarks.The chart below shows the storage footprint reduction over time as we enabled the synthetic _id feature on March 19, 2026:

Our standard time series database (TSDB) benchmark showed a reduction from 2.5 GiB to 1.9 GiB (24%). Similarly the time-series downsampling benchmark showed a comparable reduction from 3 GiB to 2.3 GiB (23%).

Another benchmark, more focused on metrics, showed an even better reduction, from 3.0 GiB to 2.0 GiB (34%):

API compatibility

An important design goal was maintaining compatibility with existing Elasticsearch APIs. With synthetic _id, all document APIs continue to work as expected: Bulk, Get, Update, Delete, Reindex, and Update/Delete by Query. This compatibility layer also limited the blast radius of the change, ensuring any issues would be contained to the internal implementation.

When the _id isn’t provided in an API request, Elasticsearch computes it from the _tsid and @timestamp fields. To check if the document already exists, it first queries the bloom filter and, if needed, falls back to doc values. The _id is also synthesized on demand from doc values when returning documents in search results or API responses.

One case that requires special handling is searching or filtering by _id prefix or pattern. Such queries require scanning many documents to find matching documents, and while this works correctly, it incurs a performance penalty compared to a direct _id lookup. We don’t expect this use case to be common for time-series indices though.

Elasticsearch 9.4 and Elastic Cloud Serverless availability

The synthetic _id feature will be released in Elasticsearch 9.4.0 and is already available on Elastic Cloud Serverless.

No configuration is required: The feature is enabled by default, and newly created time-series indices (including those created on datastream rollover) will automatically benefit from this optimization. Existing time-series indices created before 9.4 will continue to create inverted indices for the _id field.

We expect synthetic _id to perform well across all time-series use cases. However, in some very specific, update-heavy use cases, if you encounter performance issues, the feature can be disabled by setting index.mapping.synthetic_id to false for new indices.

Summary: synthetic _id storage and performance gains

In this article, we’ve presented how synthetic _id eliminates the storage and compute overhead of document identifiers in time-series indices. By computing _id on the fly from _tsid and @timestamp, and using a bloom filter for deduplication, we achieve comparable or better indexing performance with up to 34% reduction in storage footprint while maintaining full API compatibility. For users running large-scale time-series workloads, this translates directly into lower infrastructure costs.

Roadmap: what comes after synthetic _id

Synthetic _id is part of a broader effort to reduce storage overhead in Elasticsearch.

• Sequence number trimming: Every document carries a sequence number for replication and concurrency control. For append-only time-series data, these become redundant after segments are merged. Elasticsearch 9.4 now trims them during merges to reclaim even more storage: We'll cover this optimization in detail in an upcoming blog post.

• Synthetic _id beyond time-series: We’re exploring how to bring synthetic _id to regular indices by letting users declare which fields uniquely identify their documents and configuring index sorting on those fields to enable efficient lookups.

Stay tuned!

If agents can call tools on demand and models can reason over massive context windows, why not just fetch data live from every system and let the LLM figure it out?

It’s a reasonable question. It’s also the wrong conclusion.

The reality is that MCP and agent tooling don’t eliminate the need for search. They make the quality of search more critical than ever. In this blog, we’ll explore why MCP, federated search, and large context windows don’t replace search engines and why indexes remain the foundational layer for scalable, accurate, enterprise-grade AI.

What MCP actually is (and what it is not)

MCP is a coordination protocol. It standardizes how an agent requests information or actions from external systems.

What MCP doesn’t do:

• Rank results across systems.

• Understand relevance across heterogeneous data.

• Normalize schemas or metadata.

• Data transformations or enrichments at scale.

• Apply consistent security and permissions.

• Optimize for latency, cost, or scale.

In other words, MCP tells agents how to ask for data, not which data matters most.

Modern retrieval requires query intelligence, not just data access

In modern enterprise search architectures, retrieval quality is determined long before a query reaches an index. Raw queries — especially those generated by agents — may be incomplete, overly literal, schema-driven rather than intent-driven, and at times syntactically invalid.

This is why mature search platforms introduce a query intelligence layer that performs query rewriting, entity normalization, synonym expansion, and intent disambiguation before retrieval even begins.

For example, an agent-generated request such as: “Show severity 2 authentication failures from last sprint” may be rewritten to include authentication synonyms (login, SSO, OAuth), normalized severity mappings, and sprint-to-date-range translation. The result is not just more matches — it is more relevant matches.

In enterprise AI, retrieval is not a single step. It is a controlled pipeline.

This distinction is crucial because once MCP-based agents start pulling information live from multiple tools, they recreate a familiar pattern under a new name: federated search.

MCP-based retrieval is federated search in disguise

Federated search isn’t new. Enterprises have tried it for decades.

The model is simple:

• Send the user’s query to multiple systems in parallel (SharePoint, GitHub, Jira, customer relationship management [CRM]).

• Collect the responses.

• Merge and present the results.

MCP-driven tool calls follow the same pattern, except that the caller is now an agent instead of a user interface.

And the same problems resurface.

Why federated search breaks down at enterprise scale

• Latency becomes unpredictable: A federated query is only as fast as its slowest system. Enterprise systems can have wildly different response times and rate limits, so federated queries tend to be slow and jittery. Agents must wait for multiple round trips before reasoning can even begin. The result is a laggy experience and unpredictable wait times.

• Relevance is fragmented: Because each system ranks results on its own, there’s no unified relevance model. Federated search cannot apply a single ranking or semantic understanding across all content, so results often seem disjointed or incomplete. Agents may retrieve correct information but not the most useful information.

• Context is shallow and incomplete: Federated systems typically expose only what’s directly accessible through an API call.They rarely surface:

  • Usage signals, like clicks, dwell time, recency of access, popularity, or authority.

  • Relationships between documents across different systems to correlate the insights.

  • Organizational knowledge beyond a single silo. This strips agents of the broader context required for high-quality reasoning.

• Limited filtering and features: In a federated setup, you can only filter on fields that every system supports (the “lowest common denominator”). If one system doesn’t support a particular filter or facet, you lose that functionality entirely. This severely limits rich search features, like date ranges, categories, or tags.

The power of an indexed search

Search engines achieve millisecond-level retrieval at massive scale by using specialized data structures, including inverted indexes for lexical search and k‑dimensional trees (k-d trees) for vector-based retrieval. The approach is to crawl or ingest every source into search engines, creating a central place of company knowledge. This brings big advantages:

• Speed by design: Searching an index is lightning fast. Queries hit inverted indexes and specialized data structures, avoiding the need to poll each backend system.

• Relevance that compounds over time: Search engines that support semantic search are capable of comprehending the intent, and machine learning models can rerank results for enterprise contexts. In one Elastic experiment, Elastic users see more accurate results when combining vector search with a question-answering (QA) model to extract answers. It gives better precision than keyword matching.

• Advanced features: Elastic’s Graph retrieval augmented generation (RAG) solution shows how structuring an index as a knowledge graph can power more contextual retrieval. In other words, indexes aren’t just backward-looking dumps of text; they can also encode relationships and ontologies that let AI connect the dots across documents.

• Permission-aware search: Enterprise AI cannot compromise on security. Indexed search allows:

  • Document-level security.

  • Role-based access control.

  • Permission-aware retrieval for RAG and agents.

Agents see only what users are allowed to see, without leaking data into model prompts or training. Elasticsearch is suitable for the indexed search layer in the diagram below, as it provides the essential components for context engineering.

Retrieval consistency through search templates and governed execution

At scale, retrieval must be predictable, secure, and repeatable. This is where search templates become critical.

Search templates act as retrieval contracts between applications, agents, and the search platform. Instead of dynamically constructing queries at runtime, agents invoke pre-defined retrieval patterns that enforce:

• Consistent relevance logic

• Mandatory security filters

• Cost and latency guardrails

• Business-specific ranking rules

• Explicit index and field scope boundaries

In MCP-driven architectures, this becomes even more important. Agents should not dynamically invent retrieval strategies. Instead, MCP tool calls can map directly to approved search templates, ensuring that every retrieval request adheres to enterprise relevance and governance standards.

This approach shifts retrieval from ad-hoc query execution to controlled retrieval orchestration.

Retrieval is now a multi-layer engineering discipline

Modern enterprise retrieval is no longer a simple query-to-index operation. It typically includes multiple coordinated layers:

• Query understanding — rewriting, expansion, entity resolution

• Retrieval strategy selection — hybrid search, vector search, graph retrieval, or synthetic query techniques such as Hypothetical Document Embeddings (HyDE), where the system generates a representative answer or expanded context first and retrieves documents using that richer semantic signal.

• Execution governance — templates, security enforcement, and performance guardrails

• Ranking and re-ranking — blending lexical precision, semantic similarity, and interaction-derived relevance signals such as click-through patterns, dwell time, and document usage frequency.

When these layers are implemented upstream, agents receive clean, high-confidence context rather than raw, fragmented data.

This is what makes large-scale agent systems reliable in production environments.

Advanced retrieval techniques improve context quality before reasoning begins

Modern retrieval systems increasingly use AI-assisted techniques to improve recall and semantic coverage before ranking is applied.

One example is Hypothetical Document Embeddings (HyDE). Instead of embedding only the original query, the system first generates a hypothetical answer or expanded context, embeds that representation, and retrieves documents based on that richer semantic signal.

This is particularly useful in enterprise environments where:

• Users or agents may not know the exact terminology

• Knowledge is distributed across silos

• Important context is implied rather than explicitly stated

Techniques like HyDE improve the probability that relevant documents are retrieved even when the original query is underspecified.

This reinforces a key principle of enterprise AI: better context retrieval produces better reasoning outcomes.

Agents aren’t data engineers; they’re reasoning systems

They shouldn’t be responsible for stitching together raw data, reconciling schemas, or compensating for poor retrieval.

This is where a search platform such as Elasticsearch becomes foundational.

By ingesting data once and normalizing it upstream (through pipelines, mappings, enrichment processors, and prebuilt indexes), Elasticsearch resolves schema mismatches, joins signals across sources, and materializes retrieval-ready views of the data. At query time, the agent receives clean, ranked, semantically enriched results rather than fragmented raw records.

For example, instead of an agent pulling independently from CRM, ticketing, and documentation systems and attempting to reconcile customer IDs, timestamps, and formats in real time, Elasticsearch can pre-index these sources into a unified customer interaction index with hybrid (keyword + vector) search and relevance ranking. The agent then queries a single, coherent interface and immediately reasons over the most relevant context.

This separation of concerns, that is, Elasticsearch handling data integration and retrieval, and agents focusing on reasoning, planning, and decision-making,is what makes agent systems scalable, reliable, and production ready.

Elastic’s role in the AI stack

Elastic sits at the intersection of search and AI by design.

• Connectors and crawlers ingest data continuously from enterprise systems.

• Semantic and vector search enable intent-based retrieval.

• Hybrid search blends lexical precision with semantic understanding.

• RAG workflows ground LLMs in authoritative, permission-aware data.

Elastic does not compete with agents or MCP. It makes them effective.

Bigger models don’t eliminate retrieval

Some have wondered whether huge new LLMs can bypass traditional search, perhaps by letting the model read everything in one go. Large context windows feel powerful, but they introduce:

• Higher latency.

• Higher cost.

• Lower precision due to noise.

• A higher propensity for confusion, context clash, and context poisoning.

RAG wins because it filters first and then reasons.In another Elastic Search Labs experiment, RAG achieved answers in about 1 second, versus 45 seconds for the raw-LM approach, at 1/1250th the cost, and with far higher accuracy. In other words, giving an LLM a million tokens of documents is slower, more expensive, and actually less precise than filtering through an index first.

Conclusion: MCP changes the interface, not the fundamentals

MCP is a meaningful step forward in how agents interact with tools. But it doesn’t replace the need for fast, relevant, governed retrieval.

In enterprise AI:

• Context quality determines answer quality.

• Indexes create that context.

• Search is the foundation, not the legacy.

Indexes aren’t obsolete in the era of MCP. They’re the reason that MCP-based agents can work at all.

In HNSW, search proceeds by iteratively expanding candidate nodes in the graph, maintaining a bounded set of nearest neighbors discovered so far. Each expansion has a cost (vector operations, random seeks to disk, and more), and the marginal benefit of that cost tends to decrease as the search progresses.

One way to optimize HNSW graph traversal is to stop searching when the marginal likelihood of finding new true neighbors doesn’t increase. For this reason, in Elasticsearch 9.2 we introduced a new early termination mechanism. This stops the search process when visiting graph nodes doesn’t provide enough new nearest neighbors, consecutively, for a fixed number of times.

This article guides you through how we improved over the mentioned early termination mechanism in HNSW to make it better suited for different datasets and data distributions.

Early termination in HNSW

In HNSW, search proceeds by iteratively expanding candidate nodes in the proximity graph, maintaining a bounded set of nearest neighbors discovered so far, until it either has visited the whole graph or meets some early stop criteria.

Early termination is therefore not necessarily always an optimization, it’s part of the search algorithm itself. The moment we decide to stop determines the balance between efficiency and recall. In Elasticsearch, there are already a number of ways a query on HNSW can early terminate:

• A fixed maximum number of nodes is visited.

• A fixed timeout is reached.

While simple and predictable, these rules are largely agnostic to what the search is actually doing. Also they’re used mostly to make sure that the query finishes in reasonable time for the end user.

In a previous blogpost, we introduced the concept of redundancy in HNSW. In short, redundant computations occur when HNSW continues to evaluate new candidate nodes that don’t result in finding more nearest neighbors.

Patience: Measuring progress instead of effort

The notion of patience reframes early termination around progress rather than effort.

Instead of asking:

“How many steps have we taken?”

The new question becomes:

“What is the amount of computation we accept to waste, until we lose hope?”

During HNSW search, early exploration typically produces peak improvements to the top-k candidate set. During first steps of the HNSW graph exploration, the set of neighbors is continuously updated as the algorithm keeps discovering nearer and nearer neighbors to the query vector. Over time, these improvements become rarer as the search converges. Patience-based termination monitors this pattern and terminates the search once improvements have ceased for a sustained period.

In practice, while visiting the HNSW graph we also compute the queue saturation ratio as we hop through candidate nodes. This measures the percentage of nearest neighbors that were left unchanged while visiting the most recent graph node (or the inverse of the number of new neighbors introduced during the last iteration). When such a ratio becomes too big for too many consecutive iterations, we stop visiting the graph.

Conceptually, patience treats HNSW search as a diminishing returns process. When returns flatten out, continuing to explore the graph yields little benefit.

This framing is powerful because it ties termination directly to observable outcomes rather than to arbitrary fixed limits.

The benefit of using this smart early termination technique is that HNSW graph explorations tend to visit a smaller number of graph nodes while retaining an almost perfect relative recall.

To visualize this, we can plot the amount of recall per visited node that we got with the patience based early termination (labeled as et=static), when compared to the default HNSW behavior (labeled as et=no) on a couple of datasets, FinancialQA and Quora, and models, JinaV3 and E5-small.

Static thresholds and HNSW dynamics

In practice, in Elasticsearch this is implemented using static thresholds. One threshold refers to the saturation threshold: that is, the ratio of saturation that we consider suboptimal. The other threshold refers to the number of consecutive graph nodes that we allow to be visited while still having a suboptimal queue saturation: that is, the patience threshold.

When we introduced this early termination strategy in Elasticsearch 9.2, we decided to opt for conservative defaults, so as to let the recall as much as possible, while still gaining in terms of latency and memory consumption. For this reason, we set the saturation threshold to be 100% and the patience threshold to be set as a (bounded) 30% of the num_candidates in the KNN query.

In many scenarios, these settings resulted to work nicely; however, two queries requesting the same number of neighbors might have radically different convergence behaviors. Some queries encounter dense local neighborhoods and saturate quickly; others must traverse long, sparse paths before finding competitive candidates. The latter resulted to be the most difficult to handle effectively.

As a result, we sometimes noticed:

• Over-exploration for easy queries.

• Premature termination for hard queries.

Therefore, we figured that fixed threshold values encode global assumptions about convergence, whereas we could make HNSW better adapt to different dynamics.

Making HNSW early termination adaptive

Adaptive early termination approaches this problem from a different angle. Instead of enforcing predefined stopping thresholds, the algorithm infers when to stop from the search dynamics themselves.

So instead of comparing the queue saturation ratio between two consecutive candidates, we decided to introduce both an instant smoothed discovery rate (how many new neighbors were introduced for a query q, in the last visit i) together with rolling mean and standard deviation of such a discovery rate during the graph visit (using Welford’s algorithm). These statistics about the discovery rate are calculated per query, so that this information can be used to decide different degrees of patience for each query.

The previously static thresholds become adaptive to the discovery rate statistics: The saturation threshold becomes the rolling mean plus the standard deviation; whereas we make the patience adapt and scale inversely with the standard deviation.

The early exit rules remain the same; the saturation happens when the instant discovery rate is lower than the adaptive saturation threshold. The graph visit stops if the saturation persists for a number of consecutive candidate visits that’s larger than the adaptive patience.

This way, we obtain a behavior that doesn’t depend on the num_candidates parameter in the KNN query (which might be always set or left as the default, regardless of early exit) and that better adapts to each query and vector distribution dynamically.

The recall per visited node on FinancialQA and Quora with the adaptive strategy (labeled as et=adaptive) reports a higher recall per visited node, when compared to the static strategy (et=static) and the default HNSW behavior (et=no).

Adaptive early termination is turned on by default in Elasticsearch 9.3 for HNSW dense vector fields (and it can eventually be turned off via the same index level setting).

This change reduces the overhead of parsing vectors in JSON by an order of magnitude, which translates to almost a 100% improvement on indexing throughput for DiskBBQ and around 20% improvement for hierarchical navigable small world (HNSW) workloads. In this blog, we’ll take a closer look at Base64-encoded strings and the improvements it brings to vector ingestion.

What’s the problem?

At Elastic, we’re always looking for ways to improve our vector search capabilities, whether that’s enhancing existing storage formats or introducing new ones. Recently, for example, we added a new disk-friendly storage format called DiskBBQ and enabled vector indexing with NVIDIA cuVS.

In both cases, we expected to see major gains in ingestion speed. However, once these changes were fully integrated into Elasticsearch, the improvements weren’t as large as we had hoped. A flamegraph of the ingestion process made the issue clear: JSON parsing had become one of the main bottlenecks.

Parsing JSON requires walking through every element in the arrays and converting numbers from text format into 32-bit floating-point values, which is very expensive.

Why Base64-encoded strings?

The most efficient way to parse vectors is directly from their binary representation, where each element uses a 32-bit floating-point value. However, JSON is a text-based format, and the way to include binary data in it is by using Base64-encoded strings. Base64 is just a binary-to-text encoding schema.

We can now send vectors encoded as Base64 strings:

Is it worth it? Our benchmarks suggest yes. When parsing 1,000 JSON documents, using Base64 encoded strings instead of float arrays resulted in performance improvements of more than an order of magnitude, at the cost of a small encode/decode trade-off (client-side Base64 encoding and a temporary byte array on the server for decoding) in exchange for eliminating expensive per-element numeric parsing.

Give me some ingestion numbers

We can see these improvements in practice when running the so_vector rally track with the different approaches. The actual gains depend on how fast indexing is for each storage format. For bbq_disk, indexing throughput increases by about 100%, while for bbq_hnsw, the improvement is closer to 20%, since indexing is inherently slower there.

Starting with Elasticsearch v9.2, vectors are excluded from _source by default and are stored internally as 32-bit floating-point values. This behavior also applies to Base64-encoded vectors, making the choice of indexing format completely transparent at search time.

Client support

Adding a new format for indexing vectors might require changes on ingestion pipelines. To help this effort, in v9.3, Elasticsearch official clients can transform vectors with 32-bit floating-point values into Base64-encoded strings and the other way around. You might need to check the client documentation for the specific implementation.

For example, here’s a snippet for implementing bulk loading using the Python client:

The only difference from a bulk ingest using floats is that the embedding is wrapped with the pack_dense_vector() auxiliary function.

Conclusion

By switching from JSON float arrays to Base64-encoded vectors, we remove one of the largest remaining bottlenecks in Elasticsearch’s vector ingestion pipeline: numeric parsing. The result is a simple change with outsized impact: up to 2× higher throughput for DiskBBQ workloads and meaningful gains even for slower indexing strategies, like HNSW.

Because vectors are already stored internally in a binary format and excluded from _source by default, this improvement is completely transparent at search time. With official client support landing in v9.3, adopting Base64 encoding requires only minimal changes to existing ingestion code, while delivering immediate performance benefits.

If you’re indexing large volumes of embeddings, especially in high-throughput or serverless environments, Base64-encoded vectors are now the fastest and most efficient way to get your data into Elasticsearch.Those interested in the implementation details can follow the related Elasticsearch issues and pull requests: #111281 and #135943.

The most intuitive levers for doing this are boosted should clauses or rank_feature fields. These may initially appear effective, but both approaches degrade and may even fail, as query patterns shift or catalog composition changes. Their shared limitation is that they introduce additive adjustments into a scoring system whose scale varies substantially across queries. A boost like “+2” might overwhelm the base BM25 score in one query while barely registering in another. In other words, additive methods may create brittle, unpredictable ranking behavior.

In contrast, function_score with multiplicative boosting provides a stable and mathematically proportional way to shape BM25 scores without distorting their underlying structure. Your application logic determines what merits uplift; function_score expresses that intent in a predictable and explainable way that preserves the geometry (high-level relative ordering) of the BM25 relevance signal, nudging rankings in controlled ways rather than overwhelming the core text relevance.

This article builds on two earlier pieces that demonstrated practical uses of multiplicative boosting: (1) Boosting e-commerce search by profit and popularity with the function score query in Elasticsearch, and (2) How to improve e-commerce search relevance with personalized cohort-aware ranking. Here we step back from those examples to examine the architectural principle that underlies them: why multiplicative boosting via function_score is one of the most reliable and scalable ways to influence BM25-based ranking in Elasticsearch.

Why it's important to preserve base BM25 rankings

In many Elasticsearch-based applications, including e-commerce, BM25 remains a central component of how text relevance is assessed. It provides a signal that is interpretable and transparent for teams who need to understand why a product ranked where it did. These properties make BM25 particularly attractive in environments where explainability and operational predictability matter.

Because of this, most teams want to shape, rather than replace, the rankings produced by BM25. For example, they may want to allow higher-margin items to surface slightly more often, reduce exposure for low-stock products without hiding them, or highlight items aligned with a particular user segment. Ideally, this shaping should preserve the geometry of the rankings produced by the BM25 algorithm.

The difficulty arises when teams try to achieve these goals using mechanisms that add separate scoring streams on top of the base BM25 ranking. These additive adjustments are not always comparable to BM25’s scale and behave inconsistently as queries, data distributions, and catalog composition evolve. Over time, the ranking becomes brittle, unintuitive, and difficult to tune. A reliable influence mechanism must work with BM25’s scoring geometry rather than overpowering it.

The function_score query with multiplicative boosting provides this property. It allows teams to apply business influence in a proportional, explainable way while keeping BM25’s underlying structure intact.

Why many approaches to influencing ranking degrade (or break) BM25

Teams often begin with mechanisms that look straightforward: boosted should clauses, rank_feature fields, or custom script_score logic. These tools can be effective in their intended use cases, which is why they seem like natural levers for adding business influence. But when they are used to shape or influence BM25-based text relevance, they may create unstable, opaque, or brittle ranking behavior.

The underlying issue is that these approaches introduce independent additive scoring contributions into a system whose base BM25 values vary widely across queries, fields, and data sets. Without respecting that variability, the influence becomes unpredictable.

Below are the three most common patterns and why they fail in practice.

1. Additive boosts via should clauses

A boosted should clause feels intuitive: “Promote items that match this business rule.” But under the hood, the behavior is fundamentally additive.

Consider a query of the form:

This kind of query results in the following behavior:

The problem is that base_BM25 and should_BM25 do not scale together. As your dataset changes, or as different queries are issued, the magnitude of BM25 can shift dramatically. For example, the base BM25 scores for three products might be 12, 8, 4 in one context, and 0.12, 0.08, 0.04 in another. Such a change might happen after a catalog update or a modification to the query structure.

A boosted should clause adds its own BM25-style contribution to the final score. In this situation, an additive contribution (i.e. should_BM25 = +2) behaves inconsistently:

• When base_BM25 is small (0.12), +2 dominates the score — roughly an 18× increase.

• When base_BM25 is large (12), the same +2 barely shifts the document — only about a 17% increase.

This instability means that the combined must score and should score have no stable meaning across queries or catalogs. A rule that slightly promotes a brand for one query can dominate the ranking for another, or become irrelevant in a third. This is not a tuning issue; it is a structural property of additive scoring.

2. Using rank_feature for business influence

The rank_feature family is extremely useful for representing numeric qualities such as recency or popularity. It is fast, compressed, and operationally simple. However, when it is used to influence text relevance (BM25), it runs into the same structural limitation described in the previous section.

A rank_feature clause produces its own scoring contribution, which is then added to the BM25 score:

Just as with boosted should clauses, the two components do not scale together. BM25 values vary substantially across queries depending on term rarity and catalog statistics, while the feature_score follows the scale of the underlying business attribute being boosted (for example, popularity or recency), which typically bears no relationship to the scale of BM25. As a result, the two scoring streams drift apart as your corpus or query patterns evolve.

The consequence is the same as what we discussed above with relation to the should-clause problem:

• The feature score can dominate BM25 in one query and be negligible in another.

• Tuning becomes fragile because you are calibrating two independent scales — BM25, which varies with query term statistics, and the feature score, which varies with the business attribute’s own distribution.

Although rank_feature remains an excellent mechanism for representing raw numeric attributes, it is not well-suited for proportional influence on BM25, where the goal is not to add a second score but to gently shape the existing one.

Custom scoring with script_score

When boosted clauses or rank_feature fields become difficult to tune, teams often turn to script_score as a last resort. It provides complete freedom to manipulate the score, including adding, subtracting, multiplying, or replacing the BM25 value according to any business rule. A script_score query replaces Elasticsearch’s scoring pipeline with custom logic. Instead of shaping the BM25 score, the script builds a separate scoring mechanism whose behavior depends entirely on the code inside the script. While this can be powerful, it introduces three challenges that become more significant as the system grows.

1. Opacity

Scoring logic is hidden inside a script rather than expressed declaratively. When ranking behavior changes unexpectedly, it is difficult to understand whether the issue is the script itself, a data shift, or an interaction with BM25. Merchandisers and relevance engineers lose the ability to reason about why a document moved up or down.

2. Performance and operational cost

Script scoring bypasses many of Elasticsearch’s optimizations and caching pathways. Each document that matches the initial query must execute the script, often leading to higher CPU usage and unpredictable latency.

3. Fragility when combined with BM25

Because script_score allows arbitrary computations, it is easy to drift into scoring behaviors that no longer resemble BM25 or that fail to preserve its relative structure. As the dataset evolves or query patterns shift, the custom logic may interact with BM25 in unanticipated ways. A script that behaved reasonably early in development can produce surprising or unstable results once the catalog grows or data distributions change. Because script_score allows arbitrary math, two engineers working on different parts of the system may unintentionally encode competing scoring models, making ranking difficult to reason about as the organization scales.

How function_score provides predictable influence on BM25

BM25 already captures how well a document matches a query. It reflects text relevance, term rarity, document length, and the statistical shape of the corpus. When teams introduce business signals including margin, stock levels, popularity, personalization, or merchandising strategy, the goal is not to replace this relevance. The goal is to influence it.

This distinction is subtle but crucial. Most business requirements are proportional in nature:

• Promote higher-margin items modestly

• Reduce exposure for low-stock products, but don’t hide them

• Give this user segment a slight uplift for matching products

• Boost for popularity, but not so much that textual relevance is lost

These are naturally expressed as percentage adjustments rather than as fixed additive values. A merchandiser is rarely asking for “+2 points of score”; they are asking for “a little more visibility,” irrespective of the absolute numeric scale of the BM25 score. Mathematically, this means that the desired transformation is:

Where boost_factor might be 1.05, 1.2, or 1.5, depending on the signal. Multiplicative boosting does not attempt to reinvent scoring; it simply adjusts the BM25 output by a proportional factor. A multiplicative adjustment has three properties that align well with real-world ranking control:

1. The boost remains proportional. In other words, a 20% uplift is always a 20% uplift—whether BM25 is 0.12 or 12. The magnitude of the boost does not depend on the underlying BM25 scale.

2. BM25 retains its role as the primary signal. The multiplicative shaping nudges the ordering without overriding it. Strong textual matches still win; business logic influences but does not dominate.

3. Because the operation is multiplicative, not additive, changing the query or updating the corpus does not require re-tuning numeric constants. The boost has the same meaning everywhere.

Elasticsearch’s function_score query provides an elegant mechanism for expressing this pattern. By using:

• score_mode: “sum” to assemble a boost factor (building the multiplier), and

• boost_mode: “multiply” to apply the boost (multiplier) to BM25

You can express business intent in a way that remains stable and explainable as your data and query patterns evolve. Instead of adding a second score beside BM25, function_score transforms BM25 itself—shaping it gently, predictably, and in line with how merchandisers and product owners think about ranking adjustments.

Examples in practice: How multiplicative boosting behaves in real e-commerce queries

To illustrate how multiplicative boosting works in real-world ranking scenarios, it helps to look at a small, concrete example. The goal here is not to demonstrate tuning or production-scale scoring, but rather to show how function_score influences BM25 in predictable, proportional ways that align with business intent.

Consider a simple catalog with three basketball shoes from three different brands: Nike, Adidas, and Reebok. The product descriptions are intentionally crafted so the BM25 scores exhibit natural differences based on query specificity and field length—just as they would in a real catalog.

Example dataset

For the following examples, we use a small, straightforward sample dataset with the following characteristics.

Brand

Description

nike

“Nike basketball shoes”

adidas

“New Adidas basketball shoes”

reebok

“Reebok basketball shoes”

We can create an index with the above products with the following commands from Kibana Dev Tools:

With this dataset, we now evaluate three queries:

• A baseline “basketball shoes” search

• The same query with a 50% promotion for Adidas and a 25% promotion for Nike

• A specific “Reebok basketball shoes” query while the Adidas and Nike promotions are still active

Each scenario highlights a different property of multiplicative boosting.

1. Baseline ranking: No promotion

This query returns the following results where Nike and Reebok are ranked above adidas:

Rank

Brand

Score (BM25)

1/2 (tie)

nike

0.27845407

1/2 (tie)

reebok

0.27845407

3

adidas

0.24686474

2. Adding 50% Adidas uplift and 25% Nike uplift with function_score

If marketing launches a campaign where Adidas basketball shoes should receive a 50% uplift and Nike a 25% uplift, then the application layer could construct its queries to include those uplifts as follows:

How the multiplier is constructed

• Base weight = 1.0

• Adidas gets an additional +0.5

• So Adidas’s multiplier = 1.5

• Nike gets an additional +0.25

• So Nike’s multiplier = 1.25

• All other brands (including Reebok) get the base weight multiplier = 1.0

Apply multiplier:

Final score = BM25 × multiplier

Product

BM25

Multiplier

Final score

Adidas

0.24686474

1.5

0.37029710

Nike

0.27845407

1.25

0.34806758

Reebok

0.27845407

1.0

0.27845407

Result

Adidas moves to the top, Nike follows, and Reebok is at the bottom with no change in its score. This is exactly the behavior that multiplicative boosting is designed to produce:

• Adidas and Nike both gain visibility, but in proportion to their configured uplifts.

• The relative differences in BM25 still matter; we are reshaping the ranking, not replacing it.

• The ordering changes primarily where BM25 scores are close.

With additive boosts, the same “50% versus 25%” business intent would have to be approximated with numeric constants on an arbitrary BM25 scale, and the effect would vary drastically across queries.

3. Specific intent still wins: “Reebok basketball shoes”

Now run a highly specific branded query for “Reebok basketball shoes”, with the same Adidas (50%) and Nike (25%) promotions still active:

The response shows the following results:

Rank

Brand

Final score

1

reebok

1.3011196

2

adidas

0.3702971

3

nike

0.34806758

Result

Reebok wins overwhelmingly because BM25 correctly detects strong intent for “Reebok basketball shoes”. Adidas and Nike still receive their 50% and 25% promotions, respectively, but those multipliers are nowhere near enough to override the BM25 score.

This is exactly the behavior that multiplicative boosting is designed to produce:

• When BM25 scores are close, boosts can shift the relative ordering.

• When BM25 scores differ significantly (as they do here, due to strong text matching), the same boosts have little practical effect.

Promotions influence the ranking, but they do not override the core text relevance signal.

What this example demonstrates

These real queries illustrate the key properties of multiplicative boosting:

1. The influence is proportional, not arbitrary. A percentage-based uplift has the same proportional effect regardless of the underlying BM25 scale.

2. Text relevance remains in control. Strong brand-intent queries still surface the correct product.The system behaves intuitively. Merchandisers see exactly the ranking changes they expect.

3. The math is stable across queries. The same promotion works correctly whether the match is broad or highly specific.

4. Application logic stays clean. The business layer decides the uplift; Elasticsearch applies it predictably.

Multiplicative boosting through function_score preserves relevance in a predictable and controllable way, while enabling business impact.

Application logic remains the author of influence

There is a clear separation between deciding what should be boosted and applying that boost in Elasticsearch. function_score handles the second task, but the first belongs firmly to application logic.

Your application logic is where decisions are made about:

• Which margin thresholds matter for your business

• Whether popularity should rise or fall based on seasonality

• How to interpret customer behavior or cohort membership

• How to encode campaign rules

• When to surface or suppress certain product groups

These are business decisions, not scoring decisions. Elasticsearch does not infer whether a user is budget-focused or luxury-oriented, whether a promotion is active, or whether low stock requires a visibility adjustment. Those determinations occur upstream, in the part of the system that has access to user context, session features, analytics, and business configuration. After application logic produces clear numeric signals for fields such as weights, uplift factors, thresholds, and cohort tags, a function_score query provides a reliable way to express those signals as controlled multipliers on BM25.

This creates a clean architectural contract:

• Application logic: decides what should be influenced.

• BM25 provides the core text relevance.

• function_score applies influence in a mathematically stable way.

Because business logic lives outside the index, teams can adjust or experiment with uplift strategies without reindexing or restructuring documents.

Conclusion

E-commerce search must balance core text relevance with business considerations such as profitability, stock position, customer intent, seasonality, and personalization. BM25 provides a stable and interpretable foundation for text relevance, but influencing that score requires care. Business signals should shape the ranking, not overpower it.

However, the most commonly used levers such as boosted should clauses, rank_feature fields, and ad-hoc script scoring often behave unpredictably. These approaches can appear effective in early development, but their limitations emerge as soon as the catalog evolves or new query patterns arrive. Additive boosts fluctuate wildly because their impact depends entirely on the underlying scale of BM25, which varies dramatically across queries. A boost that produces a subtle nudge in one situation can dominate the ordering in another. Script scoring introduces its own challenges: opaque logic, reduced performance, and scoring behavior that becomes harder to understand or maintain over time.

Multiplicative boosting with function_score avoids these pitfalls by transforming BM25 proportionally rather than competing with it. Instead of adding a second, independent score component, it applies a controlled multiplier to BM25 itself. This produces the kind of predictable adjustments that merchandisers actually intend. For example, it allows slight promotions for high-margin items, modest reductions for low-stock products, or gentle uplifts for relevant user cohorts.

Equally important, the architecture remains clean. Application logic determines which business signals matter, and function_score applies them in a consistent, explainable way. Business teams can evolve business strategy without destabilizing relevance, and Engineering teams can refine relevance without disturbing business rules.

This principle is the foundation of the previous blogs that demonstrated how to influence e-commerce rankings: (1) Boosting e-commerce search by profit and popularity with the function score query in Elasticsearch, and (2) How to improve e-commerce search relevance with personalized cohort-aware ranking. Both approaches rely on the idea that business signals should guide BM25, not override it. Multiplicative boosting through function_score provides a practical, transparent, and scalable method for achieving that balance in real-world e-commerce search.

When we built Elasticsearch Serverless, we had to decide how to bill our users. While a charge per query may have been easier to reason about from a consumption perspective, it would be a lot harder to reason about from a resource perspective. Instead, we implemented a simple pricing scheme comprising three dimensions for compute: search, ingest, and machine learning VCUs. This means we charge users for the actual resources we allocate to fulfill your requested workloads.

VCU, ECU, and other terms

Let's start by defining a few terms that will keep coming back throughout this post.

VCU

A VCU is a Virtual Compute Unit, representing a fraction of RAM, CPU, and local disk for caching. We separate compute by the workloads they support, so we have three flavors of VCU:

1. Search VCU

2. Ingest VCU

3. Machine Learning (ML) VCU

VCU’s are charged by the hour.

Regional pricing

We have different prices for different regions and different cloud providers. You can find a full list of prices on this page.

ECU

An ECU is an Elastic Consumption Unit, which is the unit we bill you in. The nominal value of an ECU is $1.00 USD. All of the different components of consumption are charged at a specific rate of ECUs per time unit. For example, one Gigabyte of storage might cost 0.047 ECU per month, so 100 GB of storage will cost you 4.7 ECU = $4.70 for one month. Similarly, if your search workload consumed 10 VCUs in a day and the Search VCU rate in your region is 0.09 ECU, your cost for that day would be $0.90.

Interactive Dataset Size

The amount of data in your project has a direct influence on your costs. We make the distinction of “interactive dataset” primarily for time-series data, as this relates to the amount of data in the Boost Window. For non-time-series data, this is simply the amount of data in the project.

Project settings

We have three project settings that allow you to control your project's usage.

Search power

Search Power controls the speed of searches against your data. With Search Power, you can improve search performance by adding more resources for querying, or you can reduce provisioned resources to cut costs. Choose from three Search Power settings:

On-demand: Autoscales based on data and search load, with a lower minimum baseline for resource use. This flexibility results in more variable query latency and reduced maximum throughput.

Performant: Delivers consistently low latency and autoscales to accommodate moderately high query throughput.

High-availability: Optimized for high-throughput scenarios, autoscaling to maintain query latency even at very high query volumes.

Boost window

For time series use cases, the boost window is the number of days of data that constitutes your interactive dataset size. The interactive dataset is the portion of your data that we keep cached, and that we use to determine how to scale the Search tier for your project. By default, the boost window is seven days.

Data retention

You can set the number of days of data that are retained in your project, which will affect the amount of storage we need. You can do this on a per-data stream basis in your project.

Price components

Serverless Elasticsearch contains a few different pricing components. For most use cases, the components you will care most about are Search, Ingest, and ML VCUs, as well as the Elastic Inference Service's token consumption.

Search VCUs

Search VCU consumption is the most complex part of pricing. We make this simple for you by automatically determining the right amount of VCUs that are needed to fulfill your workloads. For more details on how our autoscaling logic works, see our earlier blog on the topic.

Search VCU inputs

Search VCUs are allocated based on a few factors, but mainly, we can boil it down to three inputs: the interactive dataset size, the search load on the system, and Search Power.

For traditional search use cases, the interactive dataset size will generally be your entire dataset. For time series use cases, it will be the portion of your dataset that fits inside the Boost Window.

Search load measures the amount of load being placed on the system by currently active searches. The main contributing factors are the number of searches per second, the complexity of the searches (the more that needs to be computed, the higher the load), and the size of the dataset that needs to be searched to fulfill the result. If we can get you the right number of results by scanning 10% of the dataset, then the load will be much lower than if we need to scan the full dataset.

Finally, Search Power influences the number of VCUs we allocate. Each Search Power setting defines the baseline capacity of the search tier.

In short: the larger the dataset size and the higher the search load, the more VCUs we need to fulfill your search requests. Search Power allows you to tune to what extent we will scale up and down.

Minimum VCUs

Elasticsearch Serverless is designed to align infrastructure costs directly with your application's demand.

For smaller workloads, the search infrastructure can scale down to zero VCUs during periods of inactivity. If the system detects fifteen minutes of total inactivity, the associated hardware resources are deprovisioned. This makes the platform highly cost-effective for development environments, bursty workloads, or applications with intermittent usage. Note that inactivity means actual inactivity: no user-initiated searches whatsoever. As soon as we need to serve a search of any kind, we need to allocate hardware resources to execute that search.

As your interactive dataset grows, the system eventually reaches a storage threshold where a baseline level of resources is required to maintain data availability and indexing readiness. A minimum VCU allocation is maintained to ensure your data remains "warm" and queryable, even if no active searches are occurring.

VCU consumption is not linear

Because our hardware is allocated in steps, consumption of VCUs does not necessarily scale linearly with workload size. Each scaling step can contain a wide range of workloads, and if your workload is at the bottom of that range, it may have a lot of room to grow before we need to jump to the next scaling step.

This can make estimating based on a non-representative workload hard. For example, you may be consuming 2 VCUs per hour on a small workload. It's entirely possible that you could increase your workload size by a factor of 100 and still fit in that 2 VCU per hour load before we need to start increasing the amount of VCUs we allocate to serve your workload.

We know this makes estimating your cost a little harder, and we are working on ways to make that easier for you. If you need more help estimating your likely price, you can always talk to our customer team and get more personalized assistance.

Ingest VCUs

Ingest VCUs are much simpler than Search VCUs.

Ingest VCU Inputs

Ingest VCUs have essentially three inputs: the number of indices, the ingest rate, and the ingest complexity. We need to allocate a little bit of memory for every index in your system, which is why the number of indices matters. Read indices in data streams do not count for this calculation.

The faster you ingest, the more CPU we will need to process that ingestion. And the more complex your ingest requests, the more CPU we will need. Some factors that make ingest requests more expensive to execute are complicated field mappings or a lot of post-processing.

Minimum Ingest VCUs

We do not have a minimum number of VCUs we allocate to your ingest. If you do not ingest data, we do not need to allocate any VCUs to processing ingestion. There is an exception for a large number of indices (think: thousands of indices), where we do need to keep some resources allocated to be responsive when indexing requests come in.

VCU consumption is not linear

As with Search VCUs, we allocate Ingest VCUs based on step functions. Each step can contain a wide range of workloads: it's entirely possible that if you have a minimal amount of ingest, you could increase your ingest rate by a factor of 100 and still fit in the same step, thus not actually increasing your cost.

AI workloads

When running machine learning tasks in Serverless, we give you three options:

1. You use our Elastic Inference Service (EIS) to run your inference and completion workloads. We take care of everything, and you are charged per token.

2. You use traditional Elasticsearch Machine Learning capabilities to run your workloads. These use our Trained Models capabilities. We will scale up and down based on your machine learning workload requirements.

3. You do it yourself, outside of our systems, and just bring your vectors or other inference results to store and search in Elasticsearch.

EIS

The pricing for EIS is quite straightforward: you get charged a rate per one million consumed tokens. Token consumption is generally easy to predict for inference workloads. For LLM-based tasks, particularly agentic ones, this can be more complex, and some experimentation and trial runs may be useful to determine how many tokens your workloads typically consume.

ML VCUs

Machine Learning VCUs work on one simple input: machine learning workloads. The more inference you require, the more VCUs we will consume. Once you stop performing inference, we will scale down. We will keep a trained model in memory for about 24 hours after you last used it so that we can be responsive, which means that the minimal amount of VCU required to keep that model available will remain up for 24 hours before scaling down entirely.

We generally recommend our customers use EIS instead of our Machine Learning nodes for inference, particularly if your usage is periodic. By switching to EIS, you will not have to wait for machine learning nodes to spin up, and we won't charge you for unused ML node time before scaling down. EIS charges on a per token basis.

Storage

We charge storage per gigabyte per month. Storage does serve as an input into other parts of our system, particularly Search VCUs (see Search VCU above), but the pricing for storage itself is quite straightforward.

Data Out (egress)

We charge you for the data you take out of the system.

To minimize your egress costs, we recommend a few optimizations on your queries:

1. Do not return vectors in your query responses. We do this by default for indices created after October 2025. You can always return vectors in your responses explicitly if necessary.

2. Return only the fields needed for your application. You can do this by using the fields and _source parameters.

Support

We charge support as a percentage of your total ECU usage. We currently have four levels of support:

1. Limited support

2. Base support

3. Enhanced support

4. Premium support

Project subtype profiles

We currently offer two project subtypes for Serverless Elasticsearch, referred to as “General Purpose” and “Vector Optimized”. All Serverless Elasticsearch projects created through the cloud console UI will be created using the “General Purpose” option. You may create a “Vector Optimized” by calling the API directly with the optimized_for parameter (see documentation for all options).

The difference between the two options is the allocation of resources. We allocate approximately four times more resources (aka VCUs) to the “Vector Optimized” profile, which will result in your costs being up to four times higher. This is why we recommend starting on the “General Purpose” profile and only using the “Vector Optimized” profile when your use case demands the use of uncompressed dense vectors with high dimensionality, and quantization and DiskBBQ will not serve your needs.

When Serverless Elasticsearch was envisioned years ago, we thought that vector workloads would require much more resources to remain performant. However, with innovations like semantic_text, sparse_vector models, and Better Binary Quantization (BBQ), we’ve found that many vector workloads perform well on the “General Purpose” profile at a fraction of the cost. Therefore, don’t let the “Vector Optimized” label fool you…you can get excellent price and performance for vector workloads on the “General Purpose” profile.

Monitoring costs

We recognize that keeping track of your costs, especially when you are new to Elasticsearch Serverless, is important to you. We built a few tools just for this purpose, and continue to improve them for even greater visibility.

Cloud console billing usage

The Elastic Cloud Console provides billing details for your cloud account, across all cloud-based resources, including Elasticsearch Serverless. There, you can find a breakdown of all the price components described in this article. Filters allow you to zoom in on specific time periods and resources.

To further monitor your costs, you can also configure custom budget alerts from the Budgets and notifications tab under the Billing and subscriptions page.

AutoOps monitoring

We’re bringing AutoOps to Serverless! One of the key value propositions of Elasticsearch Serverless is that we ensure everything runs smoothly, but that also means you have limited observability into the infrastructure. AutoOps for Serverless gives users visibility into what is driving usage, and, therefore, costs.

AutoOps is rolled out in new Serverless regions regularly, and we're always working to add new monitoring tools. Make sure to check out the region coverage and future planned monitoring tools.

However, when you fix this one issue, you accidentally break other queries since you couldn’t test all cases manually. But how can you or your QA team test if a change in one query has a ripple effect in other queries? Or even more importantly, how can you be sure that your changes actually improved a query?

Towards a systematic evaluation

Here is where judgment lists come in useful. Instead of depending on manual and subjective testing any time you make a change, you can define a fixed set of queries that are relevant for your business case, together with their relevant results.

This set becomes your baseline. Every time you implement a change, you use it to evaluate if your search actually improved or not.

The value of this approach is that it:

• Removes uncertainty: you no longer need to wonder if your changes impact other queries; the data will tell you.

• Stops manual testing: once the judgment sets are recorded, the test is automatic.

• Supports changes: You can show clear metrics that support the benefits of a change.

How to start building your judgment list

One of the easiest ways to start is to take a representative query and manually select the relevant documents. There are two ways to do this list:

• Binary Judgments: Each document associated with a query gets a simple tag: relevant (usually with a score of “1”) and not-relevant (“0”).

• Graded Judgments: Here, each document gets a score with different levels. For example: setting a 0 to 4 scale, similar to a Likert scale, where 0 = “not at all relevant” and 4 = “totally relevant,” with variations like “relevant,” “somewhat relevant,” etc.

Binary judgments work well when the search intent has clear limits: Should this document be in the results or not?

Graded judgements are more useful when there are grey areas: some results are better than others, so you can get “very good,” “good,” and “useless” results and use metrics that value the order of the results and the user’s feedback. However, graded scales also introduce drawbacks: different reviewers may use the scoring levels differently, which makes the judgments less consistent. And because graded metrics give more weight to higher scores, even a small change (like rating something a 3 instead of a 4) can create a much bigger shift in the metric than the reviewer intended. This added subjectivity makes graded judgments noisier and harder to manage over time.

Do I need to classify the documents myself?

Not necessarily, since there are different ways to create your judgment list, each with its own advantages and disadvantages:

• Explicit Judgments: Here, SMEs go over each query/document and manually decide if (or how) relevant it is. Though this provides quality and control, it is less scalable.

• Implicit Judgments: With this method, you infer the relevant documents based on real-user behavior like clicks, bounce rate, and purchases, among others. This approach allows you to gather data automatically, but it might be biased. For example, users tend to click top results more often, even if they are not relevant.

• AI-Generated Judgments: This last option uses models (like LLMs) to automatically evaluate queries and documents, often referred to as LLM juries. It’s fast and easy to scale, but the quality of the data depends on the quality of the model you’re using and how well LLM training data aligns with your business interests. As with human grades, LLM juries can introduce their own biases or inconsistencies, so it’s important to validate their output against a smaller set of trusted judgments. LLM models are probabilistic by nature, so it is not uncommon to see an LLM model giving different grades to the same result regardless of setting temperature parameter as 0.

Below are some recommendations to choose the best method for creating your judgment set:

• Decide how critical some features are for you that only users can properly judge (like price, brand, language, style, and product details). If those are critical, you need explicit judgments for at least some part of your judgment list.

• Use implicit judgements when your search engine already has enough traffic so you can use clicks, conversions, and lingering time metrics to detect usage trends. You should still interpret these carefully, contrasting them with your explicit judgement sets to prevent any bias (e.g: users tend to click top-ranked results more often, even if lower-ranked results are more relevant)

To address this, position debiasing techniques adjust or reweight click data to better reflect true user interest. Some approaches include:

• Results shuffling: Change the order of search results for a subset of users to estimate how position affects clicks.

• Click models includeDynamic Bayesian Network DBN, User Browsing Model UBM. These Statistical models estimate the probability of a click reflects real interest rather than just position, using patterns like scrolling, dwell time, click sequence, and returning to the results page.

Example: Movie rating app

Prerequisites

To run this example, you need a running Elasticsearch 8.x cluster, locally or Elastic Cloud (Hosted or Serverless), and access to the REST API or Kibana.

Think about an app in which users can upload their opinions about movies and also search for movies to watch. As the texts are written by users themselves, they can have typos and many variations in terms of expression. So it’s essential that the search engine is able to interpret that diversity and provide helpful results for the users.

To be able to iterate queries without impacting the overall search behavior, the business team in your company created the following binary judgment set, based on the most frequent searches:

Query

DocID

Text

DiCaprio performance

doc1

DiCaprio's performance in The Revenant was breathtaking.

DiCaprio performance

doc2

Inception shows Leonardo DiCaprio in one of his most iconic roles.

DiCaprio performance

doc3

Brad Pitt delivers a solid performance in this crime thriller.

DiCaprio performance

doc4

An action-packed adventure with stunning visual effects.

sad movies that make you cry

doc5

A heartbreaking story of love and loss that made me cry for hours.

sad movies that make you cry

doc6

One of the saddest movies ever made — bring tissues!

sad movies that make you cry

doc7

A lighthearted comedy that will make you laugh

sad movies that make you cry

doc8

A science-fiction epic full of action and excitement.

Creating the index:

BULK request:

Below is the Elasticsearch query the app is using:

From judgment to metrics

By themselves, judgment lists do not provide much information; they are only an expectation of the results from our queries. Where they really shine is when we use them to calculate objective metrics to measure our search performance.

Nowadays, most of the popular metrics include

• Precision: Measures the proportion of results that are truly relevant within all search results.

• Recall: Measures the proportion of relevant results the search engine found among x results.

• Discounted Cumulative Gain (DCG): Measures the quality of the result’s ranking, considering the most relevant results should be at the top.

• Mean Reciprocal Rank (MRR): Measures the position of the first relevant result. The higher it is in the list, the higher its score.

Using the same movie rating app as an example, we’ll calculate the recall metric to see if there’s any information that is being left out of our queries.

In Elasticsearch, we can use the judgment lists to calculate metrics via the Ranking Evaluation API. This API receives as input the judgment list, the query, and the metric you want to evaluate, and returns a value, which is a comparison of the query result with the judgment list.

Let’s run the judgment list for the two queries that we have:

We’ll use two requests to _rank_eval: one for the DiCaprio query and another for sad movies. Each request includes a query and its judgment list (ratings). We don’t need to grade all documents since the ones not included in the ratings are considered as with no judgment. To do the calculations, recall only considers the “relevant set,” the documents that are considered relevant in the rating.

In this case, the DiCaprio query has a recall of 1, while the sad movies got 0. This means that for the first query, we were able to get all relevant results, while in the second query, we did not get any. The average recall is therefore 0.5.

Maybe we’re being too strict with the minimum_should_match parameter since by demanding that 100% of the words in the query are found in the documents, we’re probably leaving relevant results out. Let’s remove the minimum_should_match parameter so that a document is considered relevant if only one word in the query is found in it.

As you can see, by removing the minimum_should_match parameter in one of the two queries, we now get an average recall of 1 in both.

In summary, removing the minimum_should_match: 100% clause, allows us to got a perfect recall for both queries.

We did it! Right?

Not so fast!

By improving recall, we open the door to a wider range of results. However, each adjustment implies a trade-off. This is why defining complete test cases, using different metrics to evaluate changes.

Using judgment lists and metrics prevents you from going in blind when making changes since you now have data to back them up. Validation is no longer manual and repetitive, and you can test your changes in more than just one use case. Additionally, A/B testing allows you to test live which configuration works best for your users and business case, thus coming full circle from technical metrics and real-world metrics.

Final recommendations for using judgment lists

Working with judgment lists is not only about measuring but also about creating a framework that allows you to iterate with confidence. To achieve this, you can follow these recommendations:

1. Start small, but start. You don’t need to have 10,000 queries with 50 judgment lists each. You only need to identify the 5–10 most critical queries for your business case and define which documents you expect to see at the top of the results. This already gives you a base. You typically want to start with the top queries plus the queries with no results. You can also start testing with an easy-to-configure metric like Precision and then work your way up in complexity.

2. Validate with users. Complement the numbers with A/B testing in production. This way, you’ll know if changes that look good in the metrics are also generating a real impact.

3. Keep the list alive. Your business case will evolve, and so will your critical queries. Update your judgment periodically to reflect new needs.

4. Make it part of the flow. Integrate judgment lists into your development pipelines. Make sure each configuration change, synonym, or text analysis is automatically validated against your base list.

5. Connect technical knowledge with strategy. Don’t stop at measuring technical metrics like precision or recall. Use your evaluation results to inform business outcomes.

Welcome to part 4 of our blog series on integrating generative AI in Elastic's customer support. This installment dives deep into the role of Retrieval-Augmented Generation (RAG) in enhancing our AI-driven Technical Support Assistant. Here, we address the challenges, solutions, and outcomes of refining search effectiveness, providing action items to further improve its capabilities using the toolset provided in the Elastic Stack version 8.11.

Implied by those actions, we have achieved a ~75% increase in top-3 results relevance and gained over 300,000 AI-generated summaries that we can leverage for all kinds of future applications. If you're new to this series, be sure to review the earlier posts that introduce the core technology and architectural setup. If you missed the last blog of the series, you can find it here.

RAG tuning: A search problem

Perfecting RAG (Retrieval-Augmented Generation) is fundamentally about hitting the bullseye in search accuracy 🎯:

• Like an archer carefully aiming to hit the center of the target, we want to focus on accuracy for each hit.

• Not only that, we also want to ensure that we have the best targets to hit – or high-quality data.

Without both together, there's the potential risk that large language models (LLMs) might hallucinate and generate misleading responses. Such mistakes can definitely shake users' trust in our system, leading to a deflecting usage and poor return on investment.

To avoid those negative implications, we've encountered several challenges that have helped us refine our search accuracy and data quality over the course of our journey. These challenges have been instrumental in shaping our approach to tuning RAG for relevance, and we're excited to share our insights with you.

That said: let's dive into the details!

Our first approach

We started with a lean, effective solution that could quickly get us a valuable RAG-powered chatbot in production. This meant focusing on key functional aspects that would bring it to operational readiness with optimal search capabilities. To get us into context, we'll make a quick walkthrough around four key vital components of the Support AI Assistant: data, querying, generation, and feedback.

Data

As showcased in the 2nd blog article of this series, our journey began with an extensive database that included over 300,000 documents consisting of Technical Support Knowledge Articles and various pages crawled from our website, such as Elastic's Product Documentation and Blogs. This rich dataset served as the foundation for our search queries, ensuring a broad spectrum of information about Elastic products was available for precise retrieval. To this end, we leveraged Elasticsearch to store and search our data.

Query

Having great data to search by, it's time to talk about our querying component. We adopted a standard Hybrid-Search strategy, which combines the traditional strengths of BM25, Keyword-based Search, with the capabilities of Semantic Search, powered by ELSER.

For the semantic search component, we used text_expansion queries against both title and summary embeddings. On the other hand, for broad keyword relevance we search multiple fields using cross_fields, with a minimum_should_match parameter tuned to better perform with longer queries. Phrase matches, which often signal greater relevance, receive a higher boost. Here’s our initial setup:

Generation

After search, we build up the system prompt with different sets of instructions, also contemplating the top 3 search results as context to be used. Finally, we feed the conversation alongside the built context into the LLM, generating a response. Here's the pseudocode showing the described behavior:

The reason for not including more than 3 search results was the limited quantity of tokens available to work within our dedicated Azure OpenAI's GPT4 deployment (PTU), allied with a relatively large user base.

Feedback

We used a third-party tool to capture client-side events, connecting to Big Query for storage and making the JSON-encoded events accessible for comprehensive analysis by everyone on the team. Here's a glance into the Big Query syntax that builds up our feedback view. The

JSON_VALUE function is a means to extract fields from the event payload:

We also took advantage of valuable direct feedback from internal users regarding the chatbot experience, enabling us to quickly identify areas where our search results did not match the user intent. Incorporating both would be instrumental in the discovery process that enabled us to refine our RAG implementation, as we're going to observe throughout the next section.

Challenges

With usage, interesting patterns started to emerge from feedback. Some user queries, like those involving specific CVEs or Product Versions for instance, were yielding suboptimal results, indicating a disconnect between the user's intent and the GenAI responses. Let's take a closer look at the specific challenges identified, and how we solved them.

#1: CVEs (Common Vulnerabilities and Exposures)

Our customers frequently encounter alerts regarding lists of open CVEs that could impact their systems, often resulting in support cases. To address questions about those effectively, our dedicated internal teams meticulously maintain CVE-type Knowledge Articles. These articles provide standardized, official descriptions from Elastic, including detailed statements on the implications, and list the artifacts affected by each CVE.

Recognizing the potential of our chatbot to streamline access to this crucial information, our internal InfoSec and Support Engineering teams began exploring its capabilities with questions like this:

For such questions, one of the key advantages of using RAG – and also the main functional goal of adopting this design – is that we can pull up-to-date information, including it as context to the LLM and thus making it available instantly to produce awesome responses. That naturally will save us time and resources over fine-tuned LLM alternatives.

However, the produced responses wouldn't perform as expected. Essential to answer those questions, the search results often lacked relevance, a fact which we can confirm by looking closely at the search results for the example:

With just one relevant hit (CVE-2019-10172), we left the LLM without the necessary context to generate proper answers:

The observed behavior prompted us with an interesting question:

To solve this, we approached the issue as a search challenge. We hypothesized that by emphasizing the title field matching for such articles, which directly contain the CVE codes, we could significantly improve the precision of our search results. This led to a strategic decision to conditionally boost the weighting of title matches in our search algorithm. By implementing this focused adjustment, we refined our query strategy as follows:

As a result, we experienced much better hits for CVE-related use cases, ensuring that CVE-2016-1837, CVE-2019-11756 and CVE-2014-6439 are top 3:

And thus generating a much better response by the LLM:

Lovely! By tuning our Hybrid Search approach, we significantly improved our performance with a pretty simple, but mostly effective Bob's Your Uncle solution (like some folks would say)! This improvement underscores that while semantic search is a powerful tool, understanding and leveraging user intent is crucial for optimizing search results and overall chat experience in your business reality. With that in mind, let's dive into the next challenge!

#2: Product versions

As we delved deeper into the challenges, another significant issue emerged with queries related to specific versions. Users frequently inquire about features, migration guides, or version comparisons, but our initial search responses were not meeting expectations. For instance, let's take the following question:

Our initial query approach would return the following top 3:

• Elasticsearch for Apache Hadoop version 8.14.1 | Elasticsearch for Apache Hadoop [8.14] | Elastic;

• APM version 8.14 | Elastic Observability [8.14] | Elastic;

• Elasticsearch for Apache Hadoop version 8.14.3 | Elasticsearch for Apache Hadoop [8.14] | Elastic.

Corresponding to the following _search response:

Being irrevocably irrelevant, they ended up resulting in a completely uninformed answer from the chatbot, affecting the overall user experience and trust in the Support AI Assistant:

Further investigating the issue we collected valuable insights. By replaying the query and looking into the search results, we noticed three serious problems with our crawled Product Documentation data that were contributing to the overall bad performance:

1. Inaccurate semantic matching: Semantically, we definitely missed the shot. Why would we match against such specific articles, including two specifically about Apache Hadoop, when the question was so much broader than Hadoop?

2. Multiple versions, same articles: Going further down on the hits of the initially asked question, we often noticed multiple versions for the same articles, with close to exactly the same content. That often led to a top 3 cluttered with irrelevant matches!

3. Wrong versions being returned: It's fair to expect that having both 8.14.1 and 8.14.2 versions of the Elasticsearch for Apache Hadoop article, we'd return the latter for our query – but that just wasn't happening consistently.

From the impact perspective, we had to stop and solve those – else, a considerable part of user queries would be affected. Let's dive into the approaches taken to solve both!

A. Inaccurate semantic matching

After some examination into our data, we've discovered that the root of our semantic matching issue lived in the fact that the summary field for Product Documentation-type articles generated upon ingestion by the crawler was just the first few characters of the body. This redundancy misled our semantic model, causing it to generate vector embeddings that did not accurately represent the document's content in relation to user queries.

As a data problem, we had to solve this problem in the data domain: by leveraging the use of GenAI and the GPT4 model, we made a team decision to craft a new AI Enrichment Service – introduced in the 2nd installment of this blog series. We decided to create our own tool for a few specific reasons:

• We had unused PTU resources available. Why not use them?

• We needed this data gap filled quickly, as this was probably the greatest relevance detractor.

• We wanted a fully customizable approach to make our own experiments.

Modeled to be generic, our usage for it boils down to generating four new fields for our data into a new index, using Enrich Processors to make them available to the respective documents on the target indices upon ingestion. Here's a quick view into the specification for each field to be generated:

After generating those fields and setting up the index Enrich Processors, the underlying RAG-search indices were enriched with a new ai_fields object, also making ELSER embeddings available under ai_fields.ml.inference:

Now, we can tune the query to use those fields, making for better overall semantic and keyword matching:

Single-handedly, that made us much more relevant. More than that – it also opened a lot of new possibilities to use the AI-generated data throughout our applications – matters of which we'll talk about in future blog posts.

Now, before retrying the query to check the results: what about the multiple versions problem?

B. Multiple versions, same articles

When duplicate content infiltrates these top positions, it diminishes the value of the data pool, thereby diluting the effectiveness of GenAI responses and leading to a suboptimal user experience. In this context, a significant challenge we encountered was the presence of multiple versions of the same article. This redundancy, while contributing to a rich collection of version-specific data, often cluttered the essential data feed to our LLM, reducing the diversity of it and therefore undermining the response quality.

To address the problem, we employed the

Elasticsearch API collapse parameter, sifting through the noise and prioritizing only the most relevant version of a single content. To do that, we computed a new slug field into our Product Documentation crawled documents to identify different versions of the same article, using it as the collapse field (or key).

Taking the Sort search results documentation page as an example, we have two versions of this article being crawled:

• Sort search results | Elasticsearch Guide [8.14] | Elastic

• Sort search results | Elasticsearch Guide [7.17] | Elastic

Those two will generate the following slug:

Taking advantage of that, we can now tune the query to use collapse:

As a result, we'll now only show the top-scored documentation in the search results, which will definitely contribute to increasing the diversity of knowledge being sent to the LLM.

C. Wrong versions being returned

Similar to the CVE matching problem, we can boost results based on the specific versions being mentioned, allied with the fact that version is a separate field in our index. To do that, we used the following simple regex-based function to pull off versions directly from the user question:

We then add one more query to the should clause, boosting the version field accordingly and getting the right versions to the top (whenever they're mentioned):

With A, B and C solved, we're probably ready to see some strong results!

Let's replay the question!

By replaying the previously tried question:

And therefore running the Elasticsearch query once again, we get dramatically better results consisting of the following articles:

• Elasticsearch version 8.14.3 | Elasticsearch Guide [master] | Elastic

• Elasticsearch version 8.14.2 | Elasticsearch Guide [master] | Elastic

• Release notes | Elasticsearch Guide [8.14] | Elastic

Consequently, we have a better answer generated by the LLM. More powerful than that – in the context of this conversation, the LLM is now conscious about versions of Elasticsearch that are newer than the model's cut-off date, crafting correct answers around those:

Exciting, right? But how can we quantify the improvements in our query at this point? Let's see the numbers together!

Measuring success

To assess the performance implied by our changes, we've compiled a test suite based on user behavior, each containing a question plus a curated list of results that are considered relevant to answer it. Those will cover a wide wide range of subjects and query styles, reflecting the diverse needs of our users. Here's a complete look into it:

But how do we turn those test cases into quantifiable success? To this end, we have employed Elasticsearch's Ranking Evaluation API alongside with the Precision at K (P@K) metric to determine how many relevant results are returned between the first K hits of a query. As we're interested in the top 3 results being fed into the LLM, we're making K = 3 here.

To automate the computation of this metric against our curated list of questions and effectively assess our performance gains, we used TypeScript/Node.js to create a simple script wrapping everything up. First, we define a function to make the corresponding Ranking Evaluation API calls:

After that, we need to define the search queries before and after the optimizations:

Then, we'll output the resulting metrics for each query:

Finally, by running the script against our development Elasticsearch instance, we can see the following output demonstrating the P@K or (P@3) values for each query, before and after the changes. That is – how many results on the top 3 are considered relevant to the response:

Improvements observed

As an archer carefully adjusts for a precise shot, our recent efforts into relevance have brought considerable improvements in precision over time. Each one of the previous enhancements, in sequence, were small steps towards achieving better accuracy in our RAG-search results, and overall user experience. Here's a look at how our efforts have improved performance across various queries:

Before and after – P@K

Query Description

P@K Before

P@K After

Change

Support Diagnostics Tool

0.333 🥉

1.000 🥇

+200%

Air Gapped Maps Service

0.333 🥉

0.667 🥈

+100%

CVE Implications

0.000 ❌

1.000 🥇

∞

Enrich Processor Setup

0.667 🥈

0.667 🥈

0%

Proxy Certificates Rotation

0.333 🥉

0.333 🥉

0%

Proxy Certificates Version-specific Rotation

0.333 🥉

0.333 🥉

0%

Searchable Snapshot Deletion

0.667 🥈

1.000 🥇

+50%

Index Lifecycle Management Usage

0.667 🥈

0.667 🥈

0%

Creating Data Views via API in Kibana

0.333 🥉

0.667 🥈

+100%

Kibana Data View Creation

1.000 🥇

1.000 🥇

0%

Comparing Elasticsearch Versions

0.000 ❌

0.667 🥈

∞

Maximum Bucket Size in Aggregations

0.000 ❌

0.333 🥉

∞

Average P@K Improvement: +78.41% 🏆🎉. Let's summarize a few observations about our results:

Significant Improvements: With the measured overall +78.41% of relevance increase, the following queries – Support Diagnostics Tool, CVE implications, Searchable Snapshot Deletion, Comparing Elasticsearch Versions – showed substantial enhancements. These areas not only reached the podium of search relevance but did so with flying colors, significantly outpacing their initial performances!

Opportunities for Optimization: Certain queries like the Enrich Processor Setup, Kibana Data View Creation and Proxy Certificates Rotation have shown reliable performances, without regressions. These results underscore the effectiveness of our core search strategies. However, those remind us that precision in search is an ongoing effort. These static results highlight where we'll focus our efforts to sharpen our aim throughout the next iterations. As we continue, we'll also expand our test suite, incorporating more diverse and meticulously selected use cases to ensure our enhancements are both relevant and robust.

What's next? 🔎

The path ahead is marked by opportunities for further gains, and with each iteration, we aim to push the RAG implementation performance and overall experience even higher. With that, let's discuss areas that we're currently interested in!

1. Our data can be futher optimized for search: Although we have a large base of sources, we observed that having semantically close search candidates often led to less effective chatbot responses. Some of the crawled pages aren't really valuable, and often generate noise that impacts relevance negatively. To solve that, we can curate and enhance our existing knowledge base by applying a plethora of techniques, making it lean and effective to ensure an optimal search experience.

2. Chatbots must handle conversations – and so must RAG searches: It's common user behavior to ask follow-up questions to the chatbot. A question asking "How to configure Elasticsearch on a Linux machine?" followed by "What about Windows?" should query something like "How to configure Elasticsearch on a Linux machine?" (not the raw 2nd question). The RAG query approach should find the most relevant content regarding the entire context of the conversation.

3. Conditional context inclusion: By extracting the semantic meaning of the user question, it would be possible to conditionally include pieces of data as context, saving token limits, making the generated content even more relevant, and potentially saving round trips for search and external services.

Conclusion

In this installment of our series on GenAI for Customer Support, we have thoroughly explored the enhancements to the Retrieval-Augmented Generation (RAG) search within Elastic's customer support systems. By refining the interaction between large language models and our search algorithms, we have successfully elevated the precision and effectiveness of the Support AI Assistant.

Looking ahead, we aim to further optimize our search capabilities and expand our understanding of user interactions. This continuous improvement will focus on refining our AI models and search algorithms to better serve user needs and enhance overall customer satisfaction.

Stay tuned for more insights and updates as we continue to push the boundaries of what's possible with AI in customer support, and don't forget to join us in our next discussion, where we'll explore how Observability plays a critical role in monitoring, diagnosing, and optimizing the performance and reliability of the Support AI Assistant as we scale!

But before we go into more detail on the changes we made to the Elastic® Support Hub and its impact on our customers, it's important that we take a moment to explain the concept of semantic search. At its core, semantic search is a method of search that uses AI to return more relevant search results. Take a look at this quick video explaining the concept:

As shown in the video, semantic search matches the intent of what the user searches to the content available rather than the words. You can read more about the AI behind it on our blog, Introducing Elastic Learned Sparse Encoder: Elastic’s AI model for semantic search. The rest of this blog tells our story about moving the Elastic Support Hub to semantic search.

Why did we make this change?

All technology news these days seems to have something to do with large language models and generative AI. Elastic is leading the charge with its vector database capabilities and built-in natural language models. It makes sense that we should build our supporting applications on the same bleeding edge that our product lives on. By making this change now, we can provide feedback to our product development teams and make the product better for everyone.

Biggest takeaway configuring semantic search

As with most new technology innovations, it requires tearing down, replacing older code, and potentially updating underlying architecture. Our internal app development team faced these challenges head-on, and we are now in a much better position to iterate on any of Elasticsearch®’s new features. From our teams' point of view, there were two significant features that stood out in the setup process:

1. Considering ELSER, Elastic’s proprietary transformer model for semantic search, is a relatively new feature in Elasticsearch (8.8), our development team was happy to see a guided UI experience to enable Elasticsearch ingest pipelines with ELSER.

This allowed our developers to quickly add the necessary text expansion configuration to the ingestion pipeline that makes semantic search possible. This made the configuration experience much easier to get started and see results quicker.

2. A machine learning model like ELSER takes dedicated machine resources to run (minimum 4GB). Since we were already running on Elastic Cloud, we were able to enable dedicated machine learning (ML) nodes with autoscaling to accommodate our resource demands and see more consistent performance.

Early evaluation of search results

We are enabling various systems to help us to understand user queries, search results, and relevancy at scale. However, in our user testing, we can already see significant improvements in various queries. For example, we tested the phrase “How to index data into Elasticsearch” on both our standard full-text search and our new semantic search implementations.

Here is a side-by-side comparison of the two search methods.

While there isn’t a single article that explains all the ways you can index data (there are a lot), you can see how fundamentally different these results are. For full-text search, we have a mix of guides, troubleshooting articles, and a blog with matched keywords, but none of them answer the question of “how.” Or to say it differently, text search didn't capture the meaning (semantically) of the query and did its best to match keywords.

For the semantic search results, you can see blogs that generally relate to the indexing of data. What is even more interesting is the fourth returned result of “How to ingest data into Elasticsearch Service” as the term ingest is actually more relevant to the process of adding data to an index. Elastic’s out-of-the-box transformer model picked up on the semantic meaning of adding data to an index and returned more relevant results regardless of the exact keywords.

What’s next?

While we see this as a gigantic leap forward in our ability to provide customers with relevant search results, we know our work is not done. Over time, we will evaluate the data we have on terms searched, results, and articles read. This data will allow us to add synonyms and configure appropriate weights and boosts to give you, our customers, the best experience when searching for Elastic content on support.elastic.co.

>> Learn more about all the Support Hub has to offer.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.