What is the Kibana chrome?

The chrome is everything that frames an application in Kibana: the global header and the navigation, along with the header of the page that you’re currently on. Every application renders inside it. Because the chrome is shared, its APIs decide how consistent the whole product feels and how hard the next redesign will be.

What inconsistent page headers looked like to users

The clearest way to see the problem is to look at what users saw:

Take something as basic as a primary action, the one button a page most wants you to click, like Create index. Before this work, primary actions lived in three different places, depending on the page:

• In the old app header.

• In the page template header.

• Somewhere in the page body itself.

Common links had the same problem, with share, feedback, and documentation links appearing in separate spots with different styling from one app to the next. Breadcrumbs were misconfigured on many pages. For example, some showed a breadcrumb for each tab within a page, and some weren't clickable. Still others repeated the title of the page you were already on.

Every one of these was the output of a well-intentioned team using a flexible API in the way it’s usually used. Users had to relearn each page's layout, and every attempt to redesign the chrome had to account for hundreds of local variations.

Goals for the page header redesign

The goal of the redesign was simple; that is, pages should express what they need (for example, a title, badge, or primary action), and the shared shell should decide how it looks and where it goes. Common links and primary actions should get one home, everywhere. If it's the primary action, it's always in the same place, styled the same way. And consistency should stop being something that each of dozens of teams has to get right by hand, forever.

Getting there wasn't a restyling exercise; it required changing the contract between applications and the platform.

How typed props replaced EuiPageHeader's open React nodes

Each page header was easy to build in isolation, and the local choices piled up into visible differences across the product: inconsistent spacing, title styling, and action placement from one page to the next.

The fix, everywhere this showed up, was the same mechanism: stricter APIs. Instead of accepting arbitrary React nodes, a control declares what it means through typed props, and the shared shell decides how it looks and where it goes. That one change buys two things at once: consistency, and governance over what the shared surface is allowed to contain.

Take the page header. With EuiPageHeader, the API exposed layout settings and accepted React nodes for nearly every visible area:

Because pageTitle, description, and rightSideItems all accept arbitrary React nodes, every team filled them in differently. One page put a badge next to the title, while another styled its own pill. One team's actions were plain buttons in one order; the next team's were a different mix in another order, some collapsing into a menu and others not. Everyone used the same shell component, yet the headers looked and behaved like they came from different products. The shared component guaranteed the wrapper, not what went inside it.

The AppHeader API closes that off. The same header is expressed as typed props:

Now there’s only one way to express each part, so every page renders it the same. Badges live in their own typed collection, separate from the title. A description carries its text and, optionally, a "Learn more" URL. Actions declare whether they’re primary or secondary, and the component decides how they look and when they collapse. Plus, richer controls fit the same shape. An editable title or a favorite toggle is a structured config rather than a bespoke React tree:

The split is consistent throughout; the application supplies state and behavior (the current name, what happens on rename, how to toggle a favorite) and the header owns presentation. It renders the controls, handles keyboard interaction, shows validation errors, and reflects pending changes. Because those roles are explicit, layout and responsive behavior stay inside the shared component, and applications don't need a coordinated update every time the header changes.

Why a shared UI shell needs a closed set of controls

The global shell had a similar problem that was one step more open-ended. Any plugin could register arbitrary content on the left or right and pick its position with a number, and no conversation with the platform team or designers was required:

That registerRight API is an open door. Any team can add a control, and the header fills up with elements no one designed together. The controls compete for space and carry their own styling, and their order is decided by whoever picked the larger number. The shell only knows that one React node follows another; it can't tell that one opens an AI assistant and another collects feedback, so it can't reason about them or keep them coherent.

The new shell replaces that open canvas with a closed set of named roles under chrome.controls and chrome.help. The old registry hasn't gone anywhere yet, because the classic header still runs alongside the new one while applications migrate, but nothing in the new shell is reachable through it:

Global chrome header versus application page header

Part of what made the old chrome hard to evolve was that "the header" was really several things tangled together. The redesign draws a hard line between two surfaces with different owners:

Chrome (global) header

Application header

Owner

The platform

The page

Scope

What's true everywhere in Kibana

What's true on this page

Contents

Navigation, project or deployment picker, search, help, AI assistant, feedback

Title, badges, description, tabs, page actions

How it's populated

Named slots under chrome.controls and chrome.help

AppHeader typed props, rendered directly or set via chrome.appHeader.set()

Because each surface has one owner and a typed contract between them, the platform can redesign the chrome without auditing hundreds of pages, and an application can evolve its header without colliding with global controls. Setting a header config returns a cleanup callback, so an application tears down its own contribution when it unmounts.

Design decisions that strict APIs force

Stricter APIs forced design decisions that flexible APIs had let everyone defer. Two are worth calling out.

Why we removed breadcrumbs from Kibana's navigation

Removing breadcrumbs from the chrome was one of the more debated calls, and the state of the data made it easier than expected. In practice, breadcrumbs across Kibana were widely misconfigured:

• Some pages generated a breadcrumb for every tab on the page.

• Some breadcrumbs weren't clickable.

• Some repeated the title of the current page.

They added visual weight without reliably adding orientation.

At the same time, the redesigned navigation already conveys hierarchy. The primary and secondary navigation show you where you are and give you a direct way back up. Keeping breadcrumbs would have meant maintaining a third (and frequently wrong) representation of the same information, so we stopped drawing the trail and let the navigation do that job.

The breadcrumb API itself didn't disappear. Applications still register breadcrumbs, and the new shell reads them to derive the back button and to fall back to a page title when an app hasn't supplied one. The data changed jobs rather than going away, which is why migrating an app to the new header rarely starts with ripping breadcrumbs out:

Before:

After:

How many action buttons a page header should show

The other recurring debate was priority. With every page's actions now flowing through one menu structure, which buttons deserve to be visible, and in what order? Priorities shift as products evolve, so this conversation is ongoing.

For launch, we made one deliberate simplification: we locked the number of buttons that can appear in the app menu. A page gets one primary action and a bounded set of secondary actions before the rest collapse into a menu. The constraint keeps any single page from dividing the user's attention across a wall of buttons, and it makes the priority conversation explicit instead of letting it be settled by whoever adds the next button.

We went through several iterations to land here. Initially, we had many visible actions. A page could have up to three buttons, plus a secondary action to the left of the primary one. That took up too much space, so we gradually simplified the layout by removing the secondary action and reducing the number of visible buttons. We also made the overflow menu more structured. Some items, such as feedback and documentation, now have a fixed place in the footer of the overflow menu.

What strict design system governance costs teams

Under the old API, a plugin could add a novel UI by choosing a side, picking an order, and mounting a React tree. Under the stricter API, a new kind of control may need a shared capability before a plugin can add it. That takes more work up front and forces a design decision that teams could previously avoid.

We accepted the cost of stricter APIs because the alternative pushed it into every later redesign. As long as arbitrary React trees mounted at arbitrary points, every change to the shell's layout or accessibility had to account for all of them.

That’s the trade we made, and it pays back on both fronts. Consistency stops being something each team has to get right by hand; there’s one way to express a control, so pages match by default. And the shared surface stays governed. Its set of controls is a deliberate design decision, not whatever accumulated at the edges. Applications describe what their controls mean, and the shell is free to decide how they look, both today and in the next redesign.

Where the redesigned Kibana chrome is available

The redesigned chrome is available now in Elastic Cloud Serverless. Open any project, and you're already using it. For Elastic Cloud Hosted and self-managed users, it ships in 9.6.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Doc values were built to make aggregations, sorting, and grouping possible on a document engine, and they do that job well. Columnar Mode changes what the columns are for, and this post walks through the five properties that separate storing columns from being a columnar database.

Five properties separate a column store from a columnar database

Doc values were an optimization on top of _source

For most of the last decade, the original JSON document was the source of truth and the columns were a derived convenience. That ordering has consequences throughout the engine.

Because the engine could always fall back to _source, per-field storage was allowed to be lossy. Text fields had no doc values at all, since text could be reread from the stored document when needed. Even synthetic source, which reconstructs a document from its fields rather than storing a copy, sometimes reads from row-shaped structures to stay faithful to the JSON that arrived, with values that exceed ignore_above and fields that arrived unmapped going into stored fields.

The result is a clear contract; whatever JSON you send, you get back, and the columns accelerate everything else. For an engine whose job is to return your documents, that’s the right way round. Columnar Mode inverts it. Every field stores itself exactly once as doc values, doc values cannot be turned off, text fields get doc values, too, and the document is reconstructed from the columns when something asks for it.

How dictionary encoding handles high-cardinality data

Sorted doc values, the default for keyword fields, store a dictionary of distinct values plus one ordinal per document pointing into it. This is an excellent trade when values repeat. A host.name field drawn from a hundred machines, or a status code that’s almost always 200, compresses beautifully and groups quickly.

It works like the index cards in a warehouse. When 50 crates hold the same product, one card and 50 pointers beats writing the product name 50 times. When 50 crates hold the same product, you can store the product name in the index with the list of 50 crate IDs. When every crate holds something unique, you might as well just put the product name on the crates; the index will help you find what crate you want, but it won't save ink.

High-cardinality fields describe a lot of real data, including URLs and trace identifiers, along with message bodies. Columnar Mode, which skips the dictionary and compresses the values in blocks instead, uses binary doc values for high-cardinality strings. Which of the two a field gets isn’t something you configure. The engine decides per field, based on the values it sees, so each column is encoded for the data it actually holds instead of one default applied to every field. Pure columnar systems have long carried cardinality in their type system, but usually as something you declare, and you own the consequences when the data shifts underneath it. Here, it’s the engine's job.

Why every field builds an inverted index by default

By default, a keyword field also builds an inverted index and a numeric field also builds a BKD tree. That happens on every field because at write time the engine doesn’t know which capability you’ll want at read time, significantly increasing the footprint of each field. Those structures also have to be rebuilt during segment merges, which costs CPU exactly when ingest is heaviest.

Our time series engine (TSDB) is proof of what happens when you stop paying for capability that the workload doesn’t use. Replacing the indices on @timestamp and dimension fields with doc value skippers, which are sparse structures holding the minimum and maximum value for each block of documents, removed 10 bytes of the original 25 bytes per OpenTelemetry (OTel) data point. There was no measurable query regression on time range and dimension filters, and indexing CPU dropped by about 10%, as a bonus.

Columnar Mode generalizes that default. Fields aren’t indexed unless something needs them to be, with only text-mapped fields keeping their inverted index for fast free-text search.

Metadata fields like _id and _routing were row-shaped

The fields you never think about followed the same document-first design. The _id field was a stored field plus an inverted index. Custom _routing was a stored field. Sequence numbers were kept for optimistic concurrency control, regardless of whether a workload ever updated a document.

TSDB deals with all three. It synthesizes _id from the _tsid and @timestamp values that already identify a data point, using a segment-level bloom filter to catch duplicates, which removes 5 bytes per data point with no loss of functionality. It trims sequence numbers once replication no longer needs them, which removes 4 bytes. Add a codec block size increase from 128 to 512 elements for another 2 bytes, and those four changes contribute across versions 9.1 through 9.4 to the 21 bytes that took OTel metrics from 25 bytes per data point down to 3.75.

Columnar Mode makes those ideas general rather than metrics-specific. By general availability (GA), all metadata fields will store themselves as doc values, while we plan to follow up and add a sort id mode synthesizing the identifier from the index sort fields, in addition to derived fields that will generalize what _tsid does for time series to any set of fields.

Columnar query execution in the ES|QL compute engine

A column store only pays off if the engine reads it as columns. Aggregations inherited the document-at-a-time shape from search, which is the natural fit for an engine built around documents. Reading columns instead lets the engine hand a whole block of values to a single instruction, and that’s where the numbers below come from.

The ES|QL compute engine changed that shape, and TSDB again shows the size of the effect:

• Vectorized execution of time series aggregations was worth up to 8x on its own.

• Decoding on-disk data straight into the primitive arrays the engine aggregates over, with no intermediate copies, was worth roughly another 10x.

• Constant blocks turned repeated values into a form of in-memory run-length encoding.

• Filter pushdown moved filters down to Lucene, where skippers can discard whole blocks unopened.

Together with the rest of the block-level query work, query latency improved by up to 160x compared to earlier versions.

That work continues. Skipper-aware operators, aggregations that group on ordinals and convert to real values as late as possible, and richer per-block summaries are all in progress, and they benefit every index mode because every mode reads doc values underneath.

What Columnar Mode changes for logs and analytical data

Storing values in columns is a storage detail. A columnar database needs five things:

1. The columns are the only copy of the data.

2. Each column is encoded for the data it actually holds.

3. Metadata is columnar, too.

4. Fields add indices only when something needs them.

5. The query engine processes blocks of values rather than records or documents.

TSDB reached all five for metrics in Elasticsearch 9.4, which is why the numbers in this post come from metrics rather than from a slide. Columnar Mode applies the same treatment to logs and security telemetry, along with analytical data. It’s in technical preview in Elasticsearch 9.5, with GA targeted for 9.7.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Start a free Elastic Cloud Serverless trial or log in to Elastic Cloud to follow along, and install the CLI via npm: 

Designing the Elastic CLI for people and AI agents

A useful side effect of building flexible tools for developers is that they’re more useful to AI agents, too. It’s also the tool that our Agent Skills now use to get things done, closing the loop that we opened in March 2026, when we said that a CLI for agent workflows was coming.

The CLI gives every public API across Elasticsearch, Kibana, and Elastic Cloud one shape: the same flags, input and output conventions, authentication method, and failure mode. Consistency is the ergonomic feature; everything else is built on it.

Agents need the same thing, only stricter. An agent won’t know how to craft a valid CLI command or notice if a tool “feels” wrong; it needs output it can parse and input it can validate before sending, along with failures it can branch on. Agents are now a first-class interface to Elastic, alongside people, whether they live on the platform or in your editor and terminal. Agent Skills, and now the CLI, are how we serve the second kind, so those needs are built into the core of the CLI rather than tacked on.

JSON input and output for every command

Agents love structured text, and almost every Elastic API already speaks JSON, so first-class JSON support was a hard requirement. Developers who are quick with a jq query will be equally satisfied.

• JSON output: Any command, like elastic version and elastic es indices delete ..., supports --json, which prints JSON-parseable output to stdout and nothing else. Failed commands print {"error": {"code": "...", "message": "..."}} to stderr.

• JSON input: Every command that takes input accepts JSON on stdin or via --input-file. Every top-level key in that JSON also works as a CLI argument, and inline arguments take precedence, so you can keep a big request body in a file and tweak a value or two per invocation.

• JSON Schema as --help output: Pass --help --json to any command, and it prints a valid JSON Schema for its input, which also feeds nicely into codegen tools. elastic cli-schema prints the whole command tree.

Exit codes that an AI agent can branch on

Agents loop on exit codes as much as they do over stdout. All failure modes are distinguishable from success, even if stdout and stderr are never read.

Safety rails: Keychain storage, allow lists, and validation

No model uses tools perfectly 100% of the time, so an agent-friendly CLI should provide safety rails wherever possible.

• Contexts and secret storage: Connection details live in named contexts in ~/.elasticrc.yml, kubectl-style; switch with --use-context. API commands never take credentials as flags. elastic config context add writes API keys to your OS keychain (macOS, Linux, Windows) and leaves a $(keychain:...) reference in the YAML; $(env:...), $(cmd:...), and $(file:...) work, too. Creating a Serverless project with --save-as writes its credentials straight to the keychain and never prints them, so nothing leaks into logs or LLM transcripts.

• Allowlists/blocklists: A commands.allowed (or commands.blocked) list in the config file, globally or per context, ensures that only the commands an administrator wants are runnable.

• Validation: Every command has a JSON Schema, so inputs are validated before any request is sent. Add --dry-run to any command that takes input, and it validates and exits without sending anything.

• Confirmation: Destructive commands prompt in a terminal. In a noninteractive session, where agents live, they refuse to run without --yes and say so in a structured error.

• Sanitization: Index, field, and pipeline names have length limits and forbidden characters. elastic sanitize index-name '<value>' (and field-name, pipeline-name, …) prints a version stripped of anything invalid.

Keeping API responses inside an agent's context window

Elastic APIs return a lot of data, and an agent’s context window is finite. Three controls help keep unnecessary text out of the context window:

• Field masks: --output-fields takes a comma-separated list, with dot notation for nested fields.

• String templates: For total control, --output-template takes a mustache-style template.

• Command profiles: --command-profile serverless (or default_profile: serverless in your config) hides Elastic Cloud Hosted commands and the Elasticsearch namespaces that don’t exist on Serverless. That means less to scroll past and less for an agent to guess wrong. It’s the profile we recommend for agents.

Control

What it does

Syntax

When to use

Field mask

Returns only the fields you name, using dot notation for nested fields

--output-fields 'name,version.number'

You want valid JSON back, just less of it. This is the default choice for agents parsing structured output.

String template

Renders the response through a mustache-style template

--output-template 'ES version: {{ version.number }}'

You need one value in a specific shape, for a shell variable, a log line, or a prompt.

Command profile

Hides commands and namespaces that don't apply to your deployment

--command-profile serverlessor default_profile: serverless

You want a smaller command surface so an agent has less to scroll past and less to guess wrong. This is recommended for agents.

Helpers for bulk ingest, scroll search, and msearch

Some of Elasticsearch’s most popular APIs have a learning curve, so elastic es helpers wraps them:

• scroll-search: Stream a large result set as NDJSON with paging handled for you.

• bulk-ingest: Ingest from a file, a directory, or stdin (NDJSON, JSON arrays, or CSV) with streaming, batching, concurrency, and retries.

• msearch: Send multiple searches in one request.

• watch: Print new documents from an index to stdout as they’re indexed. This is great for piping into logging tools.

elastic es and elastic kb are aliases for elastic stack elasticsearch and elastic stack kibana. If we don’t ship a command you need, elastic extension create scaffolds one for you.

Searching Elastic docs from the terminal

If you or your agents don’t know which API to use, elastic docs search (plus docs read and docs ask) searches Elastic’s documentation from the terminal, returning Markdown or --json. These are experimental. You’ll see a warning until you pass --accept-experimental, so explore, but don’t script against them yet.

Shell completion for Bash, Zsh, and Fish

Autocomplete hooks are available for Bash, Zsh, and Fish, and they always respect your commands.allowed or commands.blocked policy.

How Elastic Agent Skills use the CLI

Agent Skills teach an AI coding agent how an Elastic expert approaches a job; for example, which cluster health field is the verdict or how to stage a reindex so it doesn’t fall over. They capture process and judgment but not transport. A skill that embeds curl with an auth header has hard-coded a hostname, key, and runtime, and it breaks when any of those change.

So our skills now use a universal format that runs unchanged in any runtime that can execute the elastic CLI, including Claude Code, Codex, Cursor, and GitHub Copilot. The body refers to operations in HTTP shorthand (GET /_cluster/health, POST /_query), and an operations table at the end binds each to a CLI command. That table is the only place transport appears:

HTTP API (shorthand)

elastic CLI command

GET /{index}/_mapping

elastic es indices get-mapping --index '<index>'

POST /_query

elastic es esql query --format tsv --query "<esql>"

POST cloud:/api/v1/serverless/projects/elasticsearch

elastic cloud serverless projects search create --input-file <json> --wait --save-as <ctx>

Every universal skill also inherits a blunt preamble; that is, use the CLI, don’t guess credentials, don’t call the HTTP API directly, and never ask the user to paste an API key into the chat.

The two halves need each other. The skill supplies the expertise that the model doesn’t have, and the CLI supplies a way to act on it that’s validated, credential-safe, and scoped by your allowlist. Tell your agent to spin up a Serverless project and load products.csv into it, and the provisioning skill creates it with --save-as. The ingest skill dry-runs a mapping and loads with elastic es bulk, and the Elasticsearch Query Language (ES|QL) skill writes a query that parses on the first try. Every step returns JSON, exits non-zero on failure, and can only do what your policy allows.

Skills for Elastic Cloud onboarding and provisioning, Elastic Workflows, and Kubernetes investigation are available today. Skills for Elasticsearch query, ingest, reindex, and index design, plus Kibana dashboards and alerting, are close behind.

What’s in the Elastic CLI technical preview, and what’s next

The preview covers all public Elasticsearch Serverless, Kibana Serverless, and Elastic Cloud APIs. Hosted-only 9.x Elasticsearch API coverage is nearly 100%, and hosted-only 9.x Kibana APIs will be added soon.

We’re actively planning more developer experience work, including broader coverage for all supported stack releases, more helpers for common workflows, more skills in the public catalog, and loading the same skills into agents that run on the Elastic platform itself. What shapes that list is hearing how you and your agents use the CLI. Tell us what’s awkward and what’s missing, along with what you’d automate next.

Install the Elastic CLI and Agent Skills

The Elastic CLI is available now on npm (Node.js 22+). Install it and the skills together:

Then, add a context and check it:

Even without a project, you can start a free Serverless trial in about a minute, with no credit card. If you already have a project, log in and create API keys for Elastic Cloud and your Elasticsearch clusters. Before pointing an agent at anything real, start with a trial project, a read-only key, and a scoped commands.allowed list. Be sure to take five minutes to read the security notes in the skills repo.

Replace all those curl commands in your Bash scripts, and add some usage instructions to your AGENTS.md. Then let your agent’s skills work efficiently and accurately with our APIs. Let us know what you think, and don’t hesitate to open an issue if you find a bug or if your use case isn’t well supported. Your feedback directly shapes what we build next.

Elastic CLI and Agent Skills resources

• Elastic CLI on GitHub and CLI documentation

• Elastic Agent Skills on GitHub

• agentskills.io specification

• Elastic Cloud Serverless documentation

• Report a CLI issue · Report a skills issue · Discuss

PromQL compiles to the same compute engine that runs ES|QL and inherits its planner and distributed execution, along with its release process. We didn’t build a second engine for this, and there’s no plugin to install.

This post is about how we built it.

Key takeaways:

• One engine: The implementation combines Elasticsearch’s mature distributed planning, storage, and testing infrastructure with its newer compute engine, which provides a columnar execution runtime. This lets PromQL reuse proven Elasticsearch capabilities while executing through a modern, native vectorized pipeline rather than introducing a separate runtime.

• One server: Elasticsearch implements the Prometheus remote write and query APIs directly, so Prometheus-compatible ingest and queries run without any additional plugins.

• Engineered for efficiency: Supporting PromQL required new engine primitives for range-aligned evaluation grids, backward-looking windows, dynamic label grouping, pipeline result reshaping, and compact wide aggregation keys. These primitives allow PromQL queries to execute efficiently end to end, with the relevant semantics implemented directly in the compute engine rather than through external post-processing.

• Compatibility measured in real use: In addition to Prometheus compliance tests, we built a differential-testing and quality-control pipeline over 2,000 PromQL queries collected from public repositories. 

Read more:

• Query Prometheus Metrics in Elasticsearch with PromQL

• Ship Prometheus Metrics to Elasticsearch with Remote Write

• Bringing Fire to Elasticsearch: Adding Native Prometheus APIs

Why run PromQL on Elasticsearch

Many teams already store logs and traces in Elasticsearch while running metrics in Prometheus or another dedicated metrics back end.

Prometheus and its ecosystem are strong and widely adopted, but large deployments can also bring operational sprawl and scaling challenges, along with limited retention. 

So we set out to combine Elastic’s highly optimized time-series database (TSDB) with a best-in-class metrics ecosystem. The result is a smaller observability stack, with fewer systems to operate and metrics storage that scales horizontally and supports long-term retention.

One engine: PromQL and ES|QL share the same compute engine

We made an early architectural decision not to run a separate PromQL engine next to Elasticsearch.

PromQL is instead another front end to the Elasticsearch compute engine.

This puts PromQL in the normal Elasticsearch development lifecycle. It uses the same planner, distributed execution engine, testing infrastructure, and release process as Elasticsearch itself.

To learn more about Elasticsearch’s query engine, check our blog.

Like ES|QL’s time series queries that use the TS source command, PromQL is translated into a highly optimized query plan and executed across the cluster. Nodes process columnar batches through vectorized operators, while partial results move through exchanges until the final result is assembled. 

This also means that PromQL and ES|QL operate over the same execution engine and time-series data. ES|QL can additionally extend a PromQL computation with post-processing that PromQL doesn’t support, such as lookup joins and inline aggregations.

For example, assume Prometheus request counters are stored in metrics-* and keyed by the service label. A lookup index, service_registry, maps each instance to its owning team and environment and to its service tier:

This architecture requires the execution engine to support PromQL semantics natively and efficiently rather than ES|QL syntax sugar. The following sections describe the changes and new execution primitives we introduced to achieve that.

One server: Prometheus remote write and HTTP API built into Elasticsearch

Query execution is only half of the story. The Prometheus ecosystem also expects familiar ingest and query APIs.

Prometheus protocols are the de facto standard for everything metrics in almost every team’s observability stack. So we built the HTTP API directly in the Elasticsearch server, which eliminated the need for a third component and tightened integration stability and performance.

On the ingest side, we added an endpoint for the Prometheus remote write protocol. It accepts Snappy-compressed Protocol Buffer messages, maps labels to TSDS dimensions, maps the metric name/value into metric fields, infers counter versus gauge mappings, and writes directly into TSDS. The built-in template is dynamic, so users don’t have to predeclare every Prometheus label or metric.

On the query side, Elasticsearch exposes Prometheus query APIs. A request enters through the Prometheus endpoint and is executed in the compute engine.

Running PromQL efficiently in a columnar engine

Sharing an execution engine doesn’t mean treating PromQL as syntax sugar over ES|QL. PromQL has different time, grouping, and response semantics, as well as workload characteristics that matter at scale. Supporting it efficiently required extending the compute engine rather than compensating in the API layer.

PromQL time grids: Aligning evaluation steps with TSTEP

Time-series query engines are optimized for grouping over time.

Elasticsearch normally groups timestamps with TBUCKET(...), which truncates each timestamp to a fixed interval boundary. Truncation is cheap and produces deterministic bucket boundaries. It also makes intermediate results easier to reuse.

Prometheus defines evaluation points differently. For a range query, timestamps are laid out as fixed steps anchored to the query range, rather than derived by truncating each sample timestamp. Two queries with the same step but different range boundaries can therefore produce different evaluation grids.

To preserve these semantics, we introduced TSTEP(...), which derives its grouping grid from the query range and step,  rather than truncating timestamps to globally aligned boundaries.

PromQL uses TSTEP(...) internally, preserving Prometheus timestamp semantics while still lowering the operation to a native Elasticsearch execution primitive.

Some might see this as a simple problem, but the nuances matter. 

Take, for example, a query that finds a 5m rolling average of a metric: 

At evaluation time T, the result represents the average over the preceding five-minute range: 

(T - 5m, T]

When the query is executed with a five-minute step, each output value is labeled with the upper end of its corresponding five-minute window.

Elasticsearch previously lacked this semantic and supported only forward-looking window aggregation functions:

We rewrote the window-evaluation path so that both ES|QL and PromQL use a common backward-looking windowing implementation:

Together, TSTEP(...) and backward-looking windows preserve the two time semantics that matter for PromQL range evaluation.

Dynamic label grouping: How PromQL without() resolves at runtime

Time grids determine when a PromQL expression is evaluated. Aggregation determines which input series are combined and which labels identify each output series.

For most analytical query engines, that identity is known when the query is planned. The planner can allocate grouping columns and choose an aggregation strategy. It also carries a fixed key through the execution pipeline.

That’s how ES|QL works:

The output series are grouped by an explicit key (cluster, namespace).

PromQL can express the same operation in the opposite direction:

Now we know which dimensions not to use. We don’t necessarily know the full grouping key until the query is executed. This is a small language difference with significant execution consequences. 

One possible implementation is to discover every label used by the metric, subtract instance and pod, and rewrite the expression into an ordinary by(...) aggregation. That adds a discovery phase before planning. It also becomes inefficient for high-dimensional metrics where only a subset of all possible dimensions may have useful value in a particular series. Most queries need only a small subset of the available dimensions, so carrying the entire dimension universe as an aggregation key wastes memory and adds bookkeeping overhead.

We instead extended the time-series execution path with dynamic grouping columns. The engine loads dimensions as the series are read and applies the exclusions per time series. This avoids making the grouping schema a prerequisite for planning and avoids carrying a large sparse set of grouping columns through aggregation.

Dimension packing: Keeping wide PromQL grouping keys cheap

The idiomatic way of writing PromQL aggregations involves heavy use of without(...) over by(...):

Excluding labels rather than explicitly listing them makes dashboards and alerts resilient to schema evolution. If a new label is added to the metric, the query continues to preserve it unless it’s explicitly excluded.

The consequence for the engine is that effective grouping keys can be wide. Many of those labels often have low cardinality, yet each still participates in every aggregation stage.

In a columnar engine, each grouping column is normally represented as a separate vector. Ten grouping labels therefore mean 10 vectors flowing through every aggregation operator:

Dimension fields are declared in the index mapping; the planner knows the schema up front, and grouping keys stay narrow and predictable.

In the columnar engine, each grouping label is carried as a separate vector or block. A key with 10 labels therefore requires 10 vectors to be read, hashed, compared, and retained by aggregation operators. As key width grows, so does the amount of data and bookkeeping that must move through the pipeline:

To avoid paying that per-column cost for every additional label, we introduced dimension packing. Before aggregation begins, the engine encodes the full grouping key into a single compact representation. Hash and comparison operations run on the packed key rather than on each block independently:

Packing lets hashing and comparison operate on a single compact key rather than an increasing number of grouping blocks, making aggregation overhead less sensitive to key width. Because the engine is shared, ES|QL time-series queries will benefit from this optimization as well.

Building the Prometheus HTTP API response inside the pipeline

Unlike Elasticsearch's ES|QL column-oriented response format, the Prometheus response is row-oriented.  The Prometheus API returns one result row per time series, with its samples represented as timestamp-value pairs. 

To support a compatible API layer, we had to regroup in the HTTP layer converting the columnar results into boxed row objects and accumulate them in map- and list-based structures until the complete Prometheus response could be produced. 

We replaced this with the TimeSeriesCollapse compute operator. It groups rows by series and aligns samples to the query’s fixed step grid. It emits the reshaped result as ordinary columnar pages containing one row per series, with aligned multi-valued timestamp and value blocks. And it preserves compact, vectorized block representation throughout the pipeline: 

The HTTP layer can now serialize those blocks directly, avoiding the maps, lists, boxed objects, and associated allocations required by the earlier implementation.

Testing PromQL compatibility against 2,000 real queries

Prometheus compliance tests were our starting point.

Even though they gave us a strong baseline, they didn’t tell us how frequently individual PromQL features appear in real workloads. To complement that baseline, we built a second test corpus from over 2,000 PromQL queries collected from public repositories. 

We then classified those queries by the language features and expression patterns they exercise:

For each compatible query shape, we run the same query against Elasticsearch and Prometheus and compare the results. 

In addition to that, we actively rely on fuzz testing, which catches issues that unit tests alone are unlikely to expose, including differences in timestamp alignment, label retention, aggregation behavior, range-vector evaluation, and response encoding.

Which PromQL functions and APIs are supported in 9.5

Since 9.4 (technical preview), PromQL support in Elasticsearch has expanded substantially. In Elasticsearch 9.5, both the PROMQL command in ES|QL and the Prometheus HTTP APIs are generally available (GA), with more than 80% of the PromQL workflows in our real-world corpus now running without modification:

The main additions since technical preview are:

Feature

Example

Status

Prometheus remote write ingest

POST /_prometheus/api/v1/write

GA in 9.5

Range queries

/api/v1/query_range

GA in 9.5

Instant queries

/api/v1/query

GA in 9.5

Metric metadata and build info

/api/v1/metadata, /api/v1/status/buildinfo

GA in 9.5

Native histogram functions

histogram_quantile, histogram_count, histogram_sum

GA in 9.5

Per-selector offset modifiers

[5m] offset 1h

GA in 9.5

Top-level or operator

rate(a[5m]) or rate(b[5m])

GA in 9.5, up to eight operands

Prometheus remote write ingest

Elasticsearch accepts Prometheus remote write (v1) messages directly:

Snappy-compressed Protocol Buffer messages are decoded, and labels are mapped to TSDS dimensions. Metric names and values are written into the time-series index. The built-in template is dynamic, so users don’t have to predeclare every Prometheus label or metric.

Range and instant queries through the Prometheus HTTP API

Both range and instant query endpoints are available:

Range queries return matrices evaluated over a time window, and instant queries return vectors evaluated at a single timestamp. These endpoints can be used by Kibana, Grafana, or Prometheus-compatible alerting tools, and custom dashboards.

Metric metadata and build info endpoints

Elasticsearch exposes metadata about available metrics and a build-info endpoint:

The metadata endpoint returns metric types and help text, and the build-info endpoint returns the Prometheus-compatible server version. Grafana and other tools use these endpoints for feature detection and UI behavior.

Native histogram functions: histogram_quantile, count, and sum

Elasticsearch supports the main PromQL operations over native histograms:

Native histograms adapt their bucket layout to the data, providing useful precision across a wide value range without requiring users to configure every bucket boundary in advance. Classic histograms continue to work alongside native histograms.

Per-selector offset modifiers in PromQL

Offset modifiers shift a selector’s time window backward:

This returns the request rate from one hour earlier. Per-selector offsets are commonly used to compare current traffic, latency, or resource usage with an earlier baseline, such as the same period one week ago.

Top-level or operator in PromQL

Elasticsearch supports the top-level PromQL or operator:

In PromQL, or isn’t a Boolean operation. It performs a union between two sets of time series. Results from the left side are retained; a series from the right side is added only when its label set doesn’t match a series already returned by the left side. This is useful during migrations where the same logical metric may exist under an old and a new name.

The implementation follows Prometheus’s left-side precedence rules and preserves the __name__ label. Top-level chains of up to eight operands are supported.

PromQL features not yet supported in Elasticsearch

GA doesn’t mean complete PromQL compatibility. Some less common and more complex parts of PromQL remain unsupported. These gaps now define the next phase of the work: 

Feature

Example

Status

Advanced vector matching

on(instance) group_left

Planned

Sorting and ranking

topk, bottomk, limitk, sort, sort_desc

Planned

Label manipulation

label_replace, label_join

Planned

Absolute time modifier

@ 1710000000

Planned

Mixed-offset compound expressions

rate(...) - rate(... offset 1h)

Planned

Alerting and target endpoints

/api/v1/alerts, /api/v1/targets

Out of scope

Advanced vector matching with on() and group_left

Some binary operations that require Prometheus vector matching aren’t yet part of GA.

For example, this query divides per-instance request rates by a per-instance capacity metric:

The on(instance) clause specifies which labels identify matching series. group_left permits many request-rate series to match a single per-instance capacity series, while retaining the labels from the higher-cardinality left-hand side.

These expressions are common when joining a detailed metric with metadata or a lower-cardinality capacity metric. Basic binary expressions are supported where applicable, while the remaining vector-matching forms are planned work.

Sorting and ranking: topk, bottomk, and sort

Prometheus sorting and ranking functions are also not yet part of GA:

This returns the 10 services with the highest request rate. Similar queries are widely used in “top offenders” dashboards for traffic, latency, errors, and resource consumption.

The remaining functions include:

Label manipulation with label_replace and label_join

PromQL can construct or rewrite labels during query evaluation. These functions are particularly useful when dashboard variables, naming conventions, or label schemas don’t match exactly:

This creates an environment label from the cluster label.

Another common example combines existing labels into a display-oriented label:

This produces a target label, such as payments/api-7f6d9. label_replace(...) and label_join(...) aren’t yet included in GA.

Advanced time modifiers: The @ modifier and mixed offsets

Several advanced time modifiers and expression forms remain outside the GA scope.

For example, an absolute @ modifier evaluates a selector at a fixed Unix timestamp rather than at the query’s normal evaluation time:

This is useful for comparisons against a fixed historical point.

PromQL also permits expressions in which the two sides use different offsets:

This compares current traffic with traffic one hour earlier. Per-selector offset is available in GA, but not every combination of offsets and compound expressions is part of GA yet.

Prometheus API endpoints not yet implemented

In addition, the Prometheus HTTP API surface isn’t yet fully complete. Notably:

Alerting metadata through:

used by tools that inspect active alert state.

Target discovery through:

used to inspect scrape targets, health, and labels.

These endpoints concern Prometheus server and scrape-target state rather than querying metrics stored in Elasticsearch.

For the full list of limitations, see the PromQL limitations page. 

Try PromQL in Elasticsearch 9.5

To query Prometheus metrics in Elasticsearch 9.5 or Serverless, see the PromQL documentation and the Prometheus HTTP API reference.

However, with the rapid progression of coding agents, performance optimization has become a task we can tackle semiautomatically. Unlike many software engineering challenges, optimizing code offers a cheap and objective verifier. If you ask an AI model to make code faster, there’s a hard number at the end telling you exactly what happened, backed by profiling tools that explain why. This makes performance a perfect candidate for automation, provided you can actually trust the numbers.

If you simply point a coding agent at a benchmark, you typically get low signal-to-noise: wins that fall inside the variance of the environment, or variations caused by thermal throttling rather than better code. To capture optimizations that actually benefit Elasticsearch users, we had to bridge the gap between "checkable in principle" and "checked in practice." We built a highly trustworthy measurement loop: a harness that assumes the agent will be wrong a good fraction of the time but reliably catches and proves it when it’s right and then helps guide it where to look next.

Once the machinery is in place, the results speak for themselves. By letting this harness loose on the codebase, we've already begun uncovering meaningful wins across the stack. In part 2 of this post, we’ll dive into some examples it has found so far, including string conversion inefficiencies in Elasticsearch Query Language (ES|QL), an improvement to our NEON vector dot product implementation, and an upgrade opportunity for the gzip library we were using. In this part, we’ll take a look at the design choices we made and how they relate to the broader topic of effective harness development.

The AI code optimization pipeline architecture

The first step in any software engineering problem is to identify the correct high-level components. We made an architectural choice that turned out to be very helpful for this problem: separate understanding where opportunities exist from the loop making code changes. The agent starts with a real workload but only uses it to mine information about where to seek performance improvements. At this stage, it’s instructed to go broad and consider a range of performance-related signals. Once it has found and classified the hot spots, the agent reads the context of the code around them to understand the optimization opportunities. We use a separate task to condense the ranked list of hot spots into artifacts that a loop can iterate against in minutes: a microbenchmark that we prove exercises the hot path in its real operating regime. Finally, we use a proposer-verifier loop to actually make changes to the codebase to improve performance on the benchmark. This hands off to validation to assess the impact on real workloads at the end. Our CLI (atune) supplies the tools this process needs, and the rest is largely automated by a set of task-specific instructions.

For context, our high-level architecture looks like the following. Pink boxes are the humans, and teal boxes are the agent. There are three task types, one skeleton loop, and one referee.

An exploration task profiles a real workload and produces ranked opportunities; a human promotes one into an exploitation task, which iterates against an approved microbenchmark and commits each accepted experiment; a validation run on the real workload guards the result before a human reviews and opens the PR. Where no benchmark covers the hot path, a benchmark task authors one and a human approves it into a registry. A performance atlas informs every task and accumulates what each one learns.

Why performance optimization suits autonomous agents

Three properties make a task ideally suited for autonomous work, and it's worth being explicit about them because they provide a checklist you can use to evaluate automation candidates. You want:

1. An objective verdict so that the agent can be held to something other than its own opinion.

2. A dense guiding signal so that it knows where to look next instead of guessing.

3. A bounded blast radius so that being wrong is affordable.

Performance gives you all three. Benchmarks provide the verdict and profilers provide the gradient, while a rejected patch costs you wall-clock time rather than correctness. The change gets reverted, and the reason gets recorded. Life goes on. Regarding the first two, we've come to think the gradient matters more than the verdict. ″This got faster″ is binary, whereas a profile hints at what to try next. An agent can generate its next hypothesis conditioned on a rich guiding signal, the richer the better, rather than grinding through a list.

Signals are what the agent gets to see

A useful mental model is that the CLI is the agent's sensory apparatus. That changes how you design each command. Rather than exposing a capability, you design it to return a clear and concise answer to a question about the task at hand, and, when relevant, an explanation the model can reason over, instead of raw data it has to parse and interpret. These are the signals that the agent acts on, and we ended up with the following for our harness:

Signal

Question it answers

Facet-decomposed macro profile

Where in the code does real workload time go, per query type?

Allocation and lock sampling, in the same capture

Is the cost cycles, garbage, or contention?

Cost-composition classification

Is this in scope compute, other product code, GC, JIT tax, or parked threads?

Input-shape instrumentation

What does the workload actually feed this code?

Statistical verdict

Did this change help, at this measured noise floor?

Allocation-rate comparison

Did the new code end up allocating more?

Interpreted disassembly

Why did that result happen?

End-to-end A/B guard, with differential profile attribution

Did anything appear to break, and was it us?

Environment check

Is this machine even fit to measure right now?

Upstream duplicate search

Has somebody already reported or fixed this?

Four of these signals are worth dwelling on, because in each case the tool encodes a judgment that the agent would otherwise have had to keep making by hand.

Facet-decomposition is the clearest example. Blended CPU shares hide breadth: if you profile a mixed query workload, the grouping hash map insert and the percentiles sketch update can both show up as single-digit percentages of the total run time and look comparable. They aren't comparable at all, because the hash map insert is paid for by nearly every aggregation query, while the sketch update is only paid for when somebody asks for percentiles. So the profiler runs each named query facet as its own race, and every opportunity the agent records carries a breadth field (universal, broad, or narrow) and gets ranked by headroom × tractability × breadth. A universal 3% beats a narrow 10%. Putting the ranking function in the tool prevents it from having to be rediscovered on every run.

The cost-composition classification works as a router. Rather than handing the agent a flat top-N list of frame names, it buckets every sampled stack into ″in scope compute,″ ″other product compute,″ ″GC,″ ″JIT and safepoint overhead,″ ″off CPU waiting,″ and ″parked threads.″ Each of those buckets implies a different kind of investigation. If GC is above about 15%, the real target is allocation rate, and the CPU top frames will actively mislead you, because they show where objects were collected rather than where they were created. If JIT and safepoint overhead is above about 25%, you're looking at a ceiling rather than an opportunity, since no in-scope code change will move it. If threads are parked and core utilization is low, this is a concurrency problem and CPU flame graphs are the wrong instrument entirely. We wrote that mapping into the playbook as a table, so the model (even a cheap one) reads a profile the way that an experienced engineer would, rather than reaching straight for the top frame. The classification is a prior for forming a hypothesis, though, not a substitute for evidence, so the agent still has to cite specific frames when it proposes an experiment.

Interpreted disassembly is a tool we hadn't originally provided, but it most definitely earns its place. It helps answer why, the question that unblocks the next hypothesis. Flame graphs tell you where the time goes; they rarely tell you why a change made things worse. So atune asm runs the benchmark briefly with the JIT told to print the assembly for one hot method, captures both sides of the working-tree diff, reduces each to the final C2 compilation, normalizes the addresses, and diffs them. The diff alone would likely still be 4,000 lines of aarch64, so on top of it sits an interpretation layer: a per-mnemonic delta, a net instruction count, the compilation tier that was actually captured, and a vectorization signal that counts vector register references on each side and raises a warning when they're eliminated, halved, or narrowed from Advanced Vector Extensions (AVX) to Streaming SIMD Extensions (SSE) width. The playbook then maps mnemonic patterns to causes:

Pattern in the diff

Likely cause

b.eq/b.ne up, csel down

New unpredictable branches

Clusters of str/ldr against the stack pointer

The compiler ran out of registers

NEON loads replaced by scalar compares

The vector path degraded

In one experiment, the agent fused two SIMD mask extractions into one, and the benchmark regressed by 26%. The vectorization warning explained it in about 10 seconds. Without that tool, the agent has a dead end and no working model of the machine; with it, it has a corrected model and several new ideas.

The fourth signal is less a single tool than a habit; the instruments check themselves. Core utilization is derived two independent ways, from sample density and from process sampling, so the two can be compared. The classification is rejected if the unclassified bucket exceeds a budget, on the grounds that a breakdown which can't account for its own samples shouldn't be reasoned over. The disassembly capture warns when the compilation it caught isn't the steady-state one. The upstream duplicate search is restricted to read-only commands, and that restriction is enforced by a test that greps the source, so no future edit can quietly reintroduce the ability to file anything. Each of these exists because a tool that can be confidently wrong is worse than a tool that is merely absent.

One small piece of design is worth highlighting as a specific instance of good return practice. atune compare returns 0 for improved, 1 for no change, 2 for regressed, and 3 for error, and the loop branches on this code. That means no parsing and no ambiguity about what the verdict was. Plus, no tokens are spent interpreting prose.

If you take one thing away from our CLI design, it’s a broader design principle. In a general setting, the interesting thing isn't the individual signals we found useful to understand performance; it's that the CLI is capturing and packaging the judgment of an experienced performance engineer into tools that return answers rather than raw data. Structurally imposing good judgment about the problem an agent is tasked with improves outcomes. The right CLI is as much part of that story as the instructions. Furthermore, tokens are saved by tools that return decisions and digests rather than data. That means a comparison verdict instead of raw JMH output; a triage summary sitting on top of a 4,000-line disassembly diff.

From 20-second probes to hours of validation

Building a verifier that you can afford to consult is a separate problem from building one that you can trust. This covers the affordability part. Or, if you like aphorisms, real workloads are where truth lives and where iteration goes to die. A macro profile takes 45 to 60 minutes, and an end-to-end validation run takes hours. But a microbenchmark takes minutes. That's why the exploration-then-exploitation split works; you go broad on the real workload once and then hand off to a microbenchmark that you can iterate against in minutes.

The catch is that the handoff is only sound if the microbenchmark exercises the hot path in a realistic operating regime; that is, with the right cardinality and right data distribution. If you get that wrong, your fast loop spins fast but in the wrong direction. Until we finalized the handoff procedure, we saw cases where the agent accepted changes on a benchmark whose key distributions happened to flatter it, and only the end-to-end run caught the problem.

Handing off from exploration to exploitation

The handoff between the two phases is structured rather than informal. An exploration task's primary deliverable is a set of opportunity records, and each one carries the scope paths that an exploitation task would be allowed to edit, a headroom estimate, a classification (constant factor, structural,

allocation, or concurrency), and the benchmark it would gate on (or an explicit "no benchmark coverage" flag, if none exists). Each also carries a narrow test pattern so that the correctness gate stays cheap. The test pattern field exists because of a specific incident; an exploration task omitted it, and the resulting exploitation task ran a very heavy test suite on every experiment. The fix was to change the upstream artifact rather than add an instruction downstream. This is a pattern we use repeatedly; make and record decisions as early as possible rather than re-derive them each time.

Validating a new benchmark before it can gate anything

Where benchmark coverage is genuinely missing, a dedicated benchmark task authors one, and that new benchmark has to pass validity checks before anything can rely on it. Two of them are mechanical: what fraction of the benchmark’s hot self-time comes from frames that actually appear in the production profile and whether the parameters fall inside the input shapes we measured. The third is a checklist that the agent has to attest item by item. It exists to avoid the JIT getting a simpler world than production.

1. Inputs have to be reshuffled rather than fixed or sorted, so the branch predictor doesn’t get too good.

2. Results have to be consumed, or dead-code elimination deletes the thing that you meant to measure.

3. Inputs must not be compile-time constants, or they get folded away.

4. Call sites have to see roughly the product mix of types, because a monomorphic call site inlines, whereas a megamorphic one doesn’t.

A human then approves it into a hash-pinned registry. Until that happens, it's inert, because task setup refuses any task citing an unapproved benchmark. We think of that approval as the strongest gate in the system, and it's deliberately placed. An approved benchmark can decide accept or reject in every future task, so it's the one place where we ask for a human signature on an artifact rather than on a decision.

The validation ladder

Underneath all of this sits a hierarchy of feedback mechanisms with the property that each rung is cheaper and weaker than the one below it, and the cheap tiers are for rejection only.

Tier

Cost

Role

probe

~20–60 s

Directionally right? Can never accept

codegen capture

~4 min

Why did that happen?

screen

~5–15 min

Cheap statistical filter

confirm

~20–60 min

The accept decision

end-to-end

hours

Regression guard, advisory

The asymmetry between accept and reject is doing real work here. A probe is a single paired fork, whereas the accept predicate requires a full confirm run with a matching calibration record. Agents are very good at telling believable stories; indeed, they're trained on many tasks judged by both LLMs and by humans, so being convincing is actively rewarded. We don't want an agent to be able to promote a cheap signal into a decision by being persuasive about it.

For this task benchmark, wall-clock is a real cost consideration. A confirm run might take an hour. So one has to weigh carefully all the costs involved when choosing the setup. A model that lands one hypothesis in four typically beats a cheaper one landing one in 10 by a margin on end-to-end metrics. The usual instinct to down-spec the model on a long-running loop is exactly backward here. When your loop has an uncertain outcome and significant costs beyond the tokens it consumes, you may well find yourself in the same situation.

How do you know a performance improvement is real?

Is it faster? is a statistical question. So we made the accept predicate code rather than judgment and put it somewhere the agent can't bypass. There are four ideas in the accept decision, and in each case, the alternative we rejected is as informative as the choice we made.

Forks are the statistical unit

Each JMH fork collapses to its mean, and verdicts come from an exact two-sided Mann-Whitney U test at α = 0.05 plus a seeded bootstrap confidence interval over three to five fork means per side. The reason is that iterations within a fork share JIT and heap state and are therefore autocorrelated, so treating them as independent samples manufactures significance out of nothing. We rejected comparing single-run scores, which is pure noise, and iteration-level t-tests, which can be confidently wrong. Seeding the bootstrap means that a rerun reproduces the verdict exactly because the agent needs to be able to tell the difference between a result that changed and a result that was never stable.

Pair candidate and baseline in time

The screen (three forks, optionally over a subset of parameters) exists only to kill bad hypotheses in maybe 10 minutes instead of an hour. The accept decision itself comes from a confirm run that measures candidate and baseline back to back using a stash-flip. In an unstable environment, thermal and background drift only cancels if both sides ran under the same conditions, so an hours-old baseline is really a different experiment.

The noise floor is measured, not assumed

The minimum effect size that a task will accept has to clear the A/A-calibrated coefficient of variation for that specific benchmark on that specific machine, and both the confirm run and the comparison refuse to proceed without a matching calibration record. A 1–2% improvement on a laptop is indistinguishable from noise, and because the floor varies by benchmark and by JDK, any global constant you pick will be too loose somewhere and too tight somewhere else.

The accept rule is composite and deliberately conservative

A parameter combination counts as improved only if p < α, the effect clears the calibrated floor, and the confidence interval excludes zero. Overall acceptance then requires that nothing regressed (not the primary benchmarks and not the guards) and that at least one primary combination improved. The headline figure is the geometric mean of the per-combination speedup ratios, which is always positive and composes across experiments, so a task's cumulative improvement is a meaningful number rather than a sum of incomparable percentages.

What must not get slower

Guards deserve a note of their own, because they answer a different question to the primary benchmarks; not Did this get faster? but What must not get slower while it does? The task definition lists them separately for that reason, and they're typically the operations and the input regimes that the change isn't aimed at. If you're optimizing insert throughput on a hash table, iteration is a guard and so is a collision-heavy key distribution. A change that improves the common case by weakening the hash function will look good on uniformly distributed keys while catastrophically degrading more adversarial inputs. We know that because it happened; the collision distribution was missing from the matrix that accepted one of our early experiments, and the task now carries a comment telling future readers never to drop it again for a hash-quality-sensitive scope. While guards cost wall-clock on every confirm, they also surface edge-case regressions, and omitting them can be much more costly in the long run.

Runtime isn't the only thing worth guarding, either. The confirm run also captures normalized allocation rate on both sides and flags any change that buys speed with more than about 15% extra garbage. That one is advisory rather than blocking, because sometimes the trade is the right one. However, it's the kind of regression a purely time-based accept rule would happily wave through but might raise a red flag to an experienced performance engineer with better understanding of the calling context.

The end-to-end gate is one-sided and default open

The end-to-end gate is a different statistical problem: small n, high noise, and a very strong prior that we should accept based on our microbenchmark results. Our first design treated accept and reject on an equal footing, and it produced multiple clearly spurious rejections, so the redesign is one-sided and default open. An operation is flagged only when the median regression exceeds the threshold and every candidate repetition is slower than every base repetition. That full-separation criterion is the nonparametric one-sided test at this sample size, and it's robust to the single outlier repetition that would occasionally fool us otherwise. A flag then also has to be corroborated against a differential CPU flame graph, where only a rise in the task's own in-scope CPU share counts as real. Near misses get reported for transparency but don't trigger triage, and nothing is ever auto-rejected; a flag is a request for human attention rather than a verdict. We have a final backstop which is the large suite of performance tests that we already run against Elasticsearch on a daily basis.

The one-sided gate lesson generalizes beyond benchmarking. A noisy gate should be one-sided and default open where there is strong prior reason to accept. A symmetric threshold on a noisy signal doesn't just cost you real wins, it teaches the loop to distrust its own instruments, and that’s a much more expensive failure.

Exploration and exploitation need different permissions

Exploration and exploitation might look like two phases of one activity, but they have different inputs (a macro workload versus a pinned scope) and different outputs (ranked opportunities versus commits). They also have different failure modes, which means they want different permissions. We made the split a first-class property of a task, which lets us enforce it; an exploration task literally cannot commit. The baselining, comparison, checkpointing, and validation CLI all refuse exploration tasks, benchmarking allows probes only, and every probe diff is always reverted. A broad, speculative survey is safe because nothing it does can edit the code.

A nice ancillary benefit is prompt focus. Each type reads one playbook, in full, with the others explicitly not loaded. If you try to write a single document covering both "find where the headroom is" and "land a validated win inside this scope", you get something that does neither well, because the instructions for good exploration (follow the profile, widen the net, a broad survey is preferred) are close to the opposite of the instructions for good exploitation (one hypothesis at a time, minimal diff, never widen scope).

AI agent memory: Journals, knowledge bases, and postmortems

Sessions are ephemeral, but what you can learn from them isn't, so the harness accumulates three durable assets, plus one disposable view derived from them. These are a journal of the code changes we’ve tried, a knowledge base of how the code performs, postmortems of when the harness failed and, because sessions can be stopped and resumed, a session summary. What makes them work together is a clean ownership rule about which kind of fact goes where.

The journal records what we tried and measured. It's append-only, one file per task, and one record per experiment, and it's written before the code is edited. Rejections carry a forward-looking note in the form "do not retry X because Y", which is probably the highest value line, because it's what stops the next session re-deriving a dead end. Records also carry the environment and the driving model, which means that hypothesis hit rates are comparable across models.

The knowledge base records how the code works and how it performs. It's an indexed collection of per-area summaries, each stamped with the commit it was written against, and every playbook ends with an upkeep step that appends whatever durable facts the run turned up. Because the performance characteristics of the JDK also change from time to time, for example, a new Vector API might implement vector masking more efficiently on AArch64, findings from profile data are also tagged with the JVM version they apply to.

The postmortems record mistakes that the agent has made in the past, and they're indexed by symptom rather than by date. The question a session actually has when a number looks wrong is Have I seen this shape of wrongness before?, and a chronological list doesn't answer it. So the rows read like "validation fails on operations structurally unrelated to your diff", or "screen reports no matching combinations".

Keeping the journal and the knowledge base distinct sounds pedantic but isn't because without the rule, both of them turn into a diary that has a tendency to bloat the context window or miss critical information in context.

The disposable state is a session handoff, and our advice is to never write it by hand and to avoid asking a model for a session summary, if possible. Our harness regenerates a one-page digest mechanically from the journal, so a resumed session doesn’t spend time and tokens reconstructing where the task had got to. Because it’s derived rather than authored, it can’t drift from the record in the way hand-maintained content does. That’s also why it isn’t part of the audit trail; because it’s cheap to regenerate it from the journal.

Mechanical session handoffs point to two lessons, and they turn out to be the same one. Every time a person appears in the loop, it’s a source of friction and an opportunity for error. And every time you reach for a model, ask whether code can do the same job. This seems like an odd thing to advocate in a project whose primary premise is delegating to a model, but the habit is easy to fall into once you have one to hand. Judgment is expensive, wherever it happens to sit, so the person and model both have to earn their place on merit.

Two further things are critical for durable agent memory. The first is that knowledge rots, so you have to lint it. Elasticsearch's main branch moves daily, which means the knowledge base's file citations go stale, so one linter checks them. Another checks that every command, flag, and path cited in the agent-facing docs actually exists, that every postmortem is linked from the index, and that every task type has a playbook, and it runs as part of the test suite. Treating prose written for an agent as a testable artifact is the reason it stays true. The second is that loading discipline is half of memory. The instruction is to load the index and then the one or two summaries matching the task's scope; never to bulk-load the rest. Memory you can't afford to read isn't memory.

Coding agent guardrails: Containment, scope, and stop conditions

Elasticsearch is millions of lines of code, and an unscoped "make it faster" run against a codebase that size is unreviewable and unfalsifiable. It’s also expensive. The converse is that a verifier only protects what it can see, so we have to apply the same boundary to what it can change. The outcome is a task that fixes its goal, allowed paths, benchmarks, thresholds, and stop conditions before any code is edited, and none of those are things the agent may change during a run.

The human owns the safety envelope, defining scope, thresholds, and stop conditions before any code is edited, and owning every handoff that crosses a trust boundary. The agent session owns judgment: reading profiles, forming one hypothesis at a time, editing inside scope. The atune CLI owns mechanical enforcement: scope checks, correctness gates, statistics, calibration, stop conditions, environment checks, and the signals. Persisted state is the record: a worktree pinned at a base commit, an append-only journal, and generated reports.

Containment happens in two layers. The first layer is coarse and task-agnostic; it’s a single static permissions file lets the session edit the per-task worktree (which includes a local branch of Elasticsearch, all journal entries, and CLI artifacts) and the knowledge base, and it denies the pristine Elasticsearch clone, the task definitions, the harness config, and the audit trail. The second is precise and per-task: a git-level scope check that runs over tracked and untracked files before any build, so an out-of-scope edit is rejected before it can be benchmarked or committed.

Two containment layers have a nice corollary. Widening the first layer to the whole worktree costs nothing, because nothing out of scope survives the second one. Coarse containment plus precise scope beats trying to make a single mechanism do both jobs, which is what we tried first and which left us with a permissions file that needed editing for every new task.

Stop conditions are mechanical. There's a maximum number of experiments and of consecutive rejections, along with a cumulative improvement target, and proposing a new experiment is refused once one of them fires. A human can override; the agent can't. That asymmetry is what makes the gates independent of which model happens to be driving.

Tests are add-only. New test files ship with the checkpoint, and modifying or deleting an existing test is blocked by the scope check. This closes the single most tempting shortcut in the entire problem space by construction rather than by instruction, which seems like the right way to handle any shortcut you'd otherwise have to keep asking an agent not to take.

The harness lives outside the code it optimizes. The Elasticsearch clone is a separate, gitignored directory, and each task gets a worktree pinned at a base commit. The payoffs compound; the subject repo stays pristine and upstream mergeable with nothing related to the harness leaking into a PR, and the audit trail is versioned independently of a codebase that moves daily. Also, tasks are isolated from each other and from any developer checkout. Targeting a newer Elasticsearch means creating a new task rather than repointing an old one, because that task's numbers are tied to its base. It also means that the harness is retargetable in principle, since the Elasticsearch-specific parts are configuration, knowledge base, and benchmark registry rather than architecture.

The decisions the agent never makes

The rule we settled on is that the agent runs the loops and a human owns every step that crosses a trust boundary. That is creating work, blessing a measurement instrument, publishing a branch, or acting outside the repo. None of those are in the agent's allowlist, and each has a reason worth stating:

• Humans have to sign off on the task because the thing being constrained can't set its own limits.

• Calibrating the noise floor needs to be done once per benchmark, per machine, and by default is measured rather than assumed. However, it's rather expensive and we allow a human to override if they know the environment well.

• Deciding what to optimize next by promoting an opportunity is a judgment and a new scope.

• Since benchmarks go on to gate other tasks, we consider reviewing this artifact part of the correctness safety net.

• We leave outward-facing actions, such as pushing a branch or filing an issue, to a human until we're confident in the process.

• We allow actions to be forced, but the override has to sit outside the thing being overridden.

How the human actions get surfaced in the workflow matters. The generated report and the session handoff both print the human actions currently due, at the moment they become due, rather than leaving them to be inferred from the playbook.

We're deliberately not taking a position on how permanent the manual processes are. The right amount of supervision for a new technology is an empirical question, and we'd rather measure it than argue about it. We’ve started with a relatively high degree of supervision because that's the cheap direction in which to be wrong (a gate you never needed is easier to remove than a regression you shipped) and because the harness makes the question answerable. Every gate is a named, logged transition, so over time we can see which of them ever changed an outcome and which only ever cost friction. In summary, measure first, and then refine.

Building the harness is the same kind of loop

A lot of the harness design didn't fall out of an initial design document. The signal set, the ranking function, the shape of a task, and the exact wording of a playbook rule each came from watching a run go wrong. If there's one piece of advice here that generalizes, it's to use the thing before it's ready and to instrument your own disappointment.

The clearest example is a rule we now call distrust surprising results. A validation run reported that every operation had regressed, the worst of them by 14.8%. It was wrong twice over. A target operation pattern had overmatched a completely different code path, and a stale output directory from an earlier run was being read alongside the new one. Offered a coherent story, the agent took it and reverted a change that was actually good.

What went into the playbook after that incident is not "be careful." It's a three-step check to run before acting on a surprising verdict:

1. Trace the code path, and confirm that the thing which moved can even reach your diff.

2. Read the raw per-repetition data rather than the summary, and recompute one headline number by hand.

3. Compare the report's shape against a known good run, because a structurally different report implicates the pipeline rather than the code.

Alongside that, there’s another important rule, which is if the agent concludes that the harness is buggy, it must not fix it mid-run, because a mid-run harness change makes every result in that run incomparable. It should journal the evidence and stop.

Improving the harness is itself a loop worth describing. Asking the model to review its own transcripts and the harness documents, and to propose the rule itself, usually works well. It's good at spotting where its own instructions were ambiguous, in a way that's hard to reproduce by rereading the instructions yourself. What makes that output useful is having somewhere for it to land: a terse rule in the playbook, the narrative in a dated postmortem, a symptom keyed index row, and a linter that keeps the citations honest.

Restraint turns out to be part of the same discipline. The design document carries an explicit list of extension points that we've deliberately not built, because the need for them is still speculative. That's the same "don't guess, wait for evidence" rule we impose on the optimization loop, applied to ourselves.

How this applies beyond performance optimization

A few of these themes aren't specific to performance work or to Elasticsearch.

Verifiable work is the current frontier. The same insight drives reinforcement learning with verifiable rewards, and it’s what AlphaEvolve is built around. The tasks agents are consistently good at are the ones that come with a cheap oracle (tests, compilers, benchmarks), and the interesting move isn't finding more such domains but manufacturing oracles for domains that lack them. Performance is an instructive case precisely because the oracle is, in some senses, obvious, and yet building it still took most of the engineering.

The referee pattern generalizes, too. Separating a fallible optimizer from mechanical enforcement is the same shape as sandboxed execution and policy engines: judgment in the model, invariants in code. The practical consequence is that the system's safety doesn't depend on which model drives it. A weaker model wastes benchmark time, but it can't corrupt the code or accept a bogus win.

Goodhart is a standing adversary for any optimization task that uses an agent, and it has a formal treatment worth reading. An agent optimizes exactly what you tell it to, so the two-tier benchmark structure, add-only tests, benchmark approval registry, adversarial guard distributions, and a marker file that makes timing an instrumented build mechanically impossible are all one design theme wearing different clothes.

Tools are context engineering. The consensus is drifting away from "expose everything" and toward a few well-shaped tools that return digests: the principles of progressive disclosure, self-documenting interfaces, and verdicts rather than payloads.

Memory is becoming architecture. A rules file, per-task playbooks, a durable knowledge base, and an append-only journal form a hierarchy with different lifetimes, owners, and loading rules, and the hard part is eviction and staleness rather than storage; hence, the linters.

Finally, human-in-the-loop is a dial rather than a switch, so where it should sit is something to measure per domain rather than assert.

What's in part 2 of this post

The harness design is a set of hypotheses about what autonomous performance work needs, and the harness was built so that we could test them. Part 2 is that test: the first four PRs we raised using it, their gains, how many hypotheses it took to get each one, which gates actually caught something, and where the harness got in its own way. That includes the changes which didn't survive end-to-end validation, since as usual, the rejections are as informative.

For new users, this is the fastest way to get high-quality vector search running. If you already use Elasticsearch, the new offering is vector search on the platform where your data already lives, with no new system to adopt. Elasticsearch Vector Database supports a range of scenarios, from grounding a large language model (LLM), to giving an AI agent retrieval and memory, to serving hundreds of billions of vectors. Spin up a new project and get started in minutes.

One engine, every vector use case

Elasticsearch Vector Database is built for anyone building applications using vectors:

• RAG: Retrieve the right context for your LLM with dense and sparse vector retrieval, or go with hybrid search combining both vector and lexical retrieval. The quality of your generation improves with the quality of your retrieval.

• AI agents: Give agents fast, filtered retrieval over documents and conversation memory, with the low latencies that multistep agent loops demand.

• Semantic search: Match on meaning, not keywords, with one field type and zero pipeline code.

• Recommendations and similarity: Find nearest neighbors across products, images, or whatever content you have, at scale.

Everything your vector workload needs, optimized out of the box

Building a vector-based application means wiring together several separate pieces: setting up and hosting embedding models, indexing your documents through them, storing the vectors efficiently, applying the embedding model to each query, matching against the vector store, and finally, retrieving the documents behind the matches. Elasticsearch Vector Database handles all of it for you, with no additional configuration or setup.

Vector indexing with vectordb_document index mode

vectordb_document index mode, a new index configuration purpose-built for vector-first workloads, is on by default, so you get the settings that experts would choose. Here's what it turns on:

• bfloat16 by default: Vectors are stored at half the size of float32 with negligible impact on recall, cutting your disk footprint roughly in half before quantization even enters the picture.

• Source vectors excluded: In Elasticsearch, your embeddings already live in the index structures used for search; keeping a second raw copy in _source just inflates storage and slows down fetching results. We exclude the duplicate so responses return faster and you store less.

• The right files preloaded into cache: The data structures that vector queries touch first are warmed into memory ahead of time, so your first (and your thousandth) query is lightning fast.

• Parallel merging: Merging consolidates segments into better-organized vector structures, which lifts both recall and latency, and running those merges multi-threaded means you get there faster.

Vector storage, compression, and auto-tuning

• Your vectors are compressed automatically. Better Binary Quantization (BBQ) shrinks vector memory footprints by up to 32x while preserving recall, and DiskBBQ reduces memory requirements further for large-scale workloads. 

• Opt in to auto-calibration, which tunes each segment's quantization to your data and retunes on every merge as data drifts. When tested across 18 datasets, queries per second (QPS) improved by an average of 16.7%, with recall gains in most of them.

Embeddings on managed GPU inference

• Generate embeddings with native Jina AI embedding and reranking models, or bring third-party models, all on managed GPUs via Elastic Inference Service (EIS) with no model servers to operate. Or self-host, if you prefer your own.

• The semantic_text field type automatically handles chunking and embedding, along with querying, the simplest path to semantic search in the market. 

Hybrid search and filtered vector search

• Hybrid search is built in, combining full-text and vector retrieval in a single query. Blend the results with reciprocal rank fusion (RRF) or any other blending mechanism you want. Vector search is usually the hardest part of hybrid search to configure well. With Elasticsearch Vector Database, you have it handled, and your whole hybrid stack gets better. 

• With filtered vector search, apply metadata filters as part of vector retrieval itself and not as an afterthought that wrecks recall.

Enterprise on day one

You also get role-based access control (RBAC), audit logging, and the compliance certifications that pure-play vector databases generally lack.

Affordable at scale and predictable

Elasticsearch Vector Database is built to stay affordable as you grow: BBQ and DiskBBQ compression that keeps storage linear and memory low means scaling to hundreds of billions of vectors doesn't blow up your bill. And what you do pay is built from numbers you already know: how much data you store and how much you index, along with how much search capacity you need. Estimate your document count and vector dimensions, plus your query load, and you can work out what you'll pay before you create the project. You can also understand your bill line by line at the end of the month. There are no opaque compute units and no surprise charges for background operations.

How to get started with Elasticsearch Vector Database

Create a serverless vector database project

Create a new serverless Vector Database project in Elastic Cloud. Point your data at the endpoint, and you're ready to index.

Create an index using semantic_text

Vector index mode handles the vector configuration. Using semantic_text means that embeddings and chunking setup are managed for you, as is index setup, on managed GPU inference, with no embedding pipeline to build.

Ingest documents

Index text, and the embeddings are generated for you.

Run a semantic search query

Query the same semantic field you just created:

And you get results back:

Semantic search is just the start. Run fully textual queries or combine both into hybrid queries. You can even craft your own vector queries for full control. Follow the semantic search quickstart in the docs for the full instructions.

What's next for vector search in Elasticsearch

We're already working on the next improvements:

• Better multi-tenant handling: If your data needs to stay separated per tenant, we'll give you a way to do it faster and with less code.

• Automatic index optimization: From "brand new index" to "fully optimized," with as little tinkering as possible.

• Continuous infrastructure improvements: Ongoing tuning of Vector Database's settings and infrastructure so you're always getting the best throughput and fastest responses.

Try Elasticsearch Vector Database on Elastic Cloud Serverless

Go from an empty project to a hybrid, filtered vector query in minutes, with production-grade defaults doing the tuning for you. Build fast, scalable AI apps, not infrastructure.

Start on Elastic Cloud Serverless, or dive into the full documentation and API reference. You can also access the new offering on AWS Marketplace, Google Cloud Marketplace and Microsoft Marketplace.

The changes are about storage and performance, along with the out-of-the-box (OOTB) experience. Up until 9.5.0, Elasticsearch operated as a document-based search engine by default. It could be set up to behave like a columnar system storage-wise, but that wasn’t the OOTB experience. Columnar index modes make a number of fundamental changes that allow Elasticsearch to optimize columnar analytic and search use cases:

• Fields are stored once as doc values only and are no longer indexed by default.

• New multi-value semantics. The original ordering of multiple values per field per document (for example, in arrays) is preserved by default.

• Mappings are always flat, and object and passthrough fields in mappings are always auto-flattened.

How columnar index modes fit into Elasticsearch

Many of the columnar index mode changes originate from time series data streams (TSDSs). As part of making TSDB a competitive metrics solution, we improved doc values format on disk and only store dimensions and metric fields once as doc values. We also improved query performance. TSDB is already columnar today. Essentially, this makes TSDB’s storage mode columnar. The lessons learned from TSDB are now being applied more broadly to Elasticsearch. 

Note that columnar index modes are opt-in and columnar, and document-based indices can coexist in the same cluster. An enterprise search use case can use a document-oriented index mode, while a logging use case can use logsdb_columnar index mode and be fully columnar, all in the same cluster. In fact, there are currently seven index modes, and indices can all use them in the same cluster.

How columnar storage stays fast without an inverted index

Indexed fields, either an inverted index for string-based fields or block k-dimensional (BKD) tree for numeric fields, allow Elasticsearch to query or filter by field very efficiently. However, the cost for this is an additional expensive data structure that uses a lot of disk space and is expensive to build at index time and at merge time. With the columnar index modes, fields are no longer indexed by default, so what did we do for query performance to be still acceptable on fields that were no longer indexed?

One major change was improving doc values scanning performance. This is key and is the cornerstone that any columnar system relies on. Previously, the scanning of doc values was essentially document by document. Lucene’s doc values API only allowed for looking up one value at a time. Historically, this fit the execution model of a search engine. In our own doc value format, we build the capability to allow bulk reading of values for Elasticsearch Query Language (ES|QL) queries. Also over recent minor Lucene releases, Lucene doc values API added support for bulk reading. Without this, fast columnar scanning wouldn’t have been possible.

Secondly, we fully adopted doc value skippers, a hierarchical skiplist over doc values. Contrary to an inverted index or a BKD tree, doc values skippers are lightweight data structures. At its core, a skipper allows queries to skip over a range of documents that don’t match a query. It can do this because it stores information, like min and max values. So, for example, when a range query is executed, an interval of documents can be skipped based on the intermediate result and a doc value skipper’s min and max values. The effectiveness of doc value skippers depends on the order in which documents are laid out on disk. This is why index sorting should be enabled or altered to match the use case.

By significantly improving our columnar scanning and doubling down on doc values skippers, we’re able to avoid indexing fields by default. Note that a field can still be indexed; the index mapping attribute just defaults to false in columnar mode. The exceptions to this rule are text-based fields, which are still indexed by default. This is because text fields provide free text search, which includes text analysis, along with phrase and wildcard matching. This is different from just filtering.

How Elasticsearch handles high and low cardinality fields

When setting up a schema with string fields, an important configuration parameter is often cardinality; that is, whether many unique values or a few unique string values are expected. Many systems have dedicated field or column types that target low and high cardinality string fields.

Fields that have low cardinality are typically stored with a dictionary, containing all unique values. Then, for each row offset, the offset into the dictionary containing the term the row has is stored. This is often called an ordinal. For low cardinality fields, this works well, as storing an ordinal per document takes up much less space. Encoding techniques, like delta encoding, offset encoding, and bitpacking, work well for ordinals to compact the per-document storage to just a few bits. 

However, for high cardinality fields, the dictionary and ordinal approach can work counterintuitively. If a larger percentage of the documents have a unique value, building the dictionary becomes expensive and storage savings diminish. The dictionary then becomes another level of indirection for reading values. This is why most systems in that case store values in a columnar fashion using block-based compression. For example, values of multiple rows are stored in 128KB blocks using a sliding-window dictionary-based compression algorithm (like zstandard or lz4). This, in general, is a simple and effective method to store higher cardinality fields and avoids building and maintaining a dictionary. 

With document-based Elasticsearch, there are two ways to map a string: using either the keyword field mapping or one of the text-based field mappings. The former stores an inverted index and dictionary-based doc values. The latter only stores an inverted index. This is why, typically, a text field mapper is often used in combination with a keyword mapper as a multi-field. Also, keyword field mapper (as the name suggests) is meant for keywords or fields that have a lower cardinality and uses dictionary-based doc values implementation. However, in practice, keyword field mappers are also used for high cardinality.

In columnar mode, every field is only stored once by default. For keyword fields, this means only doc values are stored with no inverted index. Text-based fields now also store doc values and an inverted index by default. Text-based field mappers are different from the keyword field mapper, as these are not automatically used via Elasticsearch’s dynamic mapping logic for columnar indices, and therefore text fields keep storing an inverted index by default.

For both keyword- and text-based fields, we didn’t choose to expose a cardinality mapping attribute. It’s not always possible to know ahead of time whether a field is low or high cardinality. When flushing and merging segments to disk, Elasticsearch sees all values and can determine the cardinality of a field. This is why we’re choosing to automatically determine whether the usage of a dictionary and ordinal-based encoding is beneficial over block-based compression using a simple cardinality threshold. If a field is below this threshold, dictionary and ordinal-based encoding is used; otherwise block-based compression is used. This simplifies configuration of the mappings and makes it possible to automatically optimize storage as data evolves, since some segments may use dictionaries while others may use blocks for the same field. However, this is currently not ready yet and so, as part of 9.5.0, in columnar mode, both keyword- and text-based field mappers store values in doc values in a block-based compressed layout on disk.

The two approaches compare as follows:

Dictionary and ordinal encoding

Block-based compression

Suits

Low cardinality fields

High cardinality fields

What’s stored

A dictionary of unique values, plus one ordinal per document

Values for many documents compressed together in blocks

Compression

Delta encoding, offset encoding, and bitpacking reduce each ordinal to a few bits

Sliding-window dictionary compression, such as zstandard or lz4, typically over 128KB blocks

Read path

Resolve the ordinal, and then look up the value in the dictionary

Decompress the block, and then read the value directly

Cost as cardinality rises

Dictionary grows large, savings shrink, and the extra indirection stays

Stable, with no dictionary to build or maintain

Used in columnar mode tech preview

Not yet

Yes, for both keyword and text fields

Columnar mapping attributes: multi_value, nullability, on_failure

The columnar index modes provide more control over how data is stored as doc values. By default, Elasticsearch is lenient and accepts all non-malformed values (for example, nulls and multiple values per field and document). If documents have fields with multiple values per document or no value, doc values store additional data structures to deal with them and therefore implicitly increase costs. 

With columnar, new mapping attributes provide additional control. Note that these new mapping attributes are currently only available with the columnar index modes but will eventually also be available for all index modes.

Three new mapping attributes are involved:

Attribute

Default

Enforces

On violation

Available

multi_value

true

One value per document per field

Document indexing fails

9.5.0

nullability

true

Field must have a value

Document indexing fails

9.5.0

on_failure

fail

How the above failures are handled

Sets fail or ignore behavior

Next minor release

multi_value: Enforcing single-valued fields

By default, Elasticsearch accepts multiple values per document. To understand the implications of this, we first should take a look at how Elasticsearch (using Lucene’s doc values) stores a dense numeric field where all documents have a single value:

With this layout, all values are stored in blocks. The number of values per block depends on the index mode but is typically 128 values and is always the same within an index. All values in a block are encoded using various encoding techniques, like delta encoding and bit packing, so each block can have a different size, depending on how well the encoding techniques compress the values. This is why a block index is required. 

Lucene has the notion of a docid (internal numbering for a document), which is essentially a row identifier.

1. Queries produce matching docids.

2. In case of a dense field, the block id can be resolved from the docid directly.

3. The offset of a block can be resolved from the block index.

4. Once that has been looked up, the target block gets decoded and all values are available.

5. Finally, from docid, the ordinal within the decoded values array can be resolved, which produces the final value.

Now let’s have a look at how the data layout changes when documents have multiple values per document:

To determine how many values belong to a single docid, an offset lookup is required.

In this case, a docid has one or more offsets. Each offset points to a block index. Values for a single document are adjacent but can stretch over blocks. In general, compaction of values works well in blocks because values are similar. However, multi-value fields can cause the compaction of values to be less efficient if the number of values per field and document is large and values aren’t similar. This and the additional storage of offsets result in multi-value fields typically having a higher storage footprint on disk.

If a field is truly single-valued, you may want to enforce this property. The multi_value mapping attribute makes that possible now. An example is a log level field. Logs typically have one log level (such as debug, info, or error). Enforcing that this field is single-valued in your mappings can help avoid accidentally using more storage than anticipated. A mapping snippet example that disallows the field log.level to have multiple values per document:

Note that even if a field allows multiple values, this doesn’t mean an offset lookup is stored. This only happens when a Lucene segment has at least one document with two or more values. The multi_value mapping attribute exists just for enforcement.

nullability: Requiring every document to have a value

By default, Elasticsearch accepts documents with fields that have no value or null value. Just as  multiple values per document require additional accounting, documents with no value require additional accounting to identify which of them have at least one value.

Doc values store a docid to offset lookup (known as IndexedDISI in Lucene) in case not all documents have a value in a segment. The offset either points directly to the block index for single-valued fields or the offset lookup in case of multi-valued fields. This lookup is compact compared to the value blocks being stored. However, if it were to be created for fields that should have at least one value per document, that would be a waste.

The nullability mapping attribute allows you to control whether documents are allowed to have no value. Just like the multi_value mapping attribute, the nullability attribute exists for enforcement.  Following is a mapping snippet example that requires the log.level field to have a value:

on_failure: What happens when validation fails

What happens if a document has multiple values for a field and if the multi_value mapping attribute is set to false or when a field is mapped with nullability set to false and a document doesn’t have that field? At the moment, indexing such documents will fail with a bad request error.

As part of the next minor release, the on_failure mapping attribute will be available. This allows you to indicate how to handle these validation failures, on a per-mapped field basis. This will support two values:

1. Fail: Fail indexing of the entire document with a client error. This is the current behavior in Elasticsearch 9.5.0.

2. Ignore: Ignore the validation error, mark the field as ignored, and store values for that field in a hidden field so that it can be introspected when requesting the source. 

Trying the columnar index modes

The columnar index modes are still under active development, but we encourage you to give them a test drive. As we prepare the columnar index modes for general availability (GA), we’ll add more performance and efficiency improvements. We believe that by adapting a columnar mindset, many use cases will benefit from being more cost effective or having better performance characteristics.

Columnar mode is Elasticsearch's analytics-optimized columnar storage mode, built for scan-heavy workloads, like log analytics. In this mode, keyword fields don't get an inverted index by default, so term and wildcard queries scan doc values. DocValuesSkippers (zone maps) already trim how much data a scan touches, but these rewrites cut the cost of what's left. 

How Lucene's query rewrite mechanism works

In Lucene, every query has the option to implement a rewrite method that returns another query. This method returns a query with the same semantics but a different implementation. The query engine repeatedly calls the rewrite method until the returned query doesn’t change. This final query is the one that’s actually evaluated. Importantly, the rewrite can see the actual query arguments and specialize the implementation based on these.

For example, in a query looking for documents where a string field contains the value "foo", the rewrite method knows that the term we’re searching for is "foo". In theory, rewrite could replace the general query class with something specific to "foo". For example, the original query class ScanningBinaryDocValuesTermQuery could be replaced with FooQuery. Now this rule probably wouldn't be helpful, but it gives a sense for the level of specialization that’s achievable with rewrite rules.

Rewrite rules and query optimization in database systems

It's worth placing rewrite rules in the larger context of database systems. Lucene and Elasticsearch aren’t the first systems to use transformation rules to optimize queries. Most (or maybe all) database systems use some kind of rule system during query optimization. The most influential rewrite rule system was in IBM's Starburst database. This system's core contribution was extensibility; for example, it was possible to add new data types and storage methods, along with (most importantly to us) optimizer rewrite rules.

Each rule consisted of two parts:

1. A condition function: A predicate determining whether the rule applies to the current query graph.

2. An action function: The transformation that rewrites the query plan into a more optimal form.

A rule engine applied matching rules until a stopping condition was met.

Though Lucene's rewrite method is superficially different from these condition and action functions, it achieves the same goal. It checks whether certain conditions match, and if they do, it applies the rewrite by returning a new query. If conditions don’t match, the rewrite returns this, replacing the query with itself; that is, choosing not to apply the rule.

Why these rules live in Lucene, not the ES|QL query optimizer

Elasticsearch actually contains a separate rewrite rule system within the Elasticsearch Query Language (ES|QL) optimizer. This operates on the high-level structure of a query; for example, doing predicate pushdown to avoid unnecessary computation on documents that will be filtered out. But it’s still useful to have the rule system within Lucene. Since Lucene acts as the storage layer for ES|QL (and classic _search) queries, it’s easier to express rewrites that take advantage of the physical data format in Lucene rather than in a higher-level optimizer.

A query rewrite rule for wildcard queries: Simpler code, no automaton

Wildcard queries support the ? and * operators to match any character once or any character multiple times. These operators can appear any number of times in a wildcard query. As with regexes, to evaluate whether a string matches a wildcard query, we build an automaton from the query string and then use the string bytes to do state transitions through the automaton. This is relatively fast, but if you have to evaluate it for every document, the latency really adds up.

But maybe we don't always have to run an automaton. Consider a query like *foo*. How would you implement this if you were writing a simple query engine to find matching strings in a list of strings? Pretty much every programming language has the tool you want built in: a method that finds a substring within a given string. This function doesn't need a complicated automaton; it probably just consists of a couple of for loops.

Now of course we couldn't use this function to implement an arbitrary wildcard query, but we don't have to. The rule rewrite system isn't for the general form. It's for implementing special cases, and it can see the specific query. It knows that we’re looking for *foo* and realizes that this specific case doesn't require the heavyweight automaton machinery. And it can do the same for any query that starts and ends with a *, with some term in the middle.

The following pseudo-code shows the pattern. At the top, we have the generic WildcardQuery. It has two notable fields: the query string (for example, *foo*) and the automaton built for that query. The matches method checks whether the field value for a given docId is a match by using it to evaluate the state transitions of the automaton. More interestingly, its rewrite method checks whether the query matches our special case. We show this with a regex that checks whether the query string starts with a *, has any non-*characters at least once, and then ends in a *. If so, we return the special case as a ContainsQuery and pass in the inner query string (since it doesn't care about the *s). The ContainsQuery then just does a simple contains check to see whether the term bytes are somewhere within the value bytes.

Benchmarking the wildcard rewrite on ClickBench Q20

The wildcard rewrite is straightforward, but does it actually work? Yes, we can use the ClickBench benchmark, which has several queries of this form. Query 20 (Q20) is FROM hits | WHERE URL LIKE "*google*" | STATS count = COUNT(*). It's exactly the query shape that this rule matches: a string match against the wildcard query *google*. And since the query is just counting, we can see exactly how well this technique works. It turns out to be quite effective. Q20 saw a 1.75x improvement on median latency of hot query times, with no filter cache. All benchmarks in this post were run on an Intel Core i9-13900H.

Adding SIMD to the substring search: 1.75x to 2.3x

But can we do better? Yes, switching to a simple contains check opens up a new possibility. Instead of using the two for loops, we can swap scalar logic for single instruction, multiple data (SIMD) logic. Elasticsearch uses the Panama vector API (see our post on SIMD in Elasticsearch), which lets us implement the contains check in SIMD. This works particularly well for longer strings that can take advantage of the wide SIMD registers; for strings under 24 characters, we still use the scalar approach. With this change, we saw another 1.32x improvement, resulting in a total speedup of 2.3x over the automaton-based approach.

A query rewrite rule for empty strings: Less data, no decompression

One benefit of Lucene-based rules is that they’re low level and can fit to the data format. That’s the case for this rule, which applies to string data. 

How columnar storage encodes string data

In Elasticsearch's standard mode, string values are stored by document; this is a row-major format. But in columnar mode, unsurprisingly, the data is stored in columnar-major format. A column of string data is stored in chunks. Each chunk contains many string values and consists of an array of integer offsets and a (Zstd-compressed) blob of the strings' bytes. For a string at index i, offsets[i] points to the offset in the decompressed byte blob where the string starts. So the length of string i can be computed from offset[i+1]-offsets[i]. (There's a dummy extra offset at the end, so we can easily compute the length of the last string). The following diagram shows how a chunk with the strings ‘Feta’, ‘Asiago’, ‘’, ‘Stilton’, and ‘Brie’ is encoded.

Why a term query has to decompress the chunk

Now that we understand the columnar format, let's get back to query optimization. First, consider a term query for the query foo. We’re looking for documents where a given string field exactly matches the string foo. So how do we implement this on a string column in the above format? The algorithm is straightforward:

The bottleneck is the Zstd decompression step. But there's not much we can do about that; if we want to check the bytes, we have to decompress the chunks. But remember, we aren't trying to optimize the general case, we’re looking for special cases. (In reality, you don't just try to think up special cases. These optimizations came about by first running a useful query, realizing that it could be faster, and then looking for ways to improve it.)

Rewriting the empty string query as a length check

One special case we found that’s worth improving is a query for the term "". Admittedly, it's a silly term, but empty strings are all over the place. Since they're rarely useful, we usually filter them out with a query like term != "". Thankfully, this is a query we can optimize.

Consider the above algorithm for the empty string term. The line if value == term is a bit weird; we’re asking Does this value equal the empty string? We can do that, but there are no bytes to compare, so the check unwinds:

1. We only need to know whether the value has length 0.

2. If we only need the length, we don't need to look up the value in the decompressed chunk.

3. If we never look up a value, we don't need any bytes from the chunk at all.

4. If we need no bytes from the chunk, we don't need to decompress it.

All we need are the lengths, and those live in the offsets array. It's compressed, too, but with cheap integer compression rather than Zstd, which is much faster.

With this realization, we can rewrite empty string term queries. The one new operation we need is docValues.loadLength(docId), which reads directly from the offset array without touching the compressed bytes. After the previous example, this should look familiar. The most interesting part is TermEqualsQuery.rewrite; it finds the empty string special case and replaces the query with the simpler version that only checks the length.

Benchmarking the empty string rewrite: 1.6x faster

Now let's see how this stacks up. There aren't any pure-scan ClickBench queries that use this rule as directly as Q20 does for the previous rule, so we'll make our own. Consider the query: FROM hits | WHERE SearchPhrase != '' | STATS count(*). On this query, we see a 1.6x speedup, which is a great improvement for a fairly uncomplicated change. Better yet, ES|QL can take advantage of loadLength directly. Any time that ES|QL accesses a string's BYTE_LENGTH, without needing the string itself, the request uses this same specialized length loading to avoid unnecessary decompression.

What makes a good query rewrite rule

The two rules covered here follow the same shape: identify that a query is a special case, and then swap it for a cheaper implementation. But they reduce cost in different ways. 

Wildcard rule

Empty string rule

Query shape detected

*term*

field == ""

Replaced with

SIMD substring search

Length check on the offsets array

Cost reduced

Algorithmic work

Data access

Speedup

2.3x

1.6x

The underlying pattern is worth noting: finding a query that leaves performance on the table, finding a special case that can be optimized, and swapping in a cheaper implementation. The hard parts are finding queries that uncover these opportunities for optimization and then identifying the special cases. The actual fix is often relatively straightforward, as both rules here show. Our work on columnar mode has provided many opportunities to run interesting queries and hunt down exactly these kinds of wins.

That's also why extensibility in a rule system is so important. These rules can't be built into a database from the start; they're found through an incremental discovery process. Lucene's rewrite system makes that practical. As columnar mode grows to handle new workloads, rules like these will keep emerging.

To try columnar mode and the optimizations described in this article, use Elastic Cloud Serverless or Elasticsearch 9.5 or later, where columnar mode is available as a technical preview.

How historical metrics were loaded before backfill

Even if loading historical metrics isn’t a very common use case, it’s an important step when teams are adopting TSDB. Two scenarios have been the most prominent: bootstrapping a new time series data stream and migrating data from a different system or data stream to a time series one.

Bootstrapping a new time series data stream

You want to start a new time series data stream with a week of historical data so you have something meaningful to query from the start. With existing tooling, you had to set index.look_back_time to the seven-day maximum in the index template, and all historical data would land in a single backing index. For anything beyond seven days, you needed to create past backing indices manually.

Migrating metrics from another system

You have months worth of metrics stored on a different system and want to move your full dataset to TSDB. You need to load months of metrics history alongside live ingestion. The workaround was to manually create all the necessary past backing indices with the right time_series.start_time and time_series.end_time and to index into it directly using the index name. You then attached it to the data stream via the modify data stream API. It worked, but it required understanding the index time semantics and repeating the steps for each time window, along with coordinating that process around ongoing writes.

We wanted both scenarios to feel as close to normal bulk indexing as possible.

What time series data backfill changes

In 9.5, Elasticsearch can create backing indices covering past time ranges, which extends the eligible write window backward.

The eligible write windowis the range of @timestamp values that a time series data stream accepts for new documents. 

In the past, the eligible write window was determined only by the existing writable backing indices at the moment the request was received by Elasticsearch.

In 9.5, Elasticsearch can expand the eligible write window in the past by creating backing indices. This converts the eligible write window to a sliding window extending from the present back to the first read-only or destructive lifecycle action. Common examples of these actions, which are typically defined within your lifecycle configuration, are downsampling or searchable snapshots. Examples also include retention configurations. 

So, given that loading historical data is enabled in a cluster, the eligible write window of the data stream with the following lifecycle configuration is determined by the downsampling action, because it’s the first action that makes backing indices read-only. So, for this data stream Elasticsearch accepts documents whose @timestamp is no older than three months.

Why loading historical data into TSDB is hard

TSDB consists of data streams optimized for timestamped measurements. It uses a columnar storage layout and enforces immutable dimensions. It also organizes data into time-bound backing indices; each index covers a specific time range and accepts only documents whose @timestamp falls within it.

As time passes, rollover creates new backing indices to cover upcoming ranges. Until this release, there was no corresponding mechanism for the past. Creating indices in the past is tricky because historical data might span over a long period of time and can arrive at Elasticsearch out of order. Consequently, Elasticsearch cannot determine the write timeframe that its backing index should cover. Our solution to this is to use a preconfigured interval and lazily create past backing indices.

How Elasticsearch creates past backing indices

When a document is detected whose timestamp isn't covered by any existing backing index, Elasticsearch determines the time boundaries for the missing indices and creates them. It then adds them to the data stream in a single atomic operation. 

Lazily creating the indices ensures that a single request in the past won’t overwhelm the cluster by requiring the creation of 300 indices all at once. It also doesn’t create indices before there are docs to write into them.

Proactive vs. reactive: How we chose the index creation approach

We explored two ways to detect when a past backing index needs to be created.

The first is proactive. Inspect each incoming document's timestamp before routing, and create any missing past backing indices up front. This keeps the write path clean. By the time a document is routed, the index it needs already exists. It does require the data stream to already exist with at least one time series backing index, since that's what we inspect to determine the eligible write window and the time boundaries of the new index. The downside is that it adds work to every bulk request targeting a time series data stream, even requests that contain no past timestamps and need no backfill at all.

The second is reactive. Let the document fail the normal indexing, intercept that failure, create the missing index, and retry. This avoids any overhead on the common case, since the extra work only happens when a mismatch actually occurs. The tradeoff is more complexity in the failure handling path and a retry on every backfill document.

We ran performance tests on the proactive approach against bulk requests with no past timestamps and found no measurable regression. The overhead of inspecting timestamps turned out to be negligible. That settled it. Proactive creation is simpler and consistent with how index auto-creation already works in Elasticsearch. Plus, it adds no measurable cost to the workloads that don't use backfill.

How Elasticsearch determines past index boundaries

Each new past backing index has three properties to compute: its duration, its start time, and its end time.

Property

How it's set

Constraint

Duration

Defaults to one day, configurable via the cluster setting data_streams.past_tsdb_index_interval

Minimum one hour. If the triggering timestamp falls in a gap up to 1.3 times the configured duration, Elasticsearch collapses it into a single bridging index rather than creating many tiny ones.

Start time

Anchored to the start of the next existing backing index, working backward in multiples of the configured duration

Increased to match the end time of the previous neighboring index, where they would otherwise overlap.

End time

Start time plus the configured duration

Reduced to match the start time of the next index, where they would otherwise overlap.

Handling concurrent writes

In a distributed setup, multiple nodes can receive bulk requests with overlapping past timestamps at the same time. Each node collects the timestamps that aren’t matching any of the existing indices and sends a request to the master node. 

The master node executes a cluster update that sorts them and then, one by one, checks whether the timestamp is covered by an existing or newly created index. Otherwise, it issues a new create index request with the time boundaries calculated as described above. The cluster updates are always sequential and guaranteed to produce valid cluster states, so new indices are guaranteed to not overlap with existing indices.

How lifecycle age works for backfilled indices

Past backing indices hold old data but are new indices. Without an adjustment, lifecycle features would apply downsampling and retention based on when the index was created rather than when the data is from. We account for this by using the index.time_series.end_time as the index.lifecycle.origination_date. As a result, the age of the index as perceived by both data stream lifecycle and index lifecycle management (ILM) is based on the age of its data and not its creation time.

How to use time series data backfill

How to enable time series data backfill

Backfill support ships disabled by default. Enable it at the cluster level:

Bootstrapping with historical metrics

To load historical data into a new time series data stream:

1. Create your index template. 

2. Initialize your data stream. (This is an important step because an existing data stream is a requirement for creating past backing indices.)

3. Start indexing. 

Past backing indices are created automatically as documents with historical timestamps arrive, each covering one day of data by default. No additional configuration is needed.

Data migration into an existing data stream

Migrating data within the eligible write window

For data that falls within the eligible write window of your data stream, point your migration pipeline at the data stream and let Elasticsearch manage the rest.

Migrating data beyond a read-only action

For data older than the write window (for example, you're migrating 18 months of metrics but downsampling kicks in after seven days), you need a separate data stream without read-only lifecycle actions. Retention isn’t an issue since the data would be deleted anyway. The pattern is:

1. Create an index template for the historical data stream, using the same mappings as the original but without a lifecycle:

2. Create the historical data stream. If this step isn’t executed, the first indexing request might fail. During the first indexing request, Elasticsearch can create the data stream but it cannot yet create any past backing indices, so indexing a historical document might fail. Creating the data stream explicitly ensures that all indexing requests will be accepted:

3. Index historical data into the historical data stream while current data continues flowing into the original.

4. When the load is complete, add lifecycle. This is only supported by data stream lifecycle since this feature functions on a data stream level:

5. Query across both data streams with a wildcard pattern (my-metrics*) or a data stream alias.

6. If retention is configured, delete the historical data streams when their data expires. Data stream lifecycle will delete the data but it won't clean up the data stream itself.

As you see, the historical data needs to fit on the target tier as a whole because lifecycle will be enabled after the data is loaded. If you have a large historical import, you might choose to split it into batches. Make sure each batch can fit on the target tier as a whole at the time of indexing, to avoid running your cluster out of disk space. Lifecycle will start processing the batch's indices as soon as it's enabled, but it will need time to process the whole backlog.

Protecting the cluster during large migrations: Downsampling floodgate

When data stream lifecycle runs against a data stream with many indices that all qualify for downsampling, it queues them simultaneously. Downsampling is CPU and I/O intensive; it reads and rewrites all data in an index. Queuing dozens of operations at once can overwhelm the master node with persistent task updates while it coordinates them.

The downsampling floodgate scenario could occur before backfill support (for example, when adding a lifecycle policy to an existing data stream with months of accumulated data). Backfill makes it more likely by design.

In 9.5 and serverless, we added flood protection to data stream lifecycle. It now tracks how many indices per data stream are actively being downsampled. If that count reaches a threshold, data stream lifecycle pauses queuing further operations for that data stream until the count drops. The threshold is configurable via the cluster setting data_streams.lifecycle.downsampling.max_indices_in_progress. Other data streams aren't affected.

Limitations and prerequisites of time series data backfill

• Backfill doesn’t apply to read-only indices. If downsampling or a searchable snapshot transition has already run on a time period, documents for that period are still rejected.

• The feature requires a preexisting time series data stream with at least one time series backing index.

• System data streams are excluded.

• Replicated data streams rely on the leader data stream, so no direct backfilling is possible.

• Scaling remains your responsibility. Loading months of data can trigger significant storage usage, force merge operations, and lifecycle activity in parallel. Check that your cluster has the headroom to manage it before starting.

Conclusion

Prior to the Elasticsearch 9.5 release, loading historical data into TSDB was a manual process. By automating the generation and management of past backing indices, we aim to transform historical data migration to a native capability of your standard ingest pipelines. The inherent complexity of managing time-bound indices remains, but it has transitioned from a user responsibility into an internal Elasticsearch function. Whether you’re bootstrapping a fresh data stream or migrating extensive historical datasets, the platform now handles the heavy lifting, allowing you to focus on analyzing your metrics. We look forward to seeing how these improvements streamline your adoption of TSDB.

This can be a lot of effort, time, and context switching for what should be just a glance. In this blog, we explain how Elasticsearch Query Language’s (ES|QL’s) SPARKLINE works, what it does, and how to get started.

How it works

SPARKLINE is an ES|QL aggregate function with a straightforward signature:

SPARKLINE(aggregation, key, buckets, from, to)

• aggregation: Expression that calculates the y-axis value, including any supported aggregation: COUNT(*), SUM(bytes), AVG(latency), or others.

• key: Date expression from which to derive buckets.

• buckets: Target number of buckets.

• from / to: The time range boundaries. (In Kibana, they bind to the time picker via query parameters.)

Under the hood, SPARKLINE buckets the time range, computes the aggregation per bucket, and packs the results into a single ordered array. Empty buckets are zero-filled so every group shares the same x-axis grid for fast, easy visual comparison.

The function composes naturally with STATS ... BY, so you can combine it with any grouping.

Where sparklines shine

The first place you'll see SPARKLINE in action is Discover's log pattern analysis, starting 9.5. When you run a CATEGORIZE query, Discover constructs the SPARKLINE query under the hood and renders trend lines next to each pattern. You don't write SPARKLINE yourself here; Discover handles it when you use the “identify patterns” option in the ES|QL editor. The result is immediate: You scan dozens of log patterns and instantly see which ones are spiking right now versus which have been steady all day.

Here’s the query under the hood:

Consider a platform team investigating how they can cut logging costs. They point Discover at tens of millions of documents and let CATEGORIZE cluster them into patterns. Two patterns rise to the top: verbose lifecycle messages like "fetching resource..." and "completed resource...", each with several millions of hits. The sparklines next to those rows tell the rest of the story: flat, steady streams running around the clock. Not incident-driven. Not bursty. Just constant noise, silently consuming hundreds of terabytes of storage.

The fix in cases like this is usually straightforward: Adjust log levels, drop the pattern at ingest, or route it to a cheaper tier. The hard part was always finding it. With log pattern analysis and inline sparklines in ES|QL, that discovery takes seconds instead of hours. 

Elastic’s internal site reliability engineering (SRE) teams routinely use log pattern analysis successfully, using every ES|QL enhancement. 

These are just some concrete examples. Break down request counts by region, and see which regions are trending up. Compare latency across container IDs. Monitor queue depths by consumer group. The uses are endless.

Simple by design

We deliberately kept SPARKLINE focused:

• It's an aggregate function, not a new command. It composes with existing STATS ... BY syntax, so there's nothing new to learn structurally.

• It returns data that the consumer can render in context. The function produces an array of values. How those values are rendered, as a mini-chart in Discover, a line in a notebook, or a JSON array in an API response, is up to the consumer.

• It fills empty buckets. Every group gets the same number of values aligned to the same time grid. This is a deliberate choice: Sparklines are most useful when you can compare shapes across rows at a glance, and that requires consistent alignment.

The pattern is always the same: one query, many trend lines, instant visual triage.

What's next

SPARKLINE ships as a technical preview in Elasticsearch 9.5. Future work includes rendering sparklines in more ES|QL surfaces, beyond the initial integration, with the CATEGORIZE context in Kibana. This includes dashboards but also Elastic Observability use cases like the following:

In application performance monitoring (APM) workflows, engineers routinely analyze rate, errors, and duration (RED) metrics to understand service health. The challenge is that aggregate numbers hide dimensional outliers. A service might look healthy overall, but one region, one container, or one newly deployed version could be quietly degrading.

Today, the Elastic APM UI lets you break down metrics by transaction name, but root-cause analysis requires slicing by arbitrary dimensions: availability zone, service version, container ID, cloud region. SPARKLINE can make this practical. Break down error rate by service.version, and instantly see which version's trend line diverges from the pack.

Get started

SPARKLINE is available in Elasticsearch 9.5 as a technical preview. Try it with the ES|QL _query API or in Kibana's Discover. Check the SPARKLINE function reference for the full syntax and supported types.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Why snapshots pin primary shards in stateful Elasticsearch

In traditional stateful Elasticsearch, snapshots lock primary shards to their active nodes, completely preventing relocation. That works fine when cluster topology is stable and nodes stay online between maintenance windows, but stateless Elasticsearch works differently. Index data lives in an external object store, with local disk as a cache, and the cluster scales automatically based on CPU, memory, and data size, both vertically (upsizing nodes) and horizontally (adding nodes).

During vertical scale-up, existing nodes must vacate all shards and shut down before new hardware takes over. Since Elasticsearch version 8.13, shard snapshots can pause during node shutdowns and resume after relocation, so a long-running snapshot doesn't block an infrastructure update.

Horizontal scale-out is a different story: No nodes shut down, so pause logic never triggers. New nodes sit idle while existing nodes finish their snapshots, and as soon as a snapshot is queued, the primary shard is pinned to its node, significantly delaying relocation.

Clusters typically scale out because they're already under heavy load. Blocking shard relocations at that moment limits the cluster's ability to reduce pressure, which shows up as degraded indexing throughput and higher latency. The resource imbalance can also trigger unexpected autoscaling behavior. And even when overall topology stays the same, shard locking disrupts hotspot mitigation and workload distribution. These failures used to surface as server-side warnings: "undesired allocation due to snapshot."

Stateful Elasticsearch

Stateless Elasticsearch

Snapshot reads from

Local shard data on the node holding the primary

The object store, using file locations recorded in the commit

Primary shard during snapshot

Pinned to its node until the snapshot completes

Free to relocate at any time

Effect on horizontal scale-out

New nodes wait for in-flight snapshots before taking shards

New nodes take shards immediately, regardless of snapshot state

How stateless snapshots read directly from the object store

A shard snapshot pins primary shards because it needs to read local shard data. In stateless Elasticsearch, that data already lives in the object store, so reading from local disk is unnecessary. Letting snapshots read directly from the object store removes the requirement to lock primary shards. They can relocate freely, and backup is decoupled from cluster balancing.

Stateless commits include location information for each data file in the object store, so snapshots can read and stream directly to the snapshot repository (a separate object store bucket). In the future, we plan to look at server-side ranged copies, which object stores support natively, to skip the local copy step entirely.

Tracking commits when shards relocate mid-snapshot

A snapshot is bound to a specific commit point that determines which files to back up, and those files must remain accessible for the full duration of the operation. In stateful clusters, this is simple: The snapshotting node and the data node are the same, so the commit is managed locally and held until completion.

In a stateless model, the snapshotting node and the data node can be entirely separate, or they can diverge if a shard relocates mid-snapshot. To handle this, we added a transport action that acquires commits on remote data nodes over the network. The data node tracks which commit belongs to which snapshot and releases it once cluster state signals completion.

There's a wrinkle during relocation. A stationary shard relies on its commit point to preserve files. A relocating shard must release its commit so its local store can close cleanly. To keep files accessible through that transition, a newly recovered primary temporarily preserves all existing data files in the object store until notified of snapshot completion via cluster state. This handles both graceful relocations and ungraceful recovery from node or engine failures.

No more allocation delays and improved cache stats

After stateless snapshots shipped, the server-side "undesired allocation due to snapshot" warnings stopped. The chart below shows the before and after, with the release marked by the red arrow. Hotspot mitigation became more responsive because shard relocations no longer had to wait for backup operations.

Cache use is also improved. Snapshots that bypass local shard data stop competing with indexing for cache space. After the release (also marked in the chart), we observed the following two positive changes in cache metrics:

1. The median cache population throughput, defined as bytes per second for filling the local disk cache from the object store, increased about 50%.

2. Cache misses, where data must be retrieved from the object store to fill local disk cache, have dropped more than 60%. 

What comes after stateless snapshots

Object-store-native architectures are increasingly the standard for cloud-native data systems, and stateless snapshots are a step toward fully exploiting that model across Elasticsearch operations. Backups read from the object store, and shards move freely. Neither process waits on the other. Removing the local shard dependency is a step toward further modularizing the stateless architecture.

How Elasticsearch Shard Balancing Works

Elasticsearch uses a weights-based algorithm to create a Desired Balance, an assignment of shards to data nodes. The Balancer determines the target allocation of shards across a cluster of nodes using four key metrics weighted in a linear algorithm. A total weight is calculated per node, and the shard balancer aims to equalize the total weights across cluster nodes. A final Desired Balance shard allocation is precomputed based on the latest cluster state information, and then the elected master node initiates incremental shard moves to reach the desired shard allocation.

The four metrics are:

• Write Load: the total write threadpool activity per node, using the sum of threadpool indexing activity per data-stream shard.

• Disk Usage: the total disk usage of shards per node, using the sum of disk space used per shard.

• Shard Count: the total number of shards assigned to a node.

• Index Balance (shard anti-affinity): per index, how many shards in the index are assigned to the node.

The total weight of a node is calculated using a linear algorithm that finds the deviation from the node-level cluster average for each individual metric, applies a different weight factor multiplier to each, and then takes the sum of all resultant values. The weight factor multipliers attempt to equalize the relative magnitude of each metric so that metrics with large values do not eclipse metrics with naturally small values. Write load tends to be a small value, related to thread usage, and thus gets multiplied by a relatively larger weight factor of 10; whereas disk usage in bytes is a very large number and therefore gets multiplied by a tiny weight factor of 2e-11.

The following are the cluster settings with default values, representing the different weight factors:

cluster.routing.allocation.balance.shard: 0.45

cluster.routing.allocation.balance.index: 0.55

cluster.routing.allocation.balance.disk_usage: 2e-11

cluster.routing.allocation.balance.write_load: 10.0

The linear algorithm looks something like this:

Shard movements are triggered to ensure that the difference in total node weight across cluster nodes remains below the cluster.routing.allocation.balance.threshold with a default value of 1: whenever the threshold is exceeded, shards are moved from the most heavily weighted nodes to the least heavily weighted nodes until the difference between the most heavily weighted and least heavily weighted node is at or below the threshold. Whenever cluster activity occurs that changes shard allocation (e.g., create/delete index, add/remove node, or the disk usage grows), the Balancer rechecks the weights across nodes and triggers shard rebalancing if the delta between the most and least heavily weighted nodes exceeds the configured threshold. The threshold-based approach attempts to balance the trade-off between keeping the cluster perfectly balanced and minimizing shard movements. Large Elasticsearch deployments that use nodes with greater resources typically benefit from raising the threshold setting: a larger weight delta between nodes reduces shard rebalancing.

Shard movement is also constrained by strict shard assignment rules that prohibit certain node assignments according to cluster and index level settings. Examples include: not assigning copies of the same shard to the same node or host; not allowing further assignment of shards to a node that does not have spare disk space; and excluding node(s) as host for a particular index. More on this below.

How a Balanced Cluster Looks (Based on Weights)

Using the linear algorithm and cluster setting defaults previously described, the following is an example of what the Balancer considers balanced. Notably, it can sometimes allow considerable deviation across nodes in any one particular metric. For simplicity, index balance is not included.

Weight Node1 = 0.45 (5 - 6) + 10 (0.3 - 0.33) + 2e-11 (1e+11 - 8e+10) =   - 0.35

Weight Node2 = 0.45 (7 - 6) + 10 (0.4 - 0.33) + 2e-11 (4e+10 - 8e+10) =   0.35

Weight Node3 = 0.45 (6 - 6) + 10 (0.3 - 0.33) + 2e-11 (1e+11 - 8e+10) =   0.10

Limitations of Weights-Based Shard Allocation

Elasticsearch Serverless deployments are managed by Elastic and could run into many edge cases that, without additional configuration, the weights-based shard allocation handles poorly. In self-managed Elasticsearch deployments, it is possible to work around many of these issues by configuring the cluster to suit the user’s workload. However, Elasticsearch Serverless is configured once and must work across all customer use cases. Issues experienced by some Elasticsearch customers (early adopters of Serverless among them) include:

• Continuous rebalancing background noise in active clusters. This could be because the threshold setting needs tuning or because the cluster is very busy.

• The Balancer’s behavior cannot be tuned in a predictable manner. Adjusting the Balancer settings (individual weight factors) can lead to unpredictable outcomes due to the linear algorithm. For example, decreasing the shard count weight factor relative to the other weight factors can lead to data node OOM events when shard count balancing is deprioritized and too many shards pile up on a single node.

• No explanation of why the Balancer is making shard moves. The linear algorithm is difficult to understand without relevant node metrics.

• The linear algorithm allows a high value in one metric to cancel out a low value in another metric. For example, a node can have a higher than average (across cluster nodes) write load, but counterbalance with a lower than average shard count (or vice versa), and the linear algorithm cancels out the spikes: no shards are moved to address the write load hotspot.

• Index-level hotspots can occur when a disproportionate number of index shards are assigned to the same node, rather than spreading out across nodes, despite the index balance weight in the linear algorithm. Index balance weight can, in some situations, be little compared to the other weight factors. It can also get skewed and counterbalanced by another non-average individual weight in the linear algorithm, as described in a previous bullet.

• No search load balancing.

• Regular indices do not have write load estimate support, leaving some write load hotspots unaddressed. Only data stream indices have write load estimates.

• Write load hotspots can be missed. Write load estimates are only refreshed at rollover time, which can be infrequent in some configurations, causing new load to be ignored for some time. The write load is also the average write load activity over a potentially large window of time between index rollover events, so temporary write load increases can disappear when averaged with inactive write periods.

The above issues persist in some Elasticsearch deployments and require monitoring and workload tuning to manage when they do occur. Shard allocation balancing in Elasticsearch Serverless aims to address these issues and avoid any manual intervention requirements using a new approach that is explained in subsequent sections of this article.

Elasticsearch Serverless Shard Allocation 

Elasticsearch Serverless considers node resources individually: shards are rebalanced away from a node when any resource usage on that node grows to threaten performance, and shard movements to a node are declined when the assignment could threaten that node’s performance.

The Elasticsearch single combined score per node is replaced in Elasticsearch Serverless with independent per-resource decisions:

Elasticsearch Weights-Based Balancing

Elasticsearch Serverless Resource-Aware Deciders

Decision Basis

Single weighted sum across four metrics

Each resource evaluated independently

Metric Interaction

A high value can offset a low one

No offsetting; each decider acts separately

Decision Types

YES / NO

YES / NO / NOT_PREFERRED

Rebalancing Trigger

Weight delta across nodes exceeds threshold

Individually configurable safe limits per resource

Explainability

Can only make an educated guess

Each move traces to a named decider

The Elasticsearch Balancer has three phases, in order of priority, for shard movement decisions. The first phase is to assign unassigned shards. Assignment of unassigned shards is the top priority for data availability reasons. The second phase is to move shards that can no longer remain where they are assigned due to cluster configuration changes. Internally, AllocationDecider implementations enforce cluster settings, like index-level shard allocation filtering, shard allocation awareness, disk usage thresholds, or moving shards off of a node before shutdown. The third phase rebalances shards when the cluster.routing.allocation.balance.threshold is exceeded, using the previously described weights algorithm.

The new Serverless balancing approach adds additional logic to the Balancer’s first and second phases, leveraging the existing AllocationDecider logic, and eliminates the third phase. Previously, each AllocationDecider had simple responses of YES and NO. Now, the decision type of NOT_PREFERRED has been added, along with several new AllocationDecider implementations. An AllocationDecider will return NOT_PREFERRED when it observes that performance might suffer from a shard’s assignment to a particular cluster node. The Serverless Balancer will prefer a node assignment for the shard where all AllocationDecider implementations reply YES.

A NOT_PREFERRED shard allocation may be left uncorrected if all other node assignments return NO or NOT_PREFERRED. Such responses mean that the shard cannot be assigned elsewhere without either violating a cluster/index rule or potentially degrading the performance of another cluster node. Serverless Autoscaling activates before all cluster nodes hotspot: even one unaddressable hotspot leads to a scale-up event. New AllocationDecider implementations have also been added for important finite resources, like available heap memory (further discussion below), using only the original YES and NO decisions: exceeding certain categories of resources can lead to node unavailability.

The individual weight metrics in the Balancer’s linear algorithm have been replaced by resource-aware AllocationDecider implementations, and new AllocationDecider implementations are being built for additional resources: Serverless Search Tier load-balancing improvements are currently in development. Each shard migration will have a clear purpose to address a potential resource usage bottleneck.

Internal stats have shown far fewer shard movements in general, without any noticeable accompanying node performance degradations – one workload showed a 50% reduction in shard movements with the same write throughput. Fewer shard movements has the benefit of: avoiding momentary read/write latencies from warming up local caches; and saving on cloud infrastructure costs moving data between servers.

Serverless IndexBalanceDecider: Avoid Colocation of Index Shards

The IndexBalanceDecider ensures index shard anti-affinity much more strictly than the original weights-based linear algorithm could achieve. Colocation of index shards in excess of the index’s average shards per available node is avoided, except in the case of a strict NO assignment (essentially non-existent right now in Serverless except for shutting down nodes and rolling upgrade incompatible version checks) or NOT_PREFERRED assignment due to temporary node hotspotting.

The IndexBalanceDecider is a very effective means of pre-balancing both write load and search load before user workloads begin to generate load statistics: each index begins life with its shards distributed across as many nodes as possible.

IndexBalanceDecider Results: Even Write Load Distribution Across Cluster Nodes

Write load across data nodes became much more evenly distributed after the IndexBalanceDecider was enabled in the Serverless Production environment. Projects fleet-wide generally show even ingest load (counted in saturated WRITE threadpool threads), combining the release of the IndexBalanceDecider and many other prior improvements:

A reproducible workload demonstrates a clear before and after view of the impact of the new IndexBalanceDecider when an ingest workload was run with and without it enabled:

The IndexBalanceDecider also serves in the Serverless Search Tier to distribute shards of the same index as much as allowed, similarly limited only by the tier’s node count and the number of shards in each index.

Serverless HeapUsageDecider: Assign Shards by Available Heap

The HeapUsageDecider limits shard count on a node based on available heap to hold in-memory shard metadata and run associated write/read operations, removing the dependency on shard count limits per node. The HeapUsageDecider returns a strict YES or NO decision, rather than using the new NOT_PREFERRED decision type, because a data node risks an OOM event if the estimated available heap memory is exceeded.

HeapUsageDecider Results: Reduced Data Node OOMs

Data node OOMs in the serverless index tier decreased significantly as the HeapUsageDecider rolled out to the Serverless production environment.

Index tier OOM errors still occur from time to time in the Serverless Index Tier, though at a much reduced rate, as miscellaneous runaway memory usage edge cases are surfaced. The remaining OOM errors are being progressively resolved as they are identified, through a combination of memory usage improvements in the code, adding component level limits, and updating the internal Elasticsearch Serverless memory model service to more completely account for memory usage.

The HeapUsageDecider is not yet turned on in the Serverless Search Tier, due to the need for additional and different metrics, but that work is in active development.

Serverless WriteLoadDecider: Prevent and Correct Write Load Hotspots

The WriteLoadDecider receives periodically refreshed (every 30 seconds by default) per shard and per node write load stats and uses the data to correct and avoid write load hotspots. The master node retrieves stats directly from each data node’s write threadpool: an Elasticsearch node tracks the total time that its WRITE threadpools is in use, and each individual Elasticsearch shard instance tracks how much time it spent using its node’s WRITE threadpool.

A write load hotspot is identified at the node level. The criteria for a hotspot is the presence of WRITE threadpool queue latency above a configured threshold and sufficiently high, and sustained, WRITE threadpool thread saturation. Once that situation is detected, the Balancer is signaled to select shards to move away from a hotspotting node, until fresh non-hotspotting write load stats are received from the node. The Balancer will do nothing if all nodes are hotspotting at once, expecting the Autoscaler to solve the problem by introducing more, or bigger, data nodes to the cluster.

The WriteLoadDecider uses a heuristic to choose shards to move away from a hotspotting node that aims to minimize ingest disruptions while still effectively reducing a node’s write load. A shard write load threshold is identified on a hotspotting node: the threshold is currently calculated as ½ the ingest load of the hottest shard on that node. Shards that can be moved are then prioritized in the following order:

threshold = ½ * maxWriteLoadShardOnNode

1. Shards with write load in the range [threshold, maxWriteLoadShardOnNode), the shard at or closest to threshold preferred.

2. Shards with write load in the range (threshold, 0], the shard closest to threshold preferred.

3. Shards with write load equal to maxWriteLoadShardOnNode.

4. Shards with zero write load.

The heuristic prefers to avoid disruption to the highest ingest shards and instead chooses middlingly loaded shards. Movement of the hottest shard will cause the most latency disruption; and movement of the coldest shards will be the least effective in resolving a hotspot.

The Balancer limits write load hotspot correction shard moves to one move per hotspotting node per stats refresh period, in order to see the effect of a move in real-time node-level write load, before attempting any further corrections. This was a simple initial design that proved effective. Furthermore, the Balancer will not move a shard whose write load alone is sufficient to meet the node-level hotspot criteria: this would just relocate a hotspot to another data node, not actually resolve the hotspot. The Serverless Autoscaler and Serverless Autosharding components are relied upon to resolve hotspots that reallocation of shards cannot.

The WriteLoadDecider returns NOT_PREFERRED when acceptance of a shard could cause a node to start experiencing WRITE threadpool queue latency and create a hotspot. A shard will still be relocated to a NOT_PREFERRED node, however, and risk some performance degradation, as a better option than, say, risking a data node OOM from keeping a shard on a data node where the HeapUsageDecider returns NO.

WriteLoadDecider Results: Hotspots are Quickly Corrected 

Hotspot stats showed general improvement as the WriteLoadDecider was rolled out to Serverless production, in particular the fleet-wide time to correct a hotspot decreased greatly:

Since these graphs were collected, additional work has been released incrementally to better prevent and correct hotspots, and improvements are still in progress.

Serverless Autoscaling, Autobalancing, and Autosharding

Elasticsearch Serverless relies on both new autobalancing logic and new autoscaling logic. The Serverless Balancer must sufficiently distribute shard resource usage across nodes in order to fully saturate the cluster’s resources. The Serverless Autoscaler will trigger a scale-up event when it receives a report that a certain percentage of the total cluster resources are in use and more resources are needed. The Autoscaler will not scale up the cluster if one node is hotspotting and another node has an excess of available resources because the resources are summed across nodes. Therefore, the Balancer must first do a good job on load distribution, and then the Autoscaler will activate as needed.

Autosharding based on write load is also in progress and coming soon to address shard hotspots. Elasticsearch Serverless projects have a default number of shards per index based on the project type. These defaults generally work, but do not account for all possible workloads. Hotspots can occur when an index has too few shards, as well as too many. Too few index shards leads to the Balancer being unable to further distribute an index’s write load across available data nodes, and then the Autoscaler will not see a problem because the cluster-level resources are not fully consumed. Conversely, indices cannot by default have too many shards, since that could degrade search performance for small indices and potentially strain cluster metadata operations if the total number of shards in a cluster grew too large.

Production Example: 708 TB Data Set, 37 Index Tier Nodes (not counting Search Tier), 4,100 Indices, 30,000 Shards

The following graphs cover a period when the Index Tier, in an Elasticsearch Serverless project, scales up from 10 to 37 indexing nodes and then back down to 10 after a write load spike dissipated.

Graph of the Ingest Load Per Index Node

This graph shows fairly even distribution of load, though a little less even temporarily during scale-up. There are nearly 250 fully saturated write threads at peak load. 

Graph of CPU Saturation Per Index Node

CPU usage remains within safe bounds. Usage is mostly below 60%, except for momentary outliers that reach into the 90% range as write load rises before nodes are added to the cluster.

Graph of WRITE Threadpool Queue Latency Per Node 

When a node’s WRITE threadpool is fully saturated, tasks are placed in the threadpool’s queue. Queuing can happen with few tasks, if active write tasks are long-running, or there may simply be a lot of tasks.

This graph’s time window is zoomed in further than the others. One node reaches 75 seconds of queue latency during the scale-up spike. There are 29 nodes when the queue latency spike occurs at 19h25m, before autoscaling calls for 37 nodes at 19h28m.

Graphs of Total Cluster Ingest per Second, in Documents and MBs

Ingest rate peaks at 190,000 documents / second and 54.40MB / second. The document ingest rate is respectable at 4000-5000 docs/sec per node. The MBs ingest rate, however, is very low in this case: this can happen when indexing operations involve heavy computation. Document ingestion rate can also vary depending on the size of the documents.

What’s Next for Elasticsearch Serverless Balancing

The team is currently working on shard balancing improvements for the Serverless Search Tier, focusing on creating metrics and AllocationDecider implementations for search performance. The team is excited to share these improvements soon!

How dashboard activity logs differ from the Kibana audit log

Dashboard activity logs and the Kibana audit log both write structured logs about user actions, but they answer different questions.

Kibana audit log

Dashboard activity logs

Answers

Who accessed what, and when

Which dashboards are used, and how well they perform

Built for

Security and compliance teams

Kibana admins and dashboard owners

Tracks

Security-relevant events across Kibana

Five dashboard actions: create, update, delete, view, refresh

Enabled by

Its own setting in kibana.yml

user_activity.enabled: true in kibana.yml

What dashboard activity logs capture

The user activity service records structured events every time a user interacts with a dashboard. Each event captures what happened and to which dashboard, in addition to who did it. Five actions are tracked:

Action

Fires when

Includes duration

dashboard_create

A dashboard is created

No

dashboard_update

An edit is saved

No

dashboard_delete

A dashboard is removed

No

dashboard_view

A user opens a dashboard

Yes, time on the dashboard until they navigate away

dashboard_refresh

A user changes filters or time range, or auto-refresh runs

Yes, refresh duration

Very often, a dashboard_view event also triggers a refresh.

Every event carries the same core fields, with two that appear conditionally:

Field

What it holds

Present on

user.name

Name of the user who performed the action

Every event

user.email

Email address of the user

Every event

user.roles

Roles assigned to the user

Every event

object.name

Dashboard name

Every event

object.id

Dashboard ID

Every event

kibana.space

Kibana space the dashboard belongs to

Every event

client.ip

IP address the request came from

Every event

event.action

Which of the five actions occurred

Every event

event.outcome

Whether the action succeeded or failed

Every event

event.duration

Time taken, in nanoseconds

dashboard_view and dashboard_refresh

error.type / error.message

Error class and message when something fails

Events where event.outcome is failure

How Kibana records dashboard activity

Under the hood, Kibana plugins report events from the browser or server through a core client, and valid events are written to a dedicated logger. No data is sent to a third party; because events are standard JSON logs, you control where they go and how they’re ingested.

What you can do with dashboard usage data

Dashboard activity data answers five operational questions that would otherwise require further investigation:

• Clean up unused dashboards. Filter for dashboards with zero dashboard_view events. If nobody's looking at it, archive it. This is critical for customers who are managing thousands of dashboards. 

• Troubleshoot performance. The event.duration field tells you exactly how long each dashboard load or refresh takes. Sort by duration to find your slowest dashboards.

• Edit history. Every create, update, and delete is logged with the user who made the change. You no longer have to wonder who modified a critical dashboard or when it happened.

• Plan capacity. Identify users running heavyweight queries during peak hours. If one user's auto-refresh is hammering the cluster every 10 seconds, you'll see it.

• Monitor errors proactively. Dashboards throwing errors surface immediately through error.type and error.message fields, so you don’t need to wait for users to report them.

How to enable dashboard activity logs in Kibana

Add two lines to your kibana.yml ( the service is disabled by default):

Events will start flowing immediately using a default JSON console appender. You can customize the output appender and filter specific actions using the same logging configuration schema that Kibana already uses:

Ship these logs into an Elasticsearch index (for example, via Filebeat), and you have a fully queryable dataset of dashboard usage.

How to query dashboard activity in Discover

Once your activity logs are indexed, open Discover and point it at your user activity index pattern. You'll immediately see every dashboard interaction as a structured event, and they’re filterable by action type, user, dashboard name, and time range.

From here, you can quickly answer specific questions like the examples below. 

How many times was a dashboard viewed? 

Type your question in natural language in the Discover query editor, and press Cmd+J to automatically generate the Elasticsearch Query Language (ES|QL) query, as shown below:

Which dashboards had zero views in the last 30 days?

Dashboards with no activity simply don't appear in the logs, so you can't filter directly for zero views. Instead, this query works backward, pulling every dashboard created (and not deleted) in the past year and then checking which of those had zero views in the last 30 days.

Which dashboards took longer than 10 seconds to load?

Note that event.duration is recorded in nanoseconds, so the query converts to seconds before filtering:

Which dashboards are throwing errors, and what's failing?

This query shows dashboards with one or more panels throwing errors during dashboard_refresh events, so you can quickly spot recurring issues and prioritize fixes:

Build a dashboard usage overview with AI chat

We’re planning to add out-of-the-box dashboards along with the activity logs, but in the meantime, instead of manually building visualizations, open the AI chat in Kibana and ask it to create a dashboard from your user activity data. 

The generated dashboard gives you at-a-glance visibility into your most-viewed dashboards, heaviest users, slowest-performing panels, and recent errors; that is, exactly the operational view that large deployments need.

Get started with dashboard activity logs

User activity logs are available in Kibana 9.5. Begin by enabling the service. Then ship the logs into an index, and start building the operational visibility that your team has been asking for. For full configuration details and the complete event schema, see the user activity documentation.

Why upgrade to Redux Toolkit v2?

RTK v2 was released in late 2023. That's nearly three years of running on a major version behind in one of the most widely used state management libraries in the JavaScript ecosystem. It reflects how hard this upgrade is in a codebase of Kibana's size. A previous attempt tried the big-bang approach and stalled when the real scope became clearer. 

So what does v2 actually bring? It ships alongside Redux core 5.0, React-Redux 9.0, Reselect 5.0, and Redux Thunk 3.0. React-Redux 9.0 requires React 18 and drops the useSyncExternalStore shim that v8 carried for React 16/17. Since Kibana already runs React 18, upgrading sheds legacy compatibility code and keeps Kibana on the actively maintained Redux majors.

RTK v2 also brings genuinely useful new features, including inline selectors in createSlice and opt-in inline async thunks through a customized buildCreateSlice setup, along with a combineSlices API with slice reducer injection for code splitting. That last one is particularly interesting for Kibana's plugin architecture where lazy-loading is the norm.

How Redux is used across the Kibana monorepo

Before diving into the solution, it's worth understanding just how varied Redux usage is across Kibana. A full audit of the codebase (tracked in #239863) revealed several distinct camps:

Pattern

Plugins and packages

What the migration needs

Redux Toolkit v1

Discover, Lens, Synthetics, Security Solution

Full v1 to v2 migration

Plain Redux v4

Canvas, Maps, Index Management, Cross-Cluster Replication

redux-v4 alias only, no RTK migration

Kea

Enterprise Search (150+ files), Content Connectors

react-redux-v7 alias, no RTK migration

redux-saga

Synthetics, Graph, Uptime

Store setup only, saga is version-independent

typescript-fsa

Security Solution data-table package

Out of scope

Types and single imports

Expressions, Monitoring

Alias swap only

Plugins such as Discover, Lens, Synthetics, and Security Solution use RTK v1 APIs,including createSlice, configureStore, createAsyncThunk, and createSelector. These are the ones that actually need the v1 to v2 migration. But even here, complexity varies wildly. Lens uses stand-alone getDefaultMiddleware (removed in v2) and PreloadedState (also removed). Security Solution is the largest consumer at 300+ files, mixing modern RTK with legacy plain Redux patterns.

Canvas, Maps, Index Management, Cross-Cluster Replication, and several others still use plain Redux v4via createStore, combineReducers, applyMiddleware, and connect, which are classic patterns from the pre-RTK era. These don't need RTK migration at all since they're not using it  in the first place, but they do need the redux-v4 alias since the default redux package is now v5.

Enterprise Search and Content Connectors use kea, a Redux abstraction layer with its own logic builders (kea(), useValues, useActions). There are more than 150 files in Enterprise Search alone. RTK migration isn't applicable here, since kea is its own world. But it does depend on react-redux v7 under the hood, which is where the bundler tricks come in.

Synthetics, Graph, and Uptime use redux-saga for side effects. Saga integration is actually independent of the RTK version, but these plugins need their store setup migrated.

The Security Solution data-table package uses typescript-fsa and typescript-fsa-reducers instead of RTK entirely, with its reducer embedded into Security Solution's main store, and isn’t part of the RTK migration at all.

The Expressions plugin only imports shallowEqual from react-redux, and Monitoring only imports types. These just need an alias swap.

Asking every team to migrate simultaneously was a nonstarter. The breaking changes in RTK v2 include stricter type checking and removed APIs, like enableES5() from immer, getDefaultMiddleware and PreloadedState gone entirely, AnyAction replaced by UnknownAction, and behavioral changes in how middleware is configured.

Running Redux Toolkit v1 and v2 side by side

The solution was to flip the typical migration pattern on its head. Instead of keeping the default imports on v1 and introducing v2 under aliases, the default package names (for example, @reduxjs/toolkit, react-redux, and redux) now point to v2. The old versions live under versioned aliases:

• redux-toolkit-v1

• react-redux-v7

• redux-v4

• immer-v9

• reselect-v4

• redux-thunk-v2

This is npm's alias syntax. "react-redux-v7": "npm:react-redux@7.2.8" installs the old version under a different name. Both versions coexist in node_modules without conflicts.

The insight here is that all existing code in this pull request (PR) was moved to v1 aliases. Every import { useSelector } from 'react-redux' became import { useSelector } from 'react-redux-v7'. That's ~1,100 files touched, but the vast majority (~1,000) are mechanical one-liner import swaps. When a team is ready to migrate to v2, they switch back to the default import names. Once all v1 aliases disappear from the codebase, the old packages can be removed entirely.

This avoids the alternative, where v2 imports would end up under nonstandard names permanently, leaving nonstandard imports in the codebase for the long term.

Serving both versions through the bundler

Getting two versions of the same library to coexist at runtime is where things got interesting. Kibana uses kbn-ui-shared-deps-npm to bundle common dependencies as shared webpack externals. This needed to serve both the new v2 packages and the v1 aliases so that both are available at runtime.

Pinning @elastic/charts with yarn resolutions

Then there's @elastic/charts. It depends on RTK v1 internally and can't just be upgraded independently since it's an upstream package. Yarn resolutions pin its nested dependencies to v1 versions:

A NormalModuleReplacementPlugin in the shared deps webpack config detects when an import of immer, @reduxjs/toolkit, redux, react-redux, or reselect originates from within @elastic/charts and redirects resolution to the nested v1 copies. This ensures that @elastic/charts resolves to its compatible v1 dependency set.

Keeping Kea on React Redux v7 with webpack externals

The kea library was another fun case. It declares react-redux as a peer dependency (>= 7), so without special handling its imports resolve to Kibana's default v9 package. The migration keeps Kea consumers on react-redux-v7, so Kea must use that same React context. The fix uses function-based webpack/rspack externals that skip externalizing react-redux when the import comes from node_modules/kea, combined with a NormalModuleReplacementPlugin that rewrites it to react-redux-v7. This ensures that kea uses the v7 React context that matches the <Provider> wrapping its consumers.

Both the webpack (kbn-optimizer) and the rspack (kbn-rspack-optimizer) configs needed these changes, with a shared isKeaReactReduxImport helper extracted to keep the logic consistent.

Using an ESLint rule to prevent cross-version imports

With two versions available, accidental cross-version imports are the biggest risk. A new @kbn/imports/no_redux_toolkit_v2_imports ESLint rule catches any import of the v2 default packages (such as @reduxjs/toolkit, react-redux, or redux, among others) in code that hasn't been migrated yet. It even auto-fixes them to the v1 aliases for file imports and Jest mock paths.

The rule is scoped via an override in .eslintrc.js to the ~36 plugin and package paths currently using v1. When a team migrates, they simply remove their path from the override list. This clean, self-service approach requires no coordination.

Why mixing React Redux v7 and v9 breaks the context

This is worth calling out because it's an easy failure mode to miss during an upgrade. react-redux v9 and v7 create separate React contexts. If a component tree has a v9 <Provider> at the top but a child component calls useSelector from v7 (or vice versa), React-Redux cannot find the matching context. In development, it throws an error explaining that the component must be wrapped in a matching <Provider>; in production, the missing context causes a runtime error when the hook accesses the store.

This means that each plugin needs to be explicitly pinned to one version. Shared packages that use react-redux can only be consumed by code on the same version, since mixing isn't possible. This is a constraint that makes the migration inherently per plugin rather than per file.

Migration batches: What can move independently

The dual-version setup gives every team a clear path forward, and the dependency graph analysis from the tracking issue identified natural migration batches:

• Batch 1: Independent, self-contained stores. Packages like kbn-coloring, transform, timelines, and expandable-flyout have fully internal Redux stores with no types leaking through their public APIs. These can be migrated independently by their owning teams, with minimal risk.

• Batch 2: Coupled packages. Some packages share RTK types across boundaries and must migrate together. The machine learning (ML)/artificial intelligence for IT operations (AIOps) chain is one example: @kbn/ml-response-stream exports a streamSlice (a createSlice return value) that @kbn/aiops-log-rate-analysis embeds directly into its configureStore. Migrating one without the other causes type mismatches between v1 and v2 slice types. Similar coupling exists across the Lens ecosystem. The Lens plugin depends on @kbn/coloring (which has its own RTK store), @kbn/lens-embeddable-utils, and @kbn/lens-common, while itself being consumed by 40+ packages and plugins across chart expressions, visualizations, Maps, Canvas, and observability plugins. Whether Redux types leak through a package's public API determines if it can be migrated independently or needs coordination. kbn-coloring's store is internal to its React components so it's safe to migrate alone, but other coupling points need careful analysis.

• Batch 3+: The big ones. Discover, Security Solution, and Lens each have their own migration timelines. Security Solution's 300+ files and mix of RTK with plain Redux v4 and typescript-fsa make it the largest effort, but the different patterns can be addressed independently. Lens has the trickiest v2 breaking changes around middleware configuration; stand-alone getDefaultMiddleware and PreloadedState are both removed in v2, and it has four custom middleware files with complex typing.

Beyond the batched migrations:

• Deprecated features can stay on v1 aliases. When the feature is removed, the v1 imports disappear through code deletion, without any migration work.

• Plain Redux v4 plugins (Canvas, Maps, and others) are entirely out of scope for RTK migration. They'd benefit from modernization, but that's a separate initiative.

• Kea plugins need react-redux-v7 to react-redux alias updates eventually, but no RTK migration. The longer-term question (whether to keep Kea or migrate to RTK v2) is a separate decision.

• The dual-version approach adds measurable bundle overhead during the transition. It’s a trade-off but is acceptable for the migration period.

Lessons for other large monorepo upgrades

The ESLint rule turned out to be the linchpin. Without automated enforcement, aliased imports would drift back to default names within weeks. With it, the migration state is visible in the paths listed in the override. As of the initial PR, zero files import from @reduxjs/toolkit v2. Every RTK usage goes through the redux-toolkit-v1 alias. That's the starting line.

The preparation work also reached beyond import paths. Jest mocks referencing react-redux needed updating to react-redux-v7, as did Storybook previews, test helpers, and ambient type declarations. Multiple rounds of node scripts/eslint_all_files --no-cache --fix caught the mechanical cases; the remaining cases needed manual fixes.

If you're facing a similar major dependency upgrade in a large monorepo, the pattern of giving the new version the default name and the old version an explicit alias is worth considering. New code naturally uses the current version, while older usage stays visible and trackable until it reaches zero.

Managing context depends on good retrieval. Rather than have agents rediscover the same content for every question, burning tokens by retracing similar steps over and over again, Elastic’s agentic AI capabilities enable us to precompute these details and store them in a structured, searchable form, and they let agents load that context directly. We call this precomputed unit of context a Knowledge Indicator.

The default agentic retrieval augmented generation (RAG) pattern does the opposite. It retrieves whole documents and dumps them into the model's context at query time, paying for that retrieval in tokens and latency on every single question. Precomputing the answer as a KI moves that cost out of the hot path and does it once.

How it works: AI Index, Kibana Workflows, and the query-ki skill

Building context through AI indices has three main parts: the AI Index (a special Elasticsearch index where KIs live), Kibana Workflows to create your KIs, and a query-ki skill to help agents directly query KIs using ES|QL: 

This blog post is similar to Part 1 in that we’re using the same core building blocks. But in this post, we’re demonstrating a very different use case. Instead of precomputing index metadata, we’re distilling specific facts from our indexed documents that may be used to directly answer agents’ questions without subsequent searches. We've also provided a notebook, if you'd like to create the same KIs yourself, end to end, as you go through these examples. 

Prerequisites: Elasticsearch Serverless and an LLM API key

This tutorial assumes you have:

1. An Elasticsearch Serverless project. You can sign up for a trial if you don't have one.

2. An API key to access your Elasticsearch project.

3. An OpenAI-compatible large language model (LLM) API key, to access AI indices via Deep Agents scripts.

Load the BrowseComp-Plus sample corpus into Elasticsearch

First, we’ll need some sources. Sources can be data that already exists in your Elasticsearch indices or external data accessed via connectors or ES|QL data sources.  For this blog, we’ll create an index, browsecomp-plus, to hold our example data, with the following mappings:

and populate it with a small sample of BrowseComp-Plus data via the _bulk API. You can use the supporting notebook to load a sample of this data in your project. 

Create the AI Index that stores your KIs

Just like in Part 1, the first step is to create an AI Index:

This is preconfigured with the same required mappings as we listed out in Part 1. We perform hybrid search here using semantic_text out of the box.

How agents retrieve KIs using ES|QL

A KI is a document in the AI Index. What makes KIs useful is retrieval, or querying the AI Index to find the right content. This query is packaged within a small, portable skill that’s harness-agnostic and can be run in any agent harness. 

Here’s a sample query-ki skill:

Save this asskills/query-ki/SKILL.md.

Here’s what this skill is doing: 

• We’re defining corpus_entry as our KI use case.

• We’re performing a hybrid ES|QL search on our AI indices, filtering by the appropriate type, using reciprocal rank fusion (RRF) as the default method to fuse results.

• The KI results will directly ground the agent’s answer when determining what facts are relevant to the users’ query.

When we say that AI indices and KIs are harness-agnostic, it’s because the skill is just instructions plus a query. It will work in Elastic Agent Builder, a Kibana workflow agent, Claude Code, or any other harness. We’ll be using Deep Agents for examples of how to query it outside the Kibana ecosystem. Since an AI Index is, at its core, an Elasticsearch index, you can also explore your data directly. 

Precompute facts as KIs for agentic RAG

In this example, we extract actual facts so agents can retrieve an answer without consuming a full document. We generate one fact-based KI per selected document, though the actual number and structure of KIs you generate are completely customizable.

We'll use a sample of the BrowseComp-Plus corpus, indexed into a browsecomp-plus index, with docid, url, title, and text fields.

Baseline: Retrieving whole documents with RRF

As a baseline, here's a simple RRF query:

This drops several hundred words of raw body text into the model's context. It may work, but it's expensive, and the cost compounds with every miss.

Build the Kibana workflow

The workflow below reads a batch of documents with a single ES|QL query and writes one fact-level KI per document into the AI Index. Each iteration runs two steps: generate_ki distills a raw document into a structured KI, and sink_ki writes it to the AI Index keyed on docid so reruns are idempotent.

Copy and paste the following YAML into the Elastic Workflows editor:

Here’s what this workflow is doing: 

• query_corpus runs an ES|QL query against the browsecomp-plus index, applying some rules, like dropping documents with empty bodies and trimming each body to 12,000 chars so the agent prompt stays inside the context window.

• Note: In this example, we’re cherry-picking some concrete KI IDs, because generating KIs for every document in the index would take a long time, and we want this exercise to be short for those following along.

• loop_corpus_docs iterates over every returned document, running the following two steps per document: 

• generate_ki reads the document and calls an LLM to emit a strictly grounded, structured KI.

• sink_ki bulk-writes each KI into the AI Index (ai-index-idx-my-corpus) as a KI of type corpus_entry. It forces _id to be the same as the document’s docid so rerunning the workflow is idempotent.

To summarize, this workflow turns each raw corpus document into a compact, searchable metadata record that agents can find and judge without reading the full source into the context window.

This workflow is used for example purposes, and the same foreach caveat as in Part 1 applies. For scale, use workflow.executeAsync or native parallel support. The cheat sheet is useful for optimizing Workflows. There could also be cost and efficiency gains in production by using ai.prompt or by choosing different models with which to create KIs. 

Inspect the KIs in your AI Index

Once the workflow runs, you can query the AI Index to browse what was written:

Here’s an example of what one of the KI documents looks like: 

Query KIs from LangChain Deep Agents

We’ll use LangChain Deep Agents with an OpenAI-compatible key to show that AI indices and KIs will work with any agent harness, inside and outside of Kibana’s Agent Builder ecosystem. 

First, let’s create facts_baseline_agent.py to measure our baseline before applying KIs: 

When I ran this baseline query, What was the actress who played Torvi from Vikings also known for?, it output the following:

(Note: Deep Agents automatically adds the read_file tool to handle paginated tool results, which is why it shows up in the output.) 

Next, let’s create an agent that knows how to use our query-ki skill, facts_ki_agent.py: 

This agent will query the KI indices to generate the answer, rather than pulling every document into context. 

When I ran these results against the same query, here was the output: 

How much can precomputing facts reduce agent token usage?

Both agents had similar conclusions, but they took far different paths to get there: 

The same question and the same grounded answer result in 93% fewer tokens and two tool calls instead of eight, when answering from KIs.

Baseline (No AI Index)

With AI Index

Total tool calls

8

2

read_file calls

2

1

esql_query calls

6, all against the browsecomp-plus index

1, from ai-index-idx-*

Tokens consumed

386,187

27,625

Latency

44.86s

15.22s

Answer

Grounded, correct

Grounded, correct

Exact tool call counts, latency, and answers will vary between runs and using different agents. 

Both agents produced solid, grounded answers. The difference is cost. Querying KIs from the AI Index cut token use by 93% and cut latency by roughly two thirds. Here’s how both paths went, side by side:

That was in-depth, but it shows what AI indices and Workflows do together: the same answer, at a fraction of the tokens.

Build precomputed context in Elasticsearch Serverless

This walkthrough shows how to generate more sophisticated KIs based on documented facts and query them for knowledge retrieval use cases using Elasticsearch primitives. 

Managing context is critical in agentic search systems. And at its core, context is a retrieval problem. AI indices help you manage context within the Elastic Stack. Try it out in Serverless, and let us know what you think in our Discuss forums or the #stack-kibana channel in our Community Slack.

We’d also love to hear from you about what use cases you’d like to solve using AI indices.

In Elasticsearch 9.5, Elasticsearch Query Language (ES|QL) queries no longer fail when a field isn't in the mapping. The new unmapped_fields setting lets queries load values from _source or fill with nulls, so queries keep working even when a backing index changes and a field goes missing, and can use unmapped data without reindexing. Here’s how we built that: the design choices and the edge cases (including a class of fields we nicknamed PUNKs), along with the testing strategies that gave us the confidence to ship it in general availability (GA).

Why ES|QL queries fail when a field is unmapped

You built a visualization using an ES|QL query. You refined it, and the query grew. You’re at 15 chained commands and counting, but it does just the right thing. It works, and your dashboard is useful.

Your query uses an index from a remote cluster, say my-remote:logs-foo. But actually, logs-foo is an alias, and at some point, the remote cluster makes it point to a different backing index. The new index is missing a field that’s used in your query, and your query and visualization break.

Or maybe you have an already fairly large index, and while building ES|QL queries on top of it, you realize that you’d like to use a field in the indexed documents that unfortunately never made it into the index mapping. You could reindex the data, but that would take hours.

ES|QL’s unmapped_fields setting is meant to deal with these types of situations.

If your query looks like this:

and some_field is unmapped, ES|QL’s default behavior is to fail with a verification exception.

You can use the unmapped_fields setting to instead either fill some_field with nulls or read it from the document’s _source, like so:

How ES|QL resolves queries with field caps

Before we jump into the inner workings of unmapped_fields, we have to look into how ES|QL resolves queries regularly. Let’s consider the above query:

We said that if some_field isn’t in the mapping for index, ES|QL will reject the query. How does it make that decision?

How field caps tells ES|QL which fields exist

In a typical schema-on-write fashion, Elasticsearch clusters maintain mappings with their respective indices. As a first step, ES|QL makes an internal request to the field caps endpoint to determine which fields the index has. It then passes the query, together with the field caps response, to the query planner, which consists essentially of the query analyzer (unrelated to analyzers of text fields) and query optimizer. The analyzer makes sense of raw names, like some_field, and notices that they correspond to index fields (or not). If all went well, the query is then passed on to the optimizer, which rewrites the query for efficiency, before it’s handed to the compute engine for execution.

How the analyzer resolves field names in the query plan

Let’s zoom in to the analyzer. The parsed query is represented in a tree structure, and the analyzer partially rewrites it, one command at a time, until it either has resolved all references or not.

For illustration, let’s use a somewhat more complex query and see how the analyzer would resolve it:

The parsed tree is actually a chain here, and it looks something like this:

The analyzer then moves up through the query tree to try and resolve the field names used in every command.

This is a simplified version of how we represent parse trees in tests and when debugging. The bottom of the chain corresponds to the FROM command and contains a list of all mapped fields that we know about, obtained from the field caps endpoint. (The {f} suffix marks an actually mapped field for better distinction later.)

The two EVAL nodes on top of it correspond to the remaining commands, and their fields are still unresolved, expressed by the question mark ? in front of the name. At this point, the analyzer still has to check whether they correspond to existing index fields.

For the EVAL that defines uppercased_mapped, it can see that the previous command outputs mapped_field, so the unresolved ?mapped_field marker can be replaced by a real field reference:

Next, it encounters the topmost EVAL, which defines uppercased_unmapped. The previous tree nodes produce only two fields: [mapped_field, uppercased_mapped]. The reference ?unmapped_field thus has to remain unresolved. We bail here and emit the verification exception to the user.

How unmapped_fields LOAD and NULLIFY work

Adding unmapped fields to the query plan

When using unmapped_fields=”NULLIFY” or ”LOAD”, we do something else; we act as if the field was actually in the index. The analyzer adds unmapped_field to the From node and marks it as unmapped to signal to the compute engine that this has to be read from _source or filled with nulls. Let’s express this with a {u} (for unmapped):

After amending the From, the analyzer can continue trying to resolve the topmost Eval node. It sees that the upstream nodes produce the fields [mapped_field, unmapped_field, uppercased_mapped] and thus unmapped_field can be correctly resolved:

The query plan is now fully resolved and can be passed down the regular optimization-execution pipeline. Other than the actual value extraction mechanism, everything stays the same. Schematically, the workflow looks like this:

Example: enabling unmapped fields with the SET directive

To give an example, let’s fire up a cluster and create an index with non-dynamic mappings.

We can run the example query, above:

This should result in the error message:

Unknown column [unmapped_field], did you mean [mapped_field]?

To make things work, we can prepend SET unmapped_fields=”...”; with LOAD or NULLIFY:

Inspecting the analyzer's rewrite steps

If you want to see what the query analyzer is doing to the parse tree, you can log the query rewrite steps, like so:

This will log a line containing Rule rules.ResolveUnmapped applied with change… You’ll see that unmapped_field is added to the bottom of the parse tree as described above.

Why we have to infer the schema

Of course, this isn’t the only possible method to deal with unmapped fields. Here are some alternatives:

1. We could also scan or probe the documents in index to determine that their _source actually has the unmapped_field.

2. We could disable verifications in the analyzer and make the compute engine blindly pass unmapped fields through individual computation steps.

The first alternative front-loads more work to understand the actual schema of an index and thus generally increases latency. It doesn’t scale to large, highly distributed datasets. The second alternative isn’t viable since it means a large-scale change to how ES|QL’s compute engine is built, because it passes around streams of data with fixed columns from one operator to another.

In contrast, the approach we chose is neatly compatible with ES|QL’s existing optimization pipeline.

The trade-off is that the analyzer has to correctly infer a schema based on the actual index mappings (obtained from the field caps endpoint) and additional fields used inside the query. 

This isn’t always straightforward. There were two main challenges:

1. There are many different query shapes and commands that can be used. The mechanism needs to detect unmapped fields, update the proper FROM command, and pass the new field through the halfway resolved plan correctly in all cases.

2. There are many different mappings we have to deal with, and we specifically need to make our feature work correctly when mappings change over time on top of that.

In the following, we’ll focus on LOAD, although some problems (generally many fewer) also apply to NULLIFY.

Which index to load unmapped fields from for LOOKUP JOIN and FORK

To briefly illustrate the first problem, here are some choices we needed to make:

• Which index do we load from when using lookup joins? This one?

  The unmapped_field cannot be attributed to both indices. We chose index since this is where we expect mappings to change more often than in lookup indices.

• Similarly, how do we deal with subqueries and views or the FORK command? In the following:

  one fork branch triggers loading of an unmapped field. Is it also present in the other fork branch? (Yes, it should be, but it’s not obvious and is specifically not true if the two FORKs are replaced by independent subqueries.)

Two principles to keep queries working

The second problem, diversity of mappings and their evolution over time, is a far bigger driver for complexity. We strove for two basic usability principles:

• Queries that work in the default mode should generally still work when using unmapped_fields=”NULLIFY” and ”LOAD”.

• Queries that work when all fields are mapped should generally still work with NULLIFY and LOAD when a field becomes unmapped and vice versa.

The type of unmapped fields and inadvertent type conflicts

Let’s talk about data types to see where this leads to complexity. First, when using unmapped_fields=”LOAD”, we need to assume a data type for unmapped fields. We chose KEYWORD, which allows us to avoid type conflicts when reading from _source. One document can contain ”unmapped_field”: “foo”, and another can contain ”unmapped_field”: 123.4. It’s fine because we treat both as strings.

However, this is a violation of the second principle when a non-KEYWORD field happens to go unmapped. Consider this query:

If some_field becomes unmapped, we’ll have to assume that the KEYWORD type and the query will fail with a type conflict.

Type conflicts aren’t new and can be dealt with by using explicit casts in the query, like so:

It would be great if ES|QL just inferred a useful type to cast to, but this is something for the future.

Type conflicts with partially unmapped fields, or: making PUNKs well behaved

In addition to fully unmapped fields, partially unmapped fields are everywhere and should also work with LOAD. Let’s look at a query that uses multiple indices.

Let’s say that there are indices index and index_without_some_field, containing just the following documents.

Now let’s consider the query:

and assume that some_field is unmapped in index_without_some_field. This will return:

because ES|QL doesn’t load unmapped fields per default.

Of course, when setting unmapped_fields=”LOAD”, we want to load from _source for index_without_some_field:

As with fully unmapped fields, the case is simple when some_field is mapped as KEYWORD in index. When loading from _source for index_without_some_field,  we treat the field as KEYWORD as well, so there’s no conflict.

What makes a field a PUNK

The case is less clear when some_fieldis partially unmapped and the mapped leg is of a type other than KEYWORD. Such fields caused a lot of trouble until we found the best solution, which makes their acronym quite fitting: partially unmapped non-keyword fields, or PUNKs.

Unfortunately, PUNKs are far from being esoteric. For instance, it’s very natural to filter on a PUNK:

If some_field is mapped as INTEGER in index, the type conflict looks like this:

• Mapped as an INTEGER in index.

• Unmapped in index_without_some_field and thus treated as KEYWORD.

This can again be resolved manually by providing an explicit cast:

But this is far from acceptable. Even queries that work fine without NULLIFY and LOAD typically have some PUNKs; the unmapped leg is simply treated as null then. Both guiding principles are violated if LOAD requires an explicit cast here.

Casting implicitly to the mapped type

The solution is to introduce an implicit cast to the mapped type. In this case, we know that some_field is an INTEGER in index, and thus we treat it essentially as if the user wrote:

This means that queries that work without LOAD keep working. (ES|QL may even give you more data because we load the unmapped leg of PUNKs from _source.) Queries that used to work when a field is fully mapped also keep working when it goes unmapped in some (but not all) of its indices without having to alter the query in any way.

Behavior

Default

NULLIFY

LOAD

Unmapped field in query

Query fails

Query runs

Query runs

Values returned

None

null

Read from _source

Assumed type

n/a

NULL

KEYWORD

Partially unmapped field (PUNK)

Unmapped leg is null

Unmapped leg is null

Cast to the mapped type

Pushdown optimization

Full

Full

Per-node where fully mapped

Don't throw it all away: Keeping ES|QL query optimization with unmapped fields

There's one more thing to get right; that is, to make sure that optimizations still work correctly with LOAD. Consider the previous query:

ES|QL’s optimizer aggressively pushes down such WHERE filters and turns them into Lucene queries, so the compute engine doesn’t perform unnecessary work.

For this query, evaluating the filter in the compute engine would require fetching each and every document from the index; meaning, a full scan, very slow. If some_field was mapped as an INTEGER in both indices, we would instead perform a Lucene query, which looks like this:

The compute engine then doesn’t have to load each document separately and check whether it matches the filter. Documents with some_field <= 10 are never fetched from the Lucene index, which is very efficient at this kind of filtering. Nice.

Why filter pushdown is unsafe for unmapped fields

If some_field is unmapped in index_without_some_field, however, it’s wrong to narrow the documents down using the same Lucene query, as Lucene interprets an unmapped some_field as null and thus no documents from index_without_some_field will ever match. This edge case is easy to miss, and it doesn’t help that there are several flavors of similar pushdowns. For instance, in the query:

the compute engine pushes even the counting to Lucene. Again, this is only correct if some_field is fully mapped.

This means that such optimizations can't apply to unmapped fields. It would be disappointing if a query used hundreds of indices and only one of them happened to not map some_field, causing the whole query to run unoptimized.

How the local optimizer recovers the fast path

Luckily, this problem has a solution, too. ES|QL actually has multiple optimizer runs:

1. First, a preliminary optimizer run on the node handling the _query request.

2. Then, a second, local optimizer run on every node we fan out to because we need to fetch documents from its shards.

The workflow after the initial optimization looks more like this:

If the current node happens to map some_field in all shards, the local optimizer detects this situation and treats some_field like any other fully mapped field, including performing Lucene queries to greatly narrow down the dataset to be processed. In fact, data nodes process LIMIT queries like:

in batches of shards (to avoid loading too much data too eagerly), which includes a full local optimizer run per batch. This makes it even more likely to encounter batches where some_field is fully mapped, allowing ES|QL to run a fast Lucene query.

Is it working now? Testing unmapped_fields across every ES|QL query shape

As we have seen from the optimizer issues above, problems can hide in plain sight, even for very simple queries. Because unmapped_fields=”LOAD” can affect each and every kind of query, the surface area for bugs is essentially all of ES|QL.

Accordingly, getting good test coverage was tricky and challenged us to refine our testing strategies.

Reusing spec tests with unmapped_fields

Conveniently, ES|QL has an extensive corpus of test queries, together with expected result sets; we call them spec tests because they’re written using a simple text specification language, which looks roughly like this:

This lets us create new tests out of the existing ones by introducing slight variations. For instance, any existing test that runs without SET unmapped_fields=”...” should produce the exact same results when run with SET unmapped_fields=”NULLIFY”.

It also helped find major issues early in the development process, especially for NULLIFY. The LOAD setting changes the meaning of queries much more dramatically, limiting the usefulness of this approach. However, ES|QL also uses what we call generative testing; that is, we string together random commands, run the query, and then check whether the server reports a bug. This approach cannot confirm the correctness of results, but it still helped greatly with finding query types that didn’t work properly and resulted in some kind of error. (Property-based tests would be a refinement in the future by running the queries against a reference implementation. This way, correctness of results can also be checked.)

Testing type conflicts across different mappings

In the end, one of the most important testing dimensions was using different indices with various mappings in the same query. (Recall how, above, we had to deal with type conflicts to come up with a solid approach for PUNKs? It doesn’t end there; all kinds of type conflicts are more complex with LOAD.) Since we couldn’t automatically generate correct expected results, ES|QL’s test suite had to grow by adding more than 10,000 lines of CSV spec tests. Fortunately, adding such tests is a well-suited task for an AI agent, which has cut down the effort dramatically. (Of course, the test results were still reviewed by humans.)

All testing strategies together provided us with good confidence for the GA release of unmapped_fields with Elasticsearch 9.5.

What you'll discover

In this post, you'll learn how to:

• Use Elastic APM's built-in views to explore search latency and throughput and to explore errors.

• Define Service Level Objective (SLOs) for search, including latency targets and availability, along with search quality.

• Create SLOs in Kibana that track your search health over time, with burn rate alerting.

• Build operational dashboards with Elasticsearch Query Language (ES|QL) that show latency percentiles and time breakdowns and that include trends.

• Set up alerts for latency regressions and error spikes, along with zero-results rate increases.

• Establish an incident response pattern for search degradation.

What you'll need

• Search instrumentation from Blogs 2–4 (search spans with search.* attributes in Elastic).

• Kibana access with permissions to create SLOs and alert rules.

• Basic understanding of SLOs. (We'll explain the search-specific parts.)

• An Elastic cluster with an Enterprise subscription, an Elastic Cloud trial, or a local deployment with the trial activated.

Why search monitoring matters beyond cluster health

Search is the primary navigation path for a significant share of visitors, and search-initiated sessions tend to show stronger purchase intent than browse sessions. A search outage is a revenue event, rather than a minor feature degradation. A latency regression from 100ms to 500ms changes user behavior before anyone files a ticket.

Most platform teams monitor search at the infrastructure level, checking whether the Elasticsearch cluster is healthy and whether nodes are responding. They also determine whether the disk is full. These are all necessary but not sufficient. A cluster can be green while search quality silently degrades; for example, queries returning stale data after a bad index deployment or latency creeping up as the index grows. This could also include zero-results rates climbing because a synonym list wasn't updated.

The gap is between "search is up" and "search is working well."

A note on examples: As we have throughout this series, we use ecommerce search for concrete examples, but these reliability patterns apply equally to any search application, including content platforms, internal knowledge bases, job boards, and support portals.

OpenTelemetry search spans as monitoring signals

If your team followed Blogs 2–4 in this series, every search request already emits an OTel span with search.* attributes. These include the query text, result count, Elasticsearch execution time, and error status. Those spans land in traces-generic.otel-default in Elastic.

Following along with code? The reference project has all the instrumentation from Blogs 2–4. Generate traffic, and then follow along with the SLO and dashboard setup below. See queries/blog6_reliability.esql for ready-to-run queries.

The search team built that instrumentation for product analytics; that is, understanding what users search for and measuring click-through rates (CTRs) and conversion rates. They’ve prioritized relevance work, but the same spans contain everything you need for operational monitoring. Span duration gives you a latency signal, and search.result_count == 0 value reflects quality. Span errors point to availability signals.

This post shows how to put this operational value to work, beginning with what you can see right now in Kibana and then building SLOs and alerting on top of it, along with incident response.

Search monitoring out of the box with Elastic APM

Before building anything new, let's look at what Elastic APM already gives you out of the box.

If your search API is instrumented with Elastic Distribution of OpenTelemetry (EDOT) (as in Blog 2), it appears automatically as a service in the Elastic APM UI. Open Observability > APM > Services in Kibana, and select your search service (named search-analytics-demo if you're using the reference project). You'll immediately see the following:

Elastic APM service overview for search

The service overview page shows latency distribution and throughput over time, along with error rate, and doesn’t require configuration. You can see at a glance whether search is healthy, and the time-series charts make regressions obvious. If latency crept up after yesterday's deployment, you'll see it here.

Trace waterfall: breaking down search request latency

Click into any transaction, and you'll see the trace waterfall, which is a visual breakdown of every span in the request. For a search API call, this typically shows:

The waterfall makes the invisible visible. You can see that the 503ms API response breaks down into HTTP handling, a 241ms query rules lookup, and a 260ms Elasticsearch query,  plus the custom search span (36ms) carrying all of our search.* attributes. Click any span, and the metadata flyout shows exactly what was captured: search.query: "usb hub", search.result_count: 33, search.took_ms: 43, the index name, hit IDs, and more.

Blog 2 discussed the gap between search.took_ms and span duration. The waterfall shows you exactly where that gap lives, without writing any queries.

Automatic search error capture with OpenTelemetry

One of the most valuable things OTel auto-instrumentation gives you is automatic error capture. When an Elasticsearch query fails because of issues like a tripped circuit breaker or a timeout, or if an index isn’t found,  the span records the exception type and message, along with the stack trace. Blog 4 mentioned this as a side benefit of span-based conversion tracking; here it becomes an operational lifeline.

The errors tab on your service page automatically aggregates these, grouped by error type and frequency. The instrumentation captures the details for you, so you don't need custom error handling or logging. During an incident, this is often the fastest way to understand what's actually failing.

Service map: search API and Elasticsearch dependencies

The service map shows dependencies between your search API and Elasticsearch, making it easy to see whether a latency problem is in your service or in the cluster it depends on.

All of this is available the moment you deploy the instrumentation from Blog 2, without building any dashboards or writing any queries. This is the foundation everything else in this post builds on.

Defining service level objectives for search

An SLO defines good enough in measurable terms. You define what working means, measure it continuously, and alert when you're burning through your error budget too fast, instead of reacting when something breaks.

Elastic Observability has a built-in SLO framework that handles Service Level Indicator (SLI) calculation and budget tracking. It also takes care of burn rate alerting. You create SLOs directly in Kibana. No ES|QL or custom pipelines are required for the core indicators.

Three SLOs every search service needs

Navigate to Observability > SLOs in Kibana, and click Create SLO. The SLO creation workflow walks you through three steps: Define the SLI (what to measure), set the objective (the target), and describe the SLO.

1. Search latency SLO: 99% of queries under 250ms

Indicator type: Elastic APM latency target: 99% of searches complete in under 250ms.

The Elastic APM latency indicator is purpose-built for this. Select your search service (search-analytics-demo), and set the threshold to 250ms.  Elastic handles the rest, including calculating the percentage of transactions below the threshold and tracking your error budget over time.

Note: The Elastic APM latency indicator measures all HTTP transactions for the search-analytics-demo service, including health checks and click/cart/checkout endpoints, along with static asset requests, not just the POST /api/search endpoint. For a search-only latency SLO, use a custom Kibana Query Language (KQL) indicator with name: "search" AND attributes.search.query: * on the traces-generic.otel-default index. The Elastic APM indicator is still valuable for whole-service health. For full coverage, combine both.

This measures end-to-end span duration; that is, what the user actually experiences. If your Elasticsearch query takes 50ms but the user waits 300ms because of network overhead or slow application logic, this SLO catches it. Use search.took_ms in the Elastic APM waterfall to diagnose where the latency lives when the SLO starts burning.

2. Search availability SLO: 99.9% success rate

Indicator type: Elastic APM availability target: 99.9% of searches succeed.

The Elastic APM availability indicator calculates the percentage of successful transactions for your service. When the Elasticsearch client throws an exception or the search endpoint returns a 5xx, the span's status records an error and this SLO counts it.

Note: Like the latency SLO, the Elastic APM availability indicator covers all HTTP transactions on search-analytics-demo, not just POST /api/search. Click/cart/checkout errors will consume this budget. For a search-only availability SLO, use a custom KQL indicator with name: "search" AND attributes.search.query: * for good events and name: "search" as the total query.

An 0.1% error budget on 100,000 daily searches means that you can tolerate 100 errors per day. That's tight, but search errors are hard failures and the user gets nothing. Availability SLOs should be stricter than latency SLOs.

3. Search quality SLO: tracking zero-results rate

Indicator type: Custom KQL target: 85% of searches return at least one result (zero-results rate < 15%); index: traces-generic.otel-default; good query: name: "search" AND attributes.search.result_count > 0; total query:name: "search" AND attributes.search.query: *.

Note on KQL versus ES|QL: The SLO framework uses KQL for its indicator filters rather than ES|QL. KQL uses field: valuesyntax and is the same language you see in the Kibana search bar. The ES|QL queries throughout this series are for ad hoc analysis and dashboards; KQL here is the SLO indicator's document filter. Both query the same traces-generic.otel-default index.

This is the SLO that surprises most teams. A search that returns an empty result set isn't an error; HTTP status is 200 and the span status is OK. Plus, no exception was thrown. But from the user's perspective, it failed. They asked for something and got nothing.

The quality SLO uses the custom KQL indicator type because it relies on our custom search.result_count attribute, which the built-in Elastic APM indicators don't know about. But the SLO framework handles everything else, including budget tracking and burn rate calculation, along with alerting.

A sudden spike in zero-results rate, such as from 12% to 40% over an hour, is almost always an infrastructure event, like a failed index deployment or a mapping change that broke queries. It could also be a synonym list misconfiguration. That's an operational problem, not a relevance problem.

Reading your search health in the SLO overview

Once the latency, availability and quality SLOs are created, the SLO overview page shows your search health at a glance:

Each SLO shows the current value, the target, the remaining error budget, and the burn rate. Green means healthy:  Search availability is at 100%, and Search quality is just above its 85% target. Red means violated: Search latency is at 50% against a 99% objective, with the burn rate breached at 200x the sustainable rate. When a budget bar starts shrinking faster than expected, you know something changed, even before users complain.

Clicking into the SLO detail shows burn rate across multiple time windows (1h, 6h, 24h, 72h) and the historical SLI trend. It also shows remaining error budget. For the latency SLO, the Elastic APM latency indicator tracks the percentage of transactions below your 250ms threshold. For the quality SLO, the custom KQL indicator uses traces-generic.otel-default with the good query filtering for attributes.search.result_count > 0. This is where the custom search.* attributes from Blog 2 pay off, since they're the foundation of meaningful SLOs.

Burn rate alerting for search SLOs

When you create an SLO through the Kibana UI, a default burn rate alert rule is automatically created. This is where the real operational value lives.

Burn rate alerts improve on threshold alerts ("error rate > 1%"), which are noisy and miss slow degradation. : Burn rate alerts measure how fast you're consuming your error budget relative to the SLO window.

A burn rate of 1.0 means that you're spending budget at exactly the sustainable rate, but a burn rate of 10.0 means that you're burning 10x too fast and you'll exhaust the budget in 1/10th of the window.

The default burn rate rule uses a multi-window approach, with four severity levels:

Severity

Burn rate

Long window

Short window

What it means

Critical (page)

> 14.4x

1 hour

5 minutes

Exhausts budget in ~50 hours

High (ticket)

> 6.0x

6 hours

30 minutes

Exhausts budget in ~5 days

Medium (review)

> 3.0x

24 hours

120 minutes

Exhausts budget in ~10 days

Low (awareness)

> 1.0x

72 hours

360 minutes

Trending toward exhaustion

The short window prevents alerting on brief spikes that self-resolve, and the long window catches sustained degradation. Together, they balance responsiveness with alert fatigue.

Routing search alerts to PagerDuty, Slack and Jira

Alerts are only useful if they reach the right people in the right tools. Elastic's alerting framework supports a wide range of connectors out of the box, including:

• Incident management: PagerDuty, Opsgenie, xMatters for on-call routing.

• Chat: Slack, Microsoft Teams for team notifications.

• Case management: Jira, ServiceNow for automatic ticket creation when SLOs breach.

• Custom: Webhooks for integrating with any system via HTTP.

You can also use Elastic's built-in cases to track incidents directly within Kibana, linking alerts and traces in one place, along with investigation notes, with push to Jira or ServiceNow when escalation is needed.

A typical routing setup:

Alert

Severity

Channel

Latency SLO burn rate > 14.4

Page

PagerDuty

Availability SLO burn rate > 14.4

Page

PagerDuty + Slack

Quality SLO burn rate > 6

Ticket

Jira (auto-create) + Slack

CTR anomaly (machine learning [ML] job)

Notification

Slack (search team)

Anomaly detection for search quality

Some search degradations are gradual shifts that slip past threshold-based alerts, rather than sudden spikes. A relevance regression after a model update might reduce CTR by 15% over a week, and latency might creep up by 5ms per day as the index grows. These are real problems, but they don't trigger burn rate alerts until it's too late.

Elastic's anomaly detection is built for exactly this. It learns normal patterns in your search metrics and flags deviations automatically, and you don’t have to configure any thresholds. 

Detecting search quality degradation with ML anomaly detection

• Latency anomalies:

 Elastic APM anomaly detection can be enabled directly from the Elastic APM UI for your search service. It learns the typical latency distribution, including daily and weekly patterns, and alerts when behavior deviates. A gradual 5ms/day creep will eventually register as anomalous before it hits your SLO threshold.

• CTR drops:

A relevance regression is invisible to traditional monitoring; latency is fine and errors are zero, plus the result counts are normal, but the ranking changed and users aren't clicking. Anomaly detection on click volume per query is a practical proxy: When a query that normally receives 20 first-click events per hour drops to 5, something likely changed.

To set this up: In Kibana → Machine Learning → Anomaly Detection, create a new job. Use the Multi-metric wizard, and select traces-generic.otel-default as the index. Configure a count detector on attributes.search.first_click split by attributes.search.query. This creates a per-query click-volume baseline and alerts when individual query engagement drops outside the expected range.

Note: This job detects click-volume anomalies per query, not CTR (which requires dividing clicks by searches). Click volume is a useful proxy (a CTR regression usually manifests as a drop in absolute click count), but be aware that a traffic surge with flat click volume would show as a CTR drop without triggering this alert. For true CTR anomaly detection, use a scheduled ES|QL transform to materialize hourly CTR values and run anomaly detection on the computed ratio.

Route the resulting ML alert rule to your Slack search channel.

• Throughput shifts:

A sudden drop or unexpected surge in search volume can indicate upstream problems (like load balancer changes or traffic shifts) or downstream issues (such as search becoming unresponsive or users retrying).

Configure ML anomaly alert rules to route these to your notification channels. These complement your SLO burn rate alerts; burn rates catch budget consumption, and anomaly detection catches pattern changes.

Building a search monitoring dashboard with ES|QL

SLOs tell you whether search is healthy. When they indicate a problem, you need a dashboard that tells you why.

The search team and the on-call team need different views of the same data. A search engineer wants query-level detail, such as which queries have low CTR and which ones return nothing. They’re also interested in where to invest in relevance. But an on-call SRE wants the operational picture, including whether search is fast and whether it’s up. It also wants to know whether search is degrading, and if so, since when.

Search monitoring panels for the on-call dashboard

Build it in Kibana dashboards using Kibana Lens panels. Lens supports ES|QL as a data source, so the queries from Blogs 2–4 can power dashboard panels directly. The key panels include:

• Search throughput over time:  A sudden drop is often the first sign of a problem.

• Latency percentiles (p50, p95, p99) over time: When they diverge (p50 flat, p99 spikes), you have a subset of slow queries.

• Error rate over time: Spikes here mean hard failures.

• Zero-results rate over time: A step change upward, especially correlated with a deployment, means something changed in the index or query pipeline.

The ES|QL for each panel follows the patterns from earlier blogs. For example, a latency percentile panel:

This includes three lines on one chart. When they diverge, such as when p50 stays flat but p99 spikes, you likely have a subset of queries that are slow while the majority are fine. That's a different diagnosis than all queries slowing down (cluster-level pressure).

Drill-down panels: slowest queries and top zero-result queries

For investigation, add a few detail panels, such as:

Slowest queries: A table showing the queries with the highest p95 latency and their search volume. During an incident, this narrows the problem from "search is slow" to "these specific queries are slow."

• Top zero-result queries: A table showing which queries most frequently return nothing. When zero-results rate spikes, this panel immediately shows which queries are responsible.

These drill-down panels use the same ES|QL patterns as Blogs 2 and 3, just surfaced on a persistent dashboard instead of run ad hoc.

Search incident response using OpenTelemetry traces

As an example, an alert fires, noting that search latency has spiked. What happens now?

The trace data from Blog 2's instrumentation gives you a structured path from symptom to root cause.

Step 1: Assess scope

Start at the on-call dashboard, and get answers to the basics:

• When did it start? Narrow the time range to the degradation window.

• How bad is it? Is p50 affected (all queries slow) or just p99 (a subset)?

• Is it just search? Check the Elastic APM service map to determine whether the Elasticsearch dependency is also degraded.

Step 2: Find the problem queries

If the problem is a subset of queries (p99 spike but p50 is fine), use the slowest queries panel or run:

Adjust the 100 ms threshold to match your environment's normal range. It can be lower for a fast cluster or higher if your data volume makes 100ms typical. This narrows the problem from "search is slow" to "these specific queries are slow." That's the difference between restarting the cluster and investigating a specific query pattern.

Step 3: Drill into the trace waterfall

Pick a slow query, and open it in the Elastic APM trace view. The waterfall shows exactly where time was spent. (Refer back to the trace waterfall GIF above to see a real example of a POST /api/search trace broken down into its component spans.)

The overhead gap between search.took_ms (Elasticsearch time) and span duration (end-to-end time) is your diagnostic tool:

Scenario

search.took_ms

Span duration

Diagnosis

Elasticsearch slow

400ms

430ms

Elasticsearch problem: Check slow log, cluster metrics.

App slow

50ms

350ms

Application / network overhead: Check serialization, network.

Both slow

400ms

700ms

Multiple issues: Investigate both.

If the problem is in Elasticsearch, drill into the Search Profile API or cluster monitoring. If it's application overhead, look at the spans around the search span in the waterfall.

Step 4: Correlate with events

Check whether the degradation correlates with:

• Deployments: Did someone deploy a new version of the search service or push a new index?

• Cluster events: Is Elasticsearch under memory pressure, or are there long garbage collection pauses? Or maybe the disk is I/O saturated?

• Network: Is latency between the search service and Elasticsearch elevated?

Elastic Observability's unified platform makes this correlation straightforward because traces, logs, metrics, and infrastructure data all live in the same Kibana instance. You're adding filters in the same interface, rather than switching between tools.

Going further: Infrastructure metrics and cost attribution

This post focuses on what you get from trace data; that is, the spans your search API already emits. But OTel and Elastic Observability support a wider instrumentation picture that becomes valuable as your search infrastructure matures.

• Infrastructure metrics. Adding host and container metrics (like CPU, memory, disk I/O, and network) alongside your traces lets you correlate search performance with infrastructure use. When p99 latency spikes, you can immediately see whether the Elasticsearch nodes are under memory pressure and whether garbage collection  pauses are increasing. You can also check whether disk I/O is saturated, and you can do all this in the same Kibana interface. The Elastic Agent collects these automatically for your infrastructure, and the infrastructure monitoring UI surfaces them alongside your Elastic APM data.

• Total cost attribution (TCA). With infrastructure metrics flowing alongside traces, you can start attributing infrastructure costs to specific services and operations. How much compute does your search service consume? How does that correlate with query volume? If a new ranking model doubles CPU usage per query, you can see the cost impact directly. This is particularly valuable for teams running search on cloud infrastructure where costs scale with resource consumption; understanding the cost per search helps justify infrastructure investment and identify optimization opportunities.

• Logs correlation. OTel auto-instrumentation injects trace context (such as trace ID and span ID) into your application logs. This means that when you're investigating a slow search in the trace waterfall, you can click through to the exact log lines from that request, including Elasticsearch slow log entries and application debug output. It also includes error details that don't fit in span attributes. The logs correlation feature automatically ties them together.

These are natural next steps once you have traces working. Each one extends the same unified platform, without new tools or separate pipelines.

How search analytics and search monitoring share one data pipeline

Here's how it all fits together:

The data flows from the instrumentation you built in Blog 2. This one investment supports two audiences: The search team gets product analytics (Blogs 2–5), and the SRE team gets operational monitoring (this post). Neither team needs separate data pipelines.

Getting started with search monitoring in Elastic

This is the last post in the series, and it brings us full circle. Blog 1 describes the vision: Instrument search once with OTel, and send spans to Elastic. Then use ES|QL to answer any question about search behavior. Blogs 2–4 build the instrumentation and analytics, and Blog 5 shows how to feed that data back into relevance improvements. This post shows how the same data powers operational monitoring, including SLOs, alerting, anomaly detection, and incident response.

The key takeaway for search engineers is that the instrumentation you built for analytics already generates the signals. The SLOs and alerts are built-in capabilities of Elastic Observability, as are the dashboards. You're closer to production-grade search monitoring than you might think. Observability isn't a separate discipline you need to learn from scratch. 

If you've been following along and built the instrumentation from Blogs 2–4, start here:

1. Open Elastic APM: Look at your search service, and explore a trace waterfall. You can also check the errors tab.

2. Create three SLOs: Latency (Elastic APM latency), availability (Elastic APM availability), and quality (custom KQL for zero-results).

3. Enable anomaly detection: One click in the Elastic APM UI for latency anomalies.

4. Build the on-call dashboard: Four Lens panels with the queries from this post.

By the end of an afternoon of work, your search service can have the same observability coverage as any other critical production system.

Get started

Working code

• Reference project: Working code for the entire blog series; clone, configure, and run.

Elastic APM and traces

• Elastic APM Overview: Elastic APM concepts and trace analysis.

• Elastic APM UI: Service overview, transactions, dependencies, errors.

• Service Maps: Dependency visualization and health.

• OpenTelemetry Integration: OTel ingestion in Elastic.

SLOs and alerting

• SLOs in Elastic Observability: Creating and managing SLOs.

• Create an SLO: SLI types (like Elastic APM latency, Elastic APM availability, or custom KQL), time windows, budgeting.

• SLO Burn Rate Alerts: Multi-window burn rate alerting.

• Alert Connectors: Slack, PagerDuty, webhook, email integrations.

• Alerting Framework: Rule types and configuration.

• Jira Connector: Automatic ticket creation from alerts.

• ServiceNow Connector: ITSM integration.

• Elastic Cases: Built-in incident tracking with external push.

Anomaly detection

• Anomaly Detection Overview: Unsupervised time series anomaly detection.

• Elastic APM Anomaly Detection: Enable ML for latency, throughput, error rate.

• ML Anomaly Alert Rules: Alerting on detected anomalies.

Dashboards and ES|QL

• Kibana dashboards: Building operational dashboards.

• Lens: Visualization editor.

• ES|QL in Lens: ES|QL-powered dashboard panels.

• ES|QL Overview: Language reference.

Elasticsearch operations

• Slow Log Configuration: Threshold-based query slow logging.

• Search Profiling: Profile API for query execution analysis.

• Monitoring Elasticsearch: Cluster stats, search rate, latency.

From this series

• Modern search analytics with OpenTelemetry: The vision.

• Instrument your search API : Search spans and search.* attributes.

• Measuring search quality : CTR, MRR, click distribution.

• From clicks to conversions: Conversion tracking and revenue attribution.

• Personalizing search from behavior: Judgment lists, rank features, Learning To Rank (LTR).

This is the final post in a six-part series on search analytics with OpenTelemetry and Elastic. Start from the beginning: Modern search analytics with OpenTelemetry, or to start building, jump to Instrument your search API.

NASA Aviation Safety Reporting System (ASRS) reports describe unusual events during flights, such as missed altitudes, confusing clearances, runway issues, or mechanical problems. Each report already has an official category, like altitude deviation, course deviation, or ground encounter. In this article, we ask a different question: What does this report reveal about the pilot who wrote it? The idea is to ask a model to infer a schema grounded on the data to classify the report based on criteria that help us figure out information about the report writers. Then ask a second model to apply the labels.

You can find the full workflow definitions and helper scripts here.

What you need to run this LLM pipeline

• Elastic Stack 9.4+ or Elastic Cloud Serverless. Elastic Workflows has been generally available (GA) since 9.4.

• Elastic Agent Builder enabled in your deployment.

• A Kibana generative AI (GenAI) connector pointing at Claude Sonnet (or an equivalent reasoning model). This is the planner.

• A Mistral API key. We’ll use it to register an Elasticsearch inference endpoint.

• Python 3.10+ with elasticsearch>=9.0 and pandas. Used by the dataset loader.

How two-tier LLM orchestration works

The workflow has two jobs: Decide what labels should exist, and then apply those labels to every report.

The first job is open-ended. A large model reads a varied sample of reports and proposes a small schema of categorical fields. A field is one way to describe the writer, such as attribution_style or procedure_orientation. Each field has a few allowed values, such as self_critical, system_attributing, or balanced.

The second job is repeatable. After a human approves the schema, a smaller model reads each report and chooses one value for each field.

We use Elastic Workflows because the steps are known ahead of time: sample reports, propose labels, wait for approval, classify every document, and store the results. Writing those steps in YAML makes the process reproducible, observable, and cheaper to rerun.

Why split LLM work across two model tiers?

A small model could handle classification, but schema discovery is a different shape of problem. It requires reading a diverse sample, spotting latent patterns, and proposing complex structures. In practice, smaller models over-anchor on surface keywords and produce redundant or nonexclusive fields.

Classification is simpler, the schema exists, the values are enumerated, and the task is to pick one per field. A smaller model handles this reliably and at a fraction of the cost, since it runs once per document across the entire corpus.

Large and small here mean reasoning capability. In this article, Claude Sonnet plays the planner and Mistral Small 3.1 plays the executor.

Classifying NASA pilot reports with a two-tier LLM pipeline

We’ll use the NASA ASRS database, which collects voluntary, anonymous incident reports from pilots, controllers, and mechanics. The dataset is public, and the reports are written as free-text narratives.

What we want to ask is:

What does this report reveal about the pilot who wrote it?

The planner reads a varied sample of reports and decides which distinctions are meaningful based on how the reports are actually written.

Step

Role

Model tier

sample

Pull a diverse subset of reports from the corpus.

(no LLM)

discover

Read the question and the sample, propose a schema of fields with enum values.

Large

approve

Human reviews the proposed schema and approves or edits it.

(Human via waitForInput)

apply

Iterate over the corpus, assign one value per field to each report.

Small

store

Write the schema and the per-document field values to Elasticsearch.

(No LLM)

Registering Mistral and Claude as Elasticsearch inference endpoints

The small model will be registered as an Elasticsearch inference endpoint using the native mistral service integration. 

The alias mistral-small-latest resolves to Mistral Small 3.1. It has a 128k context window and supports JSON-mode output.

The large model will be an AI connector pointing at Claude Sonnet. The Agent Builder UI walks you through creating the connector. Take a note of the connector ID since we’ll reference it from the workflow.

Indexing NASA ASRS incident reports into Elasticsearch

The ASRS dataset is indexed with keyword mappings for aggregation fields and text mappings for the narratives the models will read.

Download the ASRS CSV (the database publishes quarterly extracts at the ASRS Database Online page), and index it. The mappings are:

The mapping types follow how each field is used. flight_phase and anomaly are mapped as keyword because we’ll run terms aggregations on them to build the sample, and aggregations need exact, non-analyzed values. narrative and synopsis are mapped as text because they hold free-form prose that the models will read. The companion notebook has the full loader script that reads the CSV and bulk-indexes the documents.

Building a stratified sample for the planning LLM

The YAML snippets in this and the following sections are steps of the workflow definition that the notebook registers via the Workflows API. The first two steps generate a representative sample: They aggregate by flight phase and by anomaly and pull a few documents per bucket with top_hits.

How the large LLM discovers a classification schema from the data

The prompt needs both the question and the sample. A question alone may produce generic labels disconnected from the corpus, and a sample alone produces descriptive clusters that ignore the angle of the question. 

When both are present and the output is structured, the model produces labels that are grounded in the data and oriented to the task: a schema of categorical fields, each with two to four mutually exclusive value options backed by evidence from the sample.

Here’s the planner step from the workflow:

The structured output schema enforces the shape of the response:

• name: Identifier for the categorical field.

• definition: What this field measures, in one sentence.

• why_useful: How this field serves the question; this also helps the downstream classifier understand the intent.

• values: Two to four mutually exclusive options. Each has a value and a definition.

Here’s an example of the produced schema. We can see how the writer is being classified and the reasons why the model decided to create the category. definitionand why_useful fields are used by the second model to classify the documents.

Human-in-the-loop schema approval with waitForInput

The proposed schema is now passed to a person for approval. Elastic Workflows has a waitForInput step that pauses the workflow with a schema, exposes a form, and resumes when the input is submitted.

waitForInput has no timeout of its own, so if nobody responds, the execution waits indefinitely. To put a limit on that, set a workflow-level settings.timeout; if it elapses before the reviewer submits the form, the execution is canceled.

When the workflow reaches this step, the execution pauses and the Kibana UI shows an "Action is required" badge. Clicking Provide action opens a form where the reviewer can paste or edit the schema JSON. Since waitForInput cannot be prepopulated from a previous step, the code polls the discover step output and prints a paste-ready JSON block that can be copied directly into this form.

The step_output helper (in the notebook) polls the execution via GET /api/workflows/executions/{id} until the discover step completes and then returns its output.

Code JSON output pasted on Kibana:

The reviewer can keep useful fields, rewrite unclear ones, merge overlapping values, and add notes. After approval, the workflow resumes and sends the final schema to the executor step.

For a new corpus, keep this human gate in place. Once the schema is stable, you can auto-approve and only fall back to review when it’s worth it: Route just the low-confidence extractions to a person, or compare a new discovery run against the schema stored in the schemas index and trigger review only when fields or values change beyond a threshold.

Classifying the full corpus with a smaller LLM

By the time the workflow reaches this step, the open-ended part of the job is over. From here, the small model takes over and classifies each report against the approved schema.

Note: The classification step uses ai.agent instead of ai.prompt because ai.agent accepts an inference-id, which lets it call the Elasticsearch _inference endpoint directly, while ai.prompt only accepts a connector-id.

The fetch_corpus step is the third elasticsearch.request in the workflow, so it’s worth saying why we read from the index again. The first two (by_phase and by_anomaly) only pulled a small stratified sample for the planner to reason over, not the data to label. Now that the schema is approved, fetch_corpus pulls the documents we actually want to classify. We cap it at 100 with match_all to keep the demo fast; this is where you would page through the full corpus.

For every field, it returns a value (or null), a confidence, and a short quote. Setting additionalProperties: true in the JSON schema lets the step return one entry per field without the workflow having to know the field names ahead of time. A stored extraction looks like this:

Here, review_required is true because procedure_orientation came back at 0.44 confidence, below our 0.5 threshold, which is the signal a confidence-based quality gate would act on.

The fetch_corpus step pulls the documents to classify. The foreachstep iterates over them sequentially, and iteration-on-failure handles the errors: retry covers transient API errors from the inference endpoint, and, if all attempts fail, the fallback step posts to Slack so the failure doesn’t pass silently. (An email connector works the same way.) continue: true then lets the loop move on to the next document instead of failing the whole run. 

For production-scale corpora, consider using executeAsync, which is the fan-out version of execute.

Writing schemas and extractions back to Elasticsearch

The workflow produces two things: the approved schema and the per-document field values. The store_schema step runs right after the human gate, before the classification step fans out:

Each extraction is written inside the foreach loop, so results are persisted as they’re produced rather than batched at the end.

The schemas index holds one document per discovery run (question, approved fields, reviewer notes). The extractions index holds one document per report per schema version. 

What this two-tier LLM orchestration pattern gives you

We built one Elastic workflow that pulls a stratified sample from an incident report index, sends it with a question to a large reasoning model to generate a classification schema based on the data and a user-defined angle, pauses for human approval, and then iterates over the full corpus with a small Mistral model that assigns one value per field. 

The approved schema and per-document field values are written back to Elasticsearch as structured data.

The point of the exercise is that two different shapes of work, schema discovery, and schema application can use two different model tiers and that a workflow lets you write the routing decision down.

Next steps for your own LLM pipeline

• Try it on a corpus of your own. The pattern doesn’t care whether the input is incident reports, customer feedback, weekly status updates, or property listings.

• Promote the foreach step to workflow.executeAsync once you’re comfortable for parallel fan-out at scale.

• Schedule the rediscovery workflow on a cron trigger so you can discover different schema variations based on the data that comes in.

• Read the Elastic Workflows documentation for the full step catalog.

Related reading

• Build AI agents with Elastic Inference Service (EIS) covers the broader multi-model wiring pattern via EIS, complementary to the Workflows-orchestrated split shown here.

• How to build AI agentic workflows with Elasticsearch is a higher-level survey of how Agent Builder and Workflows fit together.

• LangExtract and Elasticsearch tutorial explores a different extraction pattern using a hand-authored schema; useful for contrast with the discover-then-apply approach above.

Elasticsearch supports a diverse range of use cases, including observability metrics and logs and complex geospatial analysis. However, as vector search becomes a core component of modern architectures, the need for specialized optimization has grown. Achieving peak performance for vector-heavy workloads often requires navigating a complex space of configuration knobs. To reduce this operational toil, we wanted to provide opinionated, high-performance defaults through a single setting that simplifies performance tuning for production environments. The vectordb_document index mode is designed specifically for optimal vector search workloads.

How to set up vectordb_document mode

Setup is a single setting. When you’re creating an index, define the following in the index settings:

vectordb_document mode is available on all Elasticsearch subscription tiers, including Basic.

Most indices used for vector search also support other operations, such as aggregations or geo-search. They also support hybrid search. Because vector search is typically the most computationally demanding part of these mixed workloads, we recommend using the vectordb_document index mode to prioritize performance for your most intensive operations. An index in vectordb_document mode remains highly capable, supporting almost all operations available in the default standard mode, while optimizing specifically for the resource-heavy demands of vector search.

The “_document” suffix represents our roadmap. We're also developing a “vectordb_columnar” mode as another way to optimize vector search, suited to different data and access patterns.

Elasticsearch vector search benchmarks across four datasets

To validate these defaults, we performed extensive benchmarking across various datasets and two index types: bbq_hnsw and bbq_disk. We ran all benchmarks on a single-node Elasticsearch instance on AWS using a c8gd.2xlarge instance (Graviton 4, ARM64, local NVMe SSD) with a pod limited to 8GB RAM (2GB heap) and 4 CPUs, using a single shard.

Datasets:

Dataset

Vectors

Dims

Use case

laion-img-emb-512-20M-cosine

20 million

512

Pure vector search (low dim)

msmarco-v2-10M-jina-v5-1024

10 million

1024

Pure vector search (med dim)

dbpedia-openai-1M-3072-angular

1 million

3072

Pure vector search (high dim)

arxiv-for-fanns-large

2.7 million

4096

Filtered search

bbq_hnsw: HNSW index performance with vectordb_document

Query throughput and recall

On all four datasets, vectordb_document produced a better QPS–recall curve, and the shape of the advantage is itself informative. On dbpedia-openai-1M, msmarco-v2-10M, and arxiv-for-fanns-large, the curves start close together at low recall (essentially identical on arXiv) and separate as recall rises, reaching roughly 1.4×, 2.2×, and 2× at the high-recall end. On laion-img-emb-512-20M, the curves are apart from the start and settle at about 2× from recall 0.80 upward.

The gap widens with recall because higher recall is bought with oversampling, and oversampling is exactly where vectordb_document saves. The two effects compound. bfloat16 storage halves the bytes read per rescored candidate. More importantly for hierarchical navigable small world (HNSW), the oversample factor is applied to the graph search itself;  each segment is searched for k × oversample candidates, so cost scales with oversample times segment count. Unthrottled parallel merging leaves fewer, larger graphs (14–21 segments versus 22–29), and the graph and quantized vector files are preloaded into the filesystem cache, so vectordb_document pays far less for each increment of oversample.

Comparing identical search settings, rather than equal recall, makes the effect explicit. With rescoring switched off, the two modes are within 1%–28% of each other; at oversample 5, vectordb_document is 2–4× faster; at oversample 10, up to 10×. Those high-oversample settings sit off the QPS–recall frontier, which is why the curves above top out nearer 2×, but they isolate where the saving comes from.

Oversample

DBpedia

LAION

MS MARCO

arXiv

off

1.23×

1.28×

1.11×

1.01×

5

4.07×

2.66×

2.66×

2.05×

10

10.5×

2.33×

2.46×

4.03×

Indexing speed and merge behavior 

In our benchmarks, vectordb_document increased upload time by roughly 16%. This is expected: merges now run unthrottled and parallelized across threads, so they compete with indexing for CPU while documents are still being ingested. Measured indexing time rose 9%–26% across the four datasets. HNSW graph construction is CPU-bound, so that contention is felt directly.

The same changes make the post-upload phase much cheaper. Disabling auto-throttling removes merge rate limiting entirely (in default mode, DBpedia spent 57% of its merge time throttled), and bfloat16 halves the raw vector data, cutting total bytes merged by 42%–49%. Post-ingest merging finished 75%–86% sooner on three of the four datasets, which brings total time to a search-ready index to only about 6% above the baseline. DBpedia was the exception in the other direction: Its merge tail dominated, so total time actually fell 25%.

bbq_disk: disk-based vector search performance with vectordb_document

bbq_disk query throughput and recall

For bbq_disk indices, vectordb_document mode's impact on search throughput varied by dataset. On the lower-dimensional datasets, QPS was essentially unchanged at equal recall: laion-img-emb-512-20M-cosine (512 dims) and msmarco-v2-10M-jina-v5-1024 (1024 dims) track each other closely across the recall range, with vectordb_document ahead in the lower recall end and a few percent behind at the high-recall end. On the higher-dimensional datasets, we saw a consistent gain of roughly 20% at equal recall: dbpedia-openai-1M-3072-angular (3072 dims) and arxiv-for-fanns-large (4096 dims).

Our interpretation is that this is mainly a rescoring effect. vectordb_document stores vectors as bfloat16, so each rescored candidate reads 2× dims bytes instead of 4× dims. The sweep supports this directly: The advantage grows with the query-time oversample factor, which is exactly what sets how many candidates get rescored. On DBpedia, the QPS ratio rises from 1.16× at oversample 3 to 4.3× at oversample 8, and on arXiv from 1.20× to 1.56×, while on laion and MS MARCO, it stays flat or drifts just below 1. Rescoring is simply a much larger share of the query on the smaller, higher-dimensional datasets; on the 10 million and 20 million ones, scanning 1-bit posting lists dominates.

Indexing speed and merge behavior 

Across the four datasets, vectordb_document cut total time to a fully merged, search-ready index by about 20%. Every dataset improved, from 6% on msmarco-v2-10M to 50% on dbpedia-openai-1M, where the post-upload merge phase alone fell from 322 seconds to 54 seconds. Upload time on its own is less clear-cut: On the day charted below, it finished 2%–15% sooner, depending on dataset, but on the earlier run set, it was marginally slower, so we read upload as unchanged to modestly faster and treat time-to-searchable as the real result.

The gain is almost entirely in merging. In default mode, Elasticsearch rate-limits how fast merges may write, and that limiter was binding hard: 75% of all merge time on DBpedia and 73% on laion was spent paused by it. vectordb_document disables the limiter, so paused time is zero and merge time drops 63% on DBpedia and 42% on laion. bfloat16 helps for the same reason: The limiter meters bytes written, so halving the raw vector data means less to write under the cap. The gradient across datasets follows the throttling rather than the byte count: arXiv, at 21% of merge time throttled, saw 30% less merge time, while MS MARCO, never throttled, got 55% fewer bytes but only 11% less merge time.

The two index types respond differently on ingest because merging costs something different in each. Merging bbq_hnsw segments means rebuilding HNSW graphs: CPU-bound work that competes directly with the equally CPU-bound graph construction on incoming documents, and on a 4-CPU pod that contention surfaces as a slower upload. bbq_disk merges are dominated by writing bytes rather than by CPU, so lifting the rate limiter spends I/O bandwidth that the local NVMe has to spare, and bfloat16 means that there are fewer bytes to write in the first place. Both index types reach a fully merged index far sooner; the difference is only whether the upload phase pays for it.

What vectordb_document sets under the hood

element_type (dense_vector)

• Value: bfloat16.

• Impact: Stores each dimension of the raw vector as bfloat16 instead of the default float32, halving the storage of raw vectors with negligible impact on recall.

• Benefit: Lower disk footprint (reduced total cost of ownership [TCO]), faster fetching of vectors for rescoring.

Dynamic float array mapping

• Value: Float arrays with 32 or more values are dynamically mapped as dense_vector. (In the default mode, this threshold is 128.)

• Impact: No need to explicitly declare the field as a dense vector field; the system identifies it automatically.

• Benefit: Configuration simplicity.

exclude_source_vectors

• Value: true.

• Impact: Vectors are stored once in the vector index and not duplicated in _source. They’re omitted from _source in responses, but they can still be retrieved on request.

Benefit: Lower disk footprint; faster queries, since large vectors are no longer shipped with every _source fetch.

index.store.preload

• Value: ["vex", "veq", "veb", "cenivf"].

• Impact: Preloads search-time vector data structures into the filesystem cache whenever new segments are opened.

• Benefit: Reduced query latency.

index.merge.intra_merge_parallelism_enabled

• Value: true.

• Impact: Use parallel threads for segment merging to achieve optimal segment sizes.

• Benefit: Faster convergence to fewer, larger segments leads to better recall, lower query latency.

index.merge.scheduler.auto_throttle

• Value: false.

• Impact: Allows merges to proceed at full speed.

• Benefit: Merges reach optimal segment sizes sooner, resulting in better recall, lower query latency.

While these defaults are optimized for the majority of use cases, most can be overridden individually to suit unique hardware constraints or extreme performance requirements, with one exception: exclude_source_vectors: true is locked in and cannot be changed on a vectordb_document index.

Summary

The vectordb_document index mode is a production-ready foundation for vector search: By adopting high-performance defaults, teams can focus on building features rather than on hand-tuning merge or storage settings or on adjusting preload settings. 

Across four datasets, the picture is consistently favorable, though the balance differs by index type. With bbq_hnsw, QPS recall improved on every dataset, from roughly on par at low recall to as much as double at the high-recall end, in exchange for a modest ingest cost of about 16% longer upload and 6% longer time to a search-ready index. With bbq_disk, the trade runs the other way: Total time to a search-ready index dropped by about 20%, while search improved by about 20% at equal recall on the higher-dimensional datasets (3072 and 4096 dims) and was essentially unchanged on the lower-dimensional ones. The split comes down to what merging costs in each: Rebuilding HNSW graphs is CPU work that competes with indexing; whereas bbq_disk merges are write-bound and simply run faster once they’re unthrottled.

In both cases, the new defaults move the system in the direction that most users want, with no per-index tuning required. And, for the settings that genuinely depend on the data itself, such as the degree of quantization, auto-calibration now derives them for you. (See How Elasticsearch auto-tunes vector quantization to hit your recall target.)

Try it out by creating an index with the vectordb_document index mode in Stateful Elasticsearch 9.5 or in Serverless.

Observability workloads are storage-intensive by nature, and the composition of that storage determines both cost and query performance. ES95 introduces adaptive compression: rather than applying the same encoding to every numeric field, it automatically selects the encoding that best matches each field's structure. The result is a 33.6% reduction in total doc-values storage, 19% to 74% reduction on floating-point gauge metrics and a 92% reduction in @timestamp. No configuration or migration required.

Observability data is storage-intensive

Data processing systems are rarely limited by how fast they can compute. They’re limited by how fast they can move bytes: off disk, across the network, and through the memory hierarchy. Compression is how a storage engine trades CPU time for memory bandwidth, spending comparatively cheap CPU cycles so fewer bytes have to travel through the parts of the system that are usually constrained. In a read-heavy system like Elasticsearch, that trade-off pays back every time data is queried, often long after it was written.

Storage size and query performance move together; fewer bytes on disk means fewer bytes to read on every range query, every aggregation and every dashboard load. Compression is not just about saving storage. Every byte that is never written is also a byte that never has to be read.

The right encoding depends on the structure of the values themselves, and the largest wins come from exploiting the structure already present in the data rather than squeezing an opaque stream of bytes. Few workloads expose that structure more clearly than observability metrics.

A single host reports hundreds of metrics every few seconds, including CPU utilization, memory ratios, request latencies, and network throughput. Multiply that by thousands of hosts across weeks of retention, and the bytes accumulate fast. Most of that volume is structured but not uniform: timestamps arrive at near-constant intervals from thousands of concurrent series, counters increase monotonically, while gauges like 23.47 or1.15 are short decimal measurements.

A fixed compression approach cannot adapt to that variety. A timestamp column and a floating-point gauge column compress through fundamentally different techniques, but a codec that applies the same approach to both will necessarily handle one of them poorly. For most of Elasticsearch's time-series codec history, gauges were on the losing end of that trade-off.

The structure the old codec was not built to exploit

Elasticsearch stores time-series numeric values in doc values: a column-oriented structure where all values for the same field sit adjacent on disk. That adjacency makes compression possible: the codec compares consecutive values of the same field, finds patterns, and exploits them.

The time-series codec before ES95 applied the same fixed encoding to every numeric field: delta encoding followed by normalization, GCD (greatest common divisor) reduction, and bit-packing. Each encoding technique activated where it helped and skipped where it would not. For timestamps and integer counters, this approach was remarkably effective. For floating-point gauges, it could find almost nothing to work with.

The reason is how they’re stored. To support range queries, Elasticsearch stores floating-point values as integers that preserve numeric ordering. A change of 0.01 in a CPU percentage reading translates to a jump of trillions in that integer space. The codec sees those large jumps and has no strategy to further reduce their footprint. Storage stays near the original eight bytes per value.

The codec was doing the right thing with the representation it had, but the latter was chosen for querying, not compression, and the goals conflict at the bit level.

The cost of a fixed format

A compression stage for floating-point values was already on the roadmap, so the interesting part wasn’t the algorithm. The obstacle was architectural.

The previous codec baked its compression approach into the storage format. Adding a new encoding meant changing the meaning of existing bytes on disk, which forced a format migration, a rollout that can last weeks or months in large production clusters. Over time, that migration burden constrains codec development itself. The question stops being Is this a good compression idea? and becomes Is it worth another format migration? That rigidity limits the cadence of codec evolution and leads to missed compression improvements.

The right encoding without configuration

ES95 solves this at the architecture level for time-series indices. Each field's encoding is no longer baked into the format. It is selected automatically at write time, based on what the field mapping already declares: the field's name, its data type, and its metric role. Timestamps are encoded differently than counters. Counters are encoded differently than gauges. ES95 encodes all of them, and it chooses the right strategy for each.

Users already tell Elasticsearch everything the codec needs to know. The mapping describes the data; the codec chooses the compression strategy.

Compression strategy is a codec concern, not a user concern.

The alternative would have been to expose per-field encoding selection as a configuration parameter, letting users opt in to better compression for specific fields. That would shift the burden of knowing which encoding fits which data type onto those least equipped to make that call and would guarantee that most deployments never see the benefit. ES95 keeps that decision inside the codec, where it belongs. This matters most in managed and serverless deployments, where users expect the system to automatically make optimal storage decisions.

The timestamp result nobody planned for

With the adaptive architecture in place, the team set out to ship the planned float-compression algorithm. Before it arrived, the architecture proved itself by substantially improving compression for timestamps.

A time-series index is sorted first by its time-series identifier (_tsid constructed by the metric’s dimensions) and then by timestamp within each series. Timestamps on disk aren’t one smooth sequence; there are many smooth sequences laid end to end, one per series, with a large jump at every boundary where one series ends and the next begins.

The codec compresses data in fixed-size blocks without regard to those series boundaries. A block straddling a series boundary holds timestamps from two different series. The jump between them breaks monotonicity, reducing delta encoding effectiveness on blocks spanning different time series. A block that would otherwise compress to near-zero bits per value ended up needing nine or more, because bit-packing encodes every value in a block using the same fixed number of bits, so one large jump sets the cost for all of them.

In the ideal case, every block belongs to a single series: timestamps increase at near-constant intervals, delta encoding captures the regularity, and bit-packing compresses the result to near-zero bits per value. With few series, boundary blocks are rare and the overhead barely registers. On an observability cluster ingesting millions of documents across thousands of series, that changes. The cost scales along two dimensions: series count and data density. More series means more boundary events; sparser series means multiple jumps packed into single blocks. In high-churn environments, both compound, and boundary blocks accumulate into a standing tax on the most-read field in any time-series workload. It’s why @timestamp storage grew faster than the data that produced it.

The fix was to detect series boundaries and treat each run as its own independent sequence. Instead of trying to encode across the jump between two series, which forces every value in the block to pay the storage cost of that one large jump, each run is compressed on its own terms. The boundary simply becomes a seam: the jump is never seen by the encoder on either side.

That encoding is called SplitDelta. @timestamp and monotonic long counters now use it by default. No format change. No migration. Existing segments retain legacy encoding.

On the high-cardinality Elasticsearch Rally benchmark, that single unplanned encoding cut counters storage by 20%–30% and @timestamp storage by 92.4%, from 1.03 GB to 79 MB. Gigabytes to megabytes, and no, that isn’t a typo.

The pluggable pipeline had already paid for itself. SplitDelta, which wasn’t part of the original plan, slotted in without a format change or migration before ALP even shipped.

ALP: recovering the decimal that was always there

Most floating-point metrics can be expressed as short decimals with no loss of accuracy: CPU utilization at 23.47, load average at 1.15. ALP (Adaptive Lossless floating-Point compression) recovers that decimal structure from the floating-point representation, converting values into integers that the existing pipeline already handles well. ES95 feeds ALP's output into the same mature integer compression pipeline used for timestamps and counters, extracting additional savings rather than treating ALP as a standalone encoding. Values that don’t fit ALP's model (such as irregular high-precision floats or special values) fall back to direct bit-packing or the original representation without degrading the rest of the block.

ALP lets Elasticsearch treat floating-point metrics according to the structure they actually contain rather than the binary representation they happen to use. It’s applied automatically to double-valued gauge fields, selected by field type and metric role, through exactly the door the architecture had built for it.

What the numbers say

Here’s what the Elasticsearch Rally benchmark looks like on a high-cardinality workload containing 2.26 billion data points. Results are from an internal tsdb-metricsgen benchmark.

Field or metric

Storage reduction (%)

@timestamp

−92.4%

cpu.load_average.5m

−74.3%

system.cpu.utilization

−63%

memory.utilization

-19%

Total doc values

−33.6%

That overall 33.6% reduction deserves context.

A time-series index contains more than metrics. Every data point also carries the labels that identify the series: host names, IP addresses, regions, container IDs. Those dimension fields are stored as keywords. ES95 doesn’t target dimensions.

On this benchmark, two dimension fields, host.ip and host.mac, accounted for 44% of doc-values storage after ES95 ran. The 33.6% total reflects that mix. The per-field breakdown is the honest picture. Compression for dimension fields is an active area of work.

The per-field variation is the most convincing result. Some gauges shrank by nearly three quarters, while others moved by less than a fifth. That spread is direct evidence that ES95 matches compression to the structure actually present in each field. A fixed encoding treats every field identically and misses most of those wins.

Better compression without extra configuration

The storage reductions from SplitDelta and ALP are the most visible results of ES95. The more consequential result is the architecture that produced them.

Before ES95, every new compression technique required a format evolution. That reality shaped which ideas were practical to pursue. Today, new encodings become implementation decisions inside the codec rather than migration projects. Existing data never needs to move, and users gain better compression on newly written data simply by upgrading Elasticsearch. SplitDelta and ALP are the first encodings to benefit from this architecture. They will not be the last.

Asking users to choose compression algorithms would only duplicate information Elasticsearch already has. There are no per-field compression parameters to tune, and no expert knowledge is required to get good storage efficiency. Different fields get different strategies because ES95 understands what kind of data each field contains, not because a user configured it. As the codec evolves, those decisions evolve with it. The API does not.

In Elasticsearch Serverless, good defaults are part of the product. Users expect the system, not configuration, to make storage decisions. ES95 is designed to honor that expectation: encoding that starts right and gets better over time.

The compression was always there

ES95 establishes a new standard for how time-series codec evolution works. New encodings become implementation decisions, not migration projects. Users get better compression on newly written data with every Elasticsearch upgrade.

The compression was already in the data. ES95 just removed what was hiding it.

Why dynamic mapping breaks down with schemaless data

Elasticsearch wants a schema. Every field you index has a mapping that defines its type, its analyzer (for text), and how it is stored. That schema makes storage compression efficient, with fast search and cheap aggregation. It becomes a liability the moment your data stops looking like a database table.

Consider the shapes that show up in real systems:

• Log events where every service adds its own attributes. One emits labels.region, another labels.k8s.pod, and another labels.tenant_id.

• OpenTelemetry resource attributes, where the set of keys is defined by whatever agent happened to send the span.

• User-supplied metadata bags, feature flags, or tagging systems, where the keys are open-ended by design.

If you map each key as its own field, the mapping grows unbounded. This is the classic "mapping explosion." Thousands of dynamically created fields inflate the cluster state, slowing down every mapping update and eventually hitting the field limit. Each field also carries overhead in the index. You pay a structural cost for keys you didn’t plan for and may query only once.

The naive escape hatch is to store the whole object as a string and give up on querying its contents. That trades one problem for another: you keep the data but lose the ability to filter or group by anything inside it.

The flattened field type is the better path. You can index an entire JSON object under a single mapped field, keeping the keys queryable with almost none of the mapping-explosion cost. 

This post covers how a flattened field stores the data on a disk and how ES|QL reads it back.

How flattened fields index dynamic JSON keys under one mapping

Map one field as flattened:

Then write arbitrary nested JSON into it:

There is exactly one field in the mapping, labels, no matter how many keys appear across your documents. The cluster state doesn’t grow when a new key shows up. The subkeys remain individually searchable. You can reference labels.region or labels.k8s.pod in queries, even though neither was ever declared.

The catch and central tradeoff is that every leaf value is a keyword. The number 4 above is indexed as the string "4". There is no numeric typing, no date parsing, and no range math on dynamic keys. Flattened fields exchange per-field richness for schema flexibility. That fact explains almost every design decision that follows.

How Elasticsearch stores flattened field JSON keys on disk

There are two types of queries on flattened fields: an unkeyed query on the root flattened field, and a keyed query on a specific subfield. Following the previous example, a query of the form labels: "us-east-1" matches a value under any key, while the querylabels.region: "us-east-1" matches a value under the specific regionkey.

To support these two distinct query formats, the flattened mapper writes each leaf value into two distinct Lucene fields.

Take this document:

The mapper produces:

• A root field under labels, holding the bare values:

• A keyed field under labels._keyed, holding the flattened key concatenated with its value:

In the keyed field, nested objects are dot-flattened into a single key (k8s.pod), and the key is joined to its value with a reserved NULL byte (\0) as the separator. Keys that contain a NULL byte are rejected at parse time, so the separator is always unambiguous. To find the value, you split on the first NULL.

These two fields make both query shapes work:

• labels: "us-east-1" matches a value under any key, so it searches the root field.

• labels.region: "us-east-1" matches a value under a specific key. It rewrites the query to the term region\0us-east-1 and searches the keyed field.

Query restrictions on flattened field subkeys

Because every key's terms live in one sorted list, the keyed field cannot answer every query shape a plain keyword field can. Three restrictions follow:

1. No fuzzy, regexp, or wildcard on a specific subkey. Nothing about the layout makes them impossible,  but the pattern would have to be combined with the key prefix so it can’t walk past the key boundary. The mapper doesn’t do that at this writing.

2. Any range query on a subkey needs at least one bound. Elasticsearch already has a query for "this field has some value here, whatever it is": the exists query. On a flattened subkey it runs as a prefix query on key\0, which sweeps every term belonging to that key. A range with neither bound would sweep exactly the same terms. Rather than support two spellings of one scan, the mapper rejects the boundless range and asks for the exists query.

3. A range query on a subkey needs the field to be indexed. A flattened field can be mapped with index: false, which skips the inverted index and keeps only doc values, the columnar per-document storage covered in the next section. Exact-match queries survive that. With no terms to look up, Elasticsearch scans the doc values column instead, which is slower but gives the same answer. Range queries have no equivalent fallback, so a range on a subkey of an unindexed flattened field throws an Exception.

All three are limits on the Lucene query the mapper is willing to build, and where you notice them depends on how you query.

On the search API, where you name labels.region directly, they come back as errors.

In ES|QL you will not see them as errors at all. There, the same limits decide only whether a predicate is pushed into Lucene or runs in the compute engine on the extracted column. 

This is pushed to a term query on the keyed field:

This is not, so the filter runs per row on the extracted keyword:

Same answer, more work. That distinction is the subject of the second half of this post.

Under the hood: how range queries stay inside key boundaries

This part is internal. You don’t need it to use the field, but it explains where the bounds rule comes from.

For a handful of documents in one segment, the shared term list looks like this:

Each key owns a contiguous slice of that list. For example, all values for key "region" are clustered together in an ordered sublist. A range with both bounds set encodes each bound the same way a term is encoded, so a lower bound of "us-east" on the region key becomes region\0us-east and an upper bound of "us-west" becomes region\0us-west. Both endpoints already carry the key prefix, so the scan can’t leave the region slice. Nothing special is needed.

The half-open case is problematic. Handing Lucene a lower bound of region\0us-east with no upper bound would scan to the end of the term list, straight through tenant_id\0acme and every other key that sorts after region. So the mapper substitutes a sentinel for the missing side:

• A missing lower bound becomes key\0, inclusive. That’s the encoding of the empty value, and it’s the first term in the key's slice.

• A missing upper bound becomes key\1, exclusive. Byte 0x01 is the next byte after the 0x00 separator, so it sorts after every key\0value term and before the first term of any other key.

A one-sided range is therefore boxed into [key\0, key\1), which makes it exactly as safe as a closed one.

The upper sentinel also covers the case where one key is a prefix of another. If an index holds both region and regionx, then region\1 still sorts below regionx\0eu-west-1, because 0x01 is smaller than the x that follows the shared region prefix. A range on region cannot leak into regionx.

How flattened fields use the inverted index and doc values

Each leaf value can be written into two Lucene structures. Both are enabled by default, but can be disabled by the mapping configuration.

• The inverted index (when the field is indexed). Two untokenized keyword terms are indexed per value, one on the root path and one on the keyed path. This powers term, prefix, and range searches.

• Doc values (when doc_values is enabled). A columnar, document-ordered structure. This powers sorting, aggregations, and ES|QL reads.

The inverted index answers the question: "Which documents contain this term?" 

Doc values answer another question: "For this document, what are the values?" Doc values are laid out column by column so a scan touches only the bytes it needs. A flattened field uses both inverted indexes and doc values, so it can serve search and analytics from the same field.

The nature of the index means that the root field is only present in some cases. When the inverted index is disabled by the mapping, any value search requires a linear scan of the doc values for the searched value. In this case, when performing a search on the root field, there isn’t much additional overhead compared to just scanning the keyed field and ignoring the key markers. So the flattened mapper skips writing the root field, relying on the keyed field for both root and keyed queries.

Why flattened fields switched from dictionary to binary doc values

Historically, flattened field doc values used Lucene's SortedSetDocValues, which is a dictionary-compressed format. This means that every key\0value value indexed across all documents per segment is stored in one big sorted, deduplicated set of values. Each document tracks a list of ordinals into that value set.

This dictionary approach provides great compression for low-cardinality fields that tend to have repeated values. It is byte-efficient to store a value only once, and then refer to it by a single integer value. However, there is overhead associated with building and maintaining that dictionary of values, and that overhead is wasted effort when operating on high-cardinality fields that don’t repeat values.

Because flattened fields are a catch-all type, their cardinality tends to be very high. So while the dictionary approach works, it’s not the most compact or scan-friendly layout for this data, especially in time-series indices where flattened bags are common and storage pressure is real.

Recent versions switched the storage to use Lucene’s BinaryDocValues. This format just stores a literal binary blob for each document, which is compressed using Zstandard by our doc values codec when written to disk.

This new binary format provides an additional benefit: it allows us to maintain original array ordering without any overhead. Flattened fields support the mapping parameter preserve_leaf_arrays, which affects how multivalued fields are returned when using synthetic source. When configured to preserve_leaf_arrays: exact, returned values preserve the order, duplicates, and nulls from the original source value.

The dictionary encoding inherent to sorted-set doc values means the returned values are sorted, deduplicated, and de-nulled. To implement preserve_leaf_arrays, flattened fields have traditionally used an additional sidecar field, tracking the required metadata to reconstruct the original source value. However, the nature of binary doc values means that this sidecar field is no longer needed. The values are just stored and returned as indexed.

Limitations of schema on read with flattened fields

The limitations below follow from the keyword-only rule and the shared keyed field:

• No numeric, date, or Boolean typing on dynamic keys. 100 and "100" are the same term.

• No fuzzy, regexp, or wildcard on a specific subkey.

• No multi-fields (fields) or copy_to on the flattened field.

• A depth_limit (default 20) on how deeply nested the object can be.

If you need real typing for a known key, flattened now supports explicitly mapped subfields: you declare individual keys with real types under a properties block, and those keys are indexed by their own typed mapper instead of the keyed field. You get numeric ranges on labels.status_code, while everything else in labels stays dynamic and keyword-only. This is the escape hatch for the handful of keys you actually know about.

Schema on read: querying flattened field JSON keys in ES|QL

Search has supported flattened fields for years. ES|QL, the newer, piped query language built on a columnar compute engine, now reads flattened fields, too. Support began in the Technical Preview of Elasticsearch 9.5.0. It has two distinct pieces.

What ES|QL returns when you select a flattened field

ES|QL uses a dedicated data type for flattened fields, rather than folding it into keyword. When you select the root, you get the whole object back as a JSON string:

Keys come back sorted. You can carry this value through a query, count it, group by it, and run the multi-value and comparison functions on it. But an opaque JSON blob is not usually what you want to filter or aggregate on. For that, you need to reach inside it and process its contents.

How FIELD_EXTRACT reads JSON keys from flattened fields

There is no dotted-path syntax for dynamic keys in ES|QL. You cannot write labels.region for an unmapped key, because to the engine the flattened root is a single leaf value, not a set of columns. Instead you use a function:

The absence of a dotted-path syntax is a tentative limitation. The keyed field already addresses individual subkeys, so a more natural syntax for reaching into a flattened root is something we are planning to support.

FIELD_EXTRACT(field, path) takes a flattened field and a key, and returns a keyword. The rules are easier to see against a document. Take this one:

And this query:

The result is:

Four things to take from this:

• The dot is part of the key, not a navigation operator. The mapper already collapsed the nested object into the flat key k8s.pod, so "k8s.pod is a direct lookup, not a walk from k8s to pod.

• Matching is exact. "k8s" returns null because there is no leaf stored at k8s, only at k8s.pod and k8s.node. For the same reason, "host" will not find "host.name", and matching is case-sensitive, so "Region" will not find "region".

• Arrays come back multi-valued. "tags" yields a multi-valued keyword you can MV_EXPAND, count, or filter on. A missing key yields null.

• Everything is a keyword. "retries" comes back as the string "4", and a Boolean leaf comes back as "true" or "false".

JSONPath syntax is rejected outright, at parse time rather than per row. Both FIELD_EXTRACT(labels, "['k8s.pod']") and FIELD_EXTRACT(labels, "tags[0]") fail with field_extract path must be a literal flattened sub-field name.

Once extracted, the value is an ordinary keyword column. You can filter on it, group by it, sort by it, or use it as the join key in a LOOKUP JOIN:

How ES|QL pushes flattened field predicates into the columnar store

The obvious way to implement FIELD_EXTRACT would be to read the whole flattened root out of storage, render it as a JSON string, hand it to the compute engine, and parse it once per row to pull out a single key. But that means reading every key to use once and paying for a JSON parse on every document. So this obvious implementation is not performant.

ES|QL avoids this whenever possible. Between storage and the compute engine sits the block loader, the step that turns stored data into the columnar blocks the engine operates on. FIELD_EXTRACT hooks into that step instead of running after it.

When ES|QL loads a column, the flattened field type inspects the request. If the request is an extraction of a single constant key and the field has doc values, it routes straight to the keyed doc-values loader, which reads the key\0value entries for only that key out of the columnar structure. The column that arrives at the compute engine already contains only that key's values. The JSON string is never built and never parsed, and the other keys in the object are never read.

When extraction can’t be fused into the block loader, for example, because the key is computed per row or the root is the output of another function such as CASE, ES|QL falls back to the parse-per-row path. The results are identical either way. Only the cost changes.

The comparison can push down further. A predicate like FIELD_EXTRACT(labels, "region") == "us-east-1" can be pushed to Lucene as a term query against the synthetic keyed field, the same region\0us-east-1 term the search path uses. So a filter on an extracted subkey can be answered by the inverted index (if available), and the projection can be answered by doc values, exactly like a first-class field, even though the key was never in the mapping.

Ordering comparisons push down, too. The four, single-sided comparators (>, >=, <, <=) and closed BETWEEN-style ranges all become a range query on that same synthetic keyed field. This is where the key\0 / key\1 sentinels earn their keep: the single-sided forms are only pushable because the mapper can box an open bound inside the key. The pushed range is treated as a candidate, and the predicate is re-evaluated on the extracted keyword column afterwards, so multi-valued keys don’t slip through.

The values are keywords, so the ordering is lexicographic, not numeric. FIELD_EXTRACT(labels, "retries") > "10" compares strings, which means "9" is greater than "10". If you need numeric ranges on a key, map it explicitly under properties, or cast the value in ESQL.

Explicitly mapped subfields behave differently on purpose. Because they carry real types, comparison semantics diverge from the keyword path, so they are loaded and compared through their own typed mapper rather than fused into the keyed loader. And when you select a flattened root that has mapped subfields, ES|QL loads it from _source, so every leaf renders as a string and no keys are dropped silently.

When to use flattened fields vs. dynamic mapping in Elasticsearch

Use flattened fields when:

• The set of keys is open-ended or unknown ahead of time.

• You would otherwise cause a mapping explosion.

• Keyword-level filtering and grouping on the values is enough, and you don’t need numeric or date semantics on the dynamic keys.

• You have a few keys that do need real types. Map those explicitly under properties, and let the rest stay dynamic.

Avoid it, or map fields normally, when the schema is stable and you need full-text analysis, numeric aggregation, or date math across the board.

Remember that flattened is not a dumping ground for JSON you have given up on. It’s a real columnar-and-inverted store for schemaless data, and with ES|QL support, it’s now a first-class analytical citizen. You can keep the messy, unmapped parts of your data messy, and still filter, group, join, and aggregate across them as needed.

Does App A v1.2.3 support Feature X? Is it compatible with App B v9.8.7 when running on Kubernetes? Will I need more JVM heap space?

These are the kinds of questions that our field teams handle constantly. Answering them is harder than it looks. Documentation offers context, but it's an abstraction that can't anticipate every possible question. When we hit one it doesn't cover, our options are to interrupt an engineer who should be developing code or to hunt through that code ourselves. Often we don't have the time or expertise to navigate that much of it.

Coding agents do this well for a single repository on your laptop. Wouldn't it be great if we could scale that to our entire code estate? As a field engineer, I wanted that capability to serve my customers: agentic code intelligence across every project, dependency, platform, and version that we support. And I wanted it grounded in linked citations and always available to everyone as a service.

So I built it with Elasticsearch and Elastic Agent Builder and packaged it into a command line interface (CLI). I released it under an Apache 2.0 license and called it Sourcerer. This blog post reports multiple performance benchmarks of Sourcerer as a code research agent and walks through the design and rationale of its implementation.

Sourcerer

Sourcerer explores code like a frontier coding agent, searching across many versioned repositories as fast as it would in a single repository, and it generates answers with linked citations that establish trust.

Sourcerer consists of:

1. A set of configuration files for tools, skills, and agents in Agent Builder.

2. A set of index templates to store and search code from Git commit snapshots.

3. A CLI to install those assets and index and prune commit snapshots from remote Git repositories.

At Elastic, we're using Sourcerer to support our customers with verifiable information about our software directly from the source. Our internal deployment has indexed over a billion lines of code from our own public and private repositories. It also includes our core dependencies, such as Apache Lucene and OpenJDK, along with our common integrations, like Kubernetes and OpenTelemetry. Our solution architects, customer architects, consulting architects, and support engineers no longer have to hunt for answers in documentation or reach out to our engineers who should be building software rather than supporting it.

Code search benchmarks

Agentic code retrieval

SWE-Explore is a new benchmark, published on June 5, 2026, by Zhang et al., that evaluates "how well coding agents explore, localize, and rank repository context." It appears to be the only benchmark that specifically tests agentic code retrieval quality. I ran the benchmark with Sourcerer to see how it performs and compares to the other coding agents from the original paper, and again with Claude Code to measure and compare its token usage and task durations with Sourcerer's.

Retrieval scores

Sourcerer performs as well as frontier coding agents on relevance metrics for code retrieval (see Figure 1). The composite retrieval score is the arithmetic mean of all retrieval metrics weighed by their Pearson correlations (r) as reported in the paper; I did this to rank the agents by a measurement of overall retrieval quality. Sourcerer trailed Claude Code by 0.002 and surpassed Codex by 0.022 on a 0.0–1.0 scale, which should be interpreted as a statistical tie, given that the results vary slightly on each run due to the indeterminism of large language models (LLMs).

Generally, all the coding agents, including Sourcerer, performed well on precision metrics and suboptimally on recall metrics, although recall metrics had lower Pearson correlations and thus less importance. Table 1 shows Sourcerer's retrieval scores alongside the scores of the other agents tested in the original paper (page 8, table 6). "SignalReg" is the inverse of what the authors called "NoiseReg" (that is, 1 – NoiseReg); I did this to keep that metric consistent with the other metrics whose ranges imply that higher is better.

Token usage and task duration

Zhang et al. didn’t publish metrics for token usage or task durations, so I ran the benchmark again to capture those metrics for Claude Code. Given the high token cost of running the benchmark with any agent that uses an LLM, I opted to test only one coding agent, and Claude Code was the one that I expected most people would find useful as a comparison.

Compared to Claude Code, Sourcerer used ~13.9% more tokens to complete all 848 benchmark tasks. Sourcerer used 156,379,977 input tokens and 1,612,175 output tokens, while Claude Code used 137,629,435 input tokens and 1,137,176 output tokens. Sourcerer took ~8.9% longer to complete all 848 benchmark tasks. Sourcerer took 40,717 seconds, and Claude Code took 37,460 seconds.

I view these results on token usage and task duration as an acceptable modest tax in exchange for efficiently searching across multiple repositories and versions. That said, there’s room to explore optimizations to Sourcerer's tools, skills, and system prompt, the harness of Agent Builder, or the search engine of Elasticsearch and Lucene.

Single-repo vs. multi-repo scope

Critically, the SWE-Explore benchmarks only measure the retrieval scores, token usage, and task duration of agents searching within the boundaries of a single commit snapshot of a repository for any given task. This is the typical search space of a development coding agent. Sourcerer's intended scope is much broader, covering many commit snapshots of many repositories. The benchmark on "search speed and scalability," covered next in this report, shows Sourcerer's unique advantage when searching across many repositories.

Retrieval benchmark methodology

Appendix A explains the configuration of these benchmarks in detail.

Sourcerer searched all indexed representations of the SWE-Explore-Bench dataset (see Appendix A). Both benchmark runs used the same GPT-5.4 model that was used in the original paper. Sourcerer communicated with GPT-5.4 through Elastic Inference Service (EIS), while Claude Code communicated with GPT-5.4 through a shim proxy to be compatible with the OpenAI API.

I made a best effort to ensure that Sourcerer's benchmark task prompt was like-for-like with Claude Code's (see Appendix A). Both agents received identical instructions for their roles and tasks, along with output formats, and differed only in their brief harness-specific instructions. I instructed Sourcerer not to use its repo discovery skill and instead gave it explicit repo filtering instructions, ensuring that it was on a level playing field with Claude Code, which already receives the resolved directories. An alternative could have been to leave Sourcerer's repo discovery skill active while instructing Claude Code to find the repository in a filesystem that has all the repositories for the benchmark. I left Sourcerer's system prompts and skills, in addition to its tools, unmodified from their defaults, given that we're comparing the two harnesses overall, and much of which in Claude Code is closed source and not visible or controllable anyway.

Code search speed and scalability

Coding agents tend to use external tools to match substrings or regular expressions as a first line of retrieval. LLMs are trained to use shell commands, like ls and grep, when exploring code on a filesystem. Claude Code's own built-in Grep tool invokes ripgrep. This is the behavior I wanted to reproduce in Elasticsearch.

Elasticsearch has a wildcard field type that can scale regular expression matching to billions of documents. Sourcerer mimics the inputs and outputs of grep in Elasticsearch using sourcerer.code.grep, an Elasticsearch Query Language (ES|QL) tool that performs anRLIKE query on awildcard field of an index where each document has the contents of a single line of code. Sourcerer also providessourcerer.code.search, which performs a BM25-ranked MATCH query against an analyzed text field, for relevance-ranked discovery rather than exact substring retrieval.

I benchmarked the speed of both approaches against the speed of ripgrep and grep on a filesystem using two different corpus sizes and pattern rarities. Corpus sizes included one commit (7,215,509 lines of code) and 52 commits (200,325,684 lines of code) from the elastic/elasticsearch repository. The single commit covers the release tag for v9.4.3. The 52 commits cover the latest patch release tag for every major and minor release from v6.0.1 to v9.4.3. Sourcerer searched the corpus as indexed in Elasticsearch, while ripgrep and grep searched the corpus as stored on a filesystem, reflecting their respective use cases. The regular expression patterns included one that appears rarely among the commits (DiskBBQ) and one that appears commonly among the commits (XContentType).

sourcerer.code.grep

The first search I benchmarked was a rare pattern for DiskBBQ that appears only in some commits:

.*[dD][iI][sS][kK][-_]?[bB][bB][qQ].*

Search latency (in seconds) spanning a single commit (605 matches found from 7,215,509 lines of code):

Retrieval method

Cache

p0

p50

p100

stdev

vs. sourcerer.code.grep

sourcerer.code.grep

Cold

0.069s

0.124s

0.167s

0.020s

-

sourcerer.code.grep

Warm

0.027s

0.029s

0.046s

0.005s

-

ripgrep

Cold

0.788s

0.800s

0.809s

0.005s

~6.5x slower

ripgrep

Warm

0.081s

0.088s

0.109s

0.009s

~3.0x slower

grep

Cold

3.565s

3.580s

3.821s

0.055s

~28.9x slower

grep

Warm

0.825s

0.827s

0.833s

0.002s

~28.5x slower

Search latency (in seconds) spanning 52 commits (1,041 matches found from 200,325,684 lines of code):

Retrieval method

Cache

p0

p50

p100

stdev

vs. sourcerer.code.grep

sourcerer.code.grep

Cold

0.159s

0.164s

0.270s

0.028s

-

sourcerer.code.grep

Warm

0.027s

0.031s

0.053s

0.006s

-

ripgrep

Cold

22.356s

22.364s

22.459s

0.026s

~136.4x slower

ripgrep

Warm

16.017s

16.123s

16.297s

0.058s

~520.1x slower

grep

Cold

101.962s

102.386s

104.281s

0.752s

~624.3x slower

grep

Warm

73.082s

73.507s

74.915s

0.536s

~2,371.2x slower

Table 2. p0/p50/p100/stdev retrieval speeds of sourcerer.code.grep, ripgrep , and grep, under cold and warm caches, at two corpus scopes (20 runs per method per cache state; three warmup runs discarded before each warm-cache measurement). All percentiles computed via linear interpolation. Ratios are computed against sourcerer.code.grep's p50 at the matching cache state. See the “Methodology” section for cache definitions and query/command syntax.

The wildcard field type indexes trigrams of each value and uses them as a filter to narrow the candidate set for a regular expression before verifying full matches. For a selective pattern like this one (605 matches out of 7.2 million lines, 1041 out of 200 million) this approaches sublinear time complexity relative to corpus size. ripgrep and grep both perform scans with linear time complexity, with ripgrep using multithreading and single instruction, multiple data–accelerated (SIMD-accelerated) literal prefiltering to speed up searches, but neither has a mechanism to skip the vast majority of a corpus the way that an indexed trigram search can.

This shows up starkly in how each approach scales. Going from the single-commit corpus to the all-commits corpus is a 27.8x increase in line count. sourcerer.code.grep warm-cache time barely moves from 0.029 seconds to 0.031 seconds. ripgrep's warm-cache time goes from 0.089 seconds to 16.1 seconds, a 183x change; and grep's goes from 0.83 seconds to 1.2 minutes, an 89x change. Both filesystem tools scale worse than linearly with corpus size on this hardware, while the indexed approach is nearly flat.

The second search I benchmarked was a common pattern for XContentType that appears in all commits:

.*[xX][cC][oO][nN][tT][eE][nN][tT][tT][yY][pP][eE].*

Search latency (in seconds) spanning a single commit (7,999 matches found from 7,215,509 lines of code):

Retrieval method

Cache

p0

p50

p100

stdev

vs. sourcerer.code.grep

sourcerer.code.grep

Cold

0.158s

0.172s

0.224s

0.015s

-

sourcerer.code.grep

Warm

0.055s

0.057s

0.140s

0.023s

-

ripgrep

Cold

0.794s

0.804s

0.824s

0.007s

~4.7x slower

ripgrep

Warm

0.093s

0.095s

0.096s

0.001s

~1.7x slower

grep

Cold

3.385s

3.399s

3.421s

0.010s

~19.8x slower

grep

Warm

0.681s

0.683s

0.686s

0.001s

~12.1x slower

Search latency (in seconds) spanning 52 commits (290,662 matches found from 200,325,684 lines of code):

Retrieval method

Cache

p0

p50

p100

stdev

vs. sourcerer.code.grep

sourcerer.code.grep

Cold

1.587s

1.644s

1.756s

0.047s

-

sourcerer.code.grep

Warm

1.480s

1.541s

1.627s

0.040s

-

ripgrep

Cold

22.426s

22.440s

22.548s

0.028s

~13.6x slower

ripgrep

Warm

16.021s

16.276s

16.498s

0.119s

~10.6x slower

grep

Cold

97.699s

97.922s

99.150s

0.404s

~59.6x slower

grep

Warm

68.883s

69.184s

70.374s

0.360s

~44.9x slower

Table 3. p0/p50/p100/stdev retrieval speeds for the pattern .*[xX][cC][oO][nN][tT][eE][nN][tT][tT][yY][pP][eE].* under the same conditions as Table 2.

The relative search latencies of sourcerer.code.grep compared to grep shows why the pattern's match count matters as much as the corpus size:

Corpus

Corpus size

Cache

Speed of sourcerer.code.grep with a rare pattern (DiskBBQ)

Speed of sourcerer.code.grep with a common pattern (XContentType)

1 commit

7,215,509 lines

Cold

~28.9x faster

~19.8x faster

1 commit

7,215,509 lines

Warm

~28.5x faster

~12.1x faster

52 commits

200,325,684 lines

Cold

~624.3x faster

~59.6x faster

52 commits

200,325,684 lines

Warm

~2,371.2x faster

~44.9x faster

Table 4. This table shows how much faster sourcerer.code.grep was compared to grep when searching across two different corpus sizes and two different pattern rarities.

The pattern with far more matches shows a dramatically smaller Elasticsearch advantage, most strikingly at all-commits scope, where the advantage drops from 2,371x to 45x. The reason is visible in the absolute numbers: sourcerer.code.grep's warm-cache time at all-commits scope jumps from 0.031 seconds (DiskBBQ) to 1.541 seconds (XContentType), a 49.7x increase for a 279x increase in match count, while grep's warm-cache time barely changes (73.5 seconds to 69.2 seconds, effectively flat, since it scans the same number of bytes regardless of how many of them match). The wildcard field's trigram index has sublinear-in-corpus-size behavior that comes specifically from narrowing the candidate set before verification. Once a pattern matches hundreds of thousands of lines, the bottleneck shifts from narrowing candidates to collecting and serializing all of them, a cost that scales with match count rather than corpus size. sourcerer.code.grep still wins by a wide margin even in this less favorable case, but the margin depends heavily on how selective the search is, not just how large the corpus is.

sourcerer.code.search

Sourcerer's other retrieval tool, sourcerer.code.search, performs a BM25-ranked MATCH query rather than an exact-substring regex match. Its speed on both patterns is included below for reference.

Pattern

Corpus scope

p0

p50

p100

stdev

Matches found

DiskBBQ

One commit

0.013s

0.017s

0.018s

0.002s

288

DiskBBQ

52 commits

0.014s

0.020s

0.046s

0.007s

493

XContentType

One commit

0.062s

0.063s

0.154s

0.020s

7,177

XContentType

52 commits

2.203s

2.359s

2.611s

0.125s

249,737

Table 5. p0/p50/p100/stdev retrieval speeds of sourcerer.code.search, warm cache only, for both patterns at both corpus scopes (20 runs per row; cold-cache figures omitted; see Methodology). Match counts are sourcerer.code.search's own, not the regex-based methods' BM25 matches on tokens rather than substrings, so these aren’t directly comparable to Tables 2–4.

What drives the speed advantage

sourcerer.code.grep outperformed ripgrep and grep at every corpus scale and pattern rarity tested, along with every cache state tested. But it wasn’t by a fixed margin. The advantage ranged from ~3–30x on a common pattern to over 2,300x on a rare pattern, because indexed and brute-force search respond to different things. The trigram narrowing of the wildcard field does less work as a pattern gets more selective, while grep and ripgrep do the same amount of work regardless of how much of the corpus happens to match. sourcerer.code.search is a third option for the cases where the exact string isn't known at all.

Figure 2: Sourcerer's sourcerer.code.grep tool outperformed ripgrep and grep in every combination of corpus size and pattern rarity benchmarked, sometimes by multiple orders of magnitude. Its outperformance was strongest when searching rare patterns in large corpora and weakest when searching common patterns in small corpora.

These search speed benchmarks show what scalable means in practice for an enterprise code search agent. It's being able to search the histories of any number of repositories at interactive speeds, just like how a developer coding agent searches the working state of a single repository on a filesystem.

Speed benchmark methodology

Appendix B explains the configuration of this benchmark. Note that while the Elasticsearch deployment had two data nodes each with the same specs as the virtual machine used for ripgrep and grep, the benchmark consisted of one index with one primary shard and one replica shard. Each search ran on a single shard, which means that sourcerer.code.grep had the same amount of vCPUs and memory available per search as ripgrep and grep, despite having twice as much capacity across the overall deployment. Having two data nodes actually incurred a slight latency penalty compared to just one data node. For brevity, I've omitted results from the benchmark with one data node. We don't recommend single-node deployments in production, so it's worth including the realistic latency overhead that comes with a multi-node deployment in this benchmark.

I compared all retrieval methods under both cold and warm caches, 20 cold runs and 20 warm runs per method. The definitions of cold and warm caches weren’t like-for-like between the Elasticsearch and filesystem benchmarks. For ripgrep and grep, a cold cache meant performing a full-page cache drop (sync; echo 3 > /proc/sys/vm/drop_caches) immediately before each cold run; and a warm cache meant three discarded warmup executions immediately followed by the 20 measured runs, with no cache drops in between. For ES|QL, a cold cache meant calling POST /_cache/clear before each run. This only clears Elasticsearch's internal caches, not the OS page caches of the data nodes, which can't be cleared by hand on Elastic Cloud Hosted (ECH). A warm cache for ES|QL meant three discarded warmup queries immediately before the 20 measured runs, to help ensure that the caches on both data nodes would be warm. sourcerer.code.search's cold-cache figures are omitted from Table 5 for the same reason discussed elsewhere in this post: Its cold-cache measurements came back statistically indistinguishable from its own warm-cache measurements, evidence that the OS-level cache-clearing limitation affects it more than it affects sourcerer.code.grep, which showed a consistent, physically sensible cold/warm gap throughout.

Code search indexing throughput

I didn't conduct a formal benchmark of indexing throughout. I'll share my general observations instead.

Typically, I see a sustained indexing throughput of 20K–25K lines per second on data nodes that each have ~16GiB RAM and ~8 vCPU on c4a-highcpu instances on Google Cloud Platform (GCP). That includes writing to a primary shard and its replica shard. I've seen throughput as high as ~60K lines per second on Elastic Cloud Serverless with Search Power set to "Performant."

An engineering team at Elastic compared Sourcerer's indexing throughput to semantic code search implementations that used either sparse vector generation (Elastic Learned Sparse EncodeR [ELSER]) or dense vector embedding generation (Jina). Sourcerer indexed ~25x faster than .elser-2-elastic and ~15x faster than .jina-embeddings-v5-text-small, while retrieval quality was similar among all of them. More concretely, what took ~6 hours to index with ELSER took ~14 minutes to index with Sourcerer.

Code search solution design and rationale

The remainder of this blog post explains the rationale for my design decisions of Sourcerer, giving expert insights for practitioners of Elasticsearch and generative AI (GenAI).

Goals

Ultimately, we want an agent that answers questions about deployed software and its supporting infrastructure by searching the primary sources of truth (the code itself) and generating verifiable responses that cite those sources so they can be trusted. Inspired by the success of coding agents with grep, my main functional goal for Sourcerer was to reproduce the search behavior of a coding agent and generate responses with citations, all using Agent Builder. My nonfunctional goals were to keep it fast and scalable, as well as accurate, when searching across many versioned repositories, while maintaining acceptable costs and ease of use. Of these goals, reproducing the search behaviors of coding agents would be the most consequential, as it would dictate the access pattern, index design, and query design, plus their effects on nonfunctional goals.

Access pattern

From a human perspective, the intended access pattern is simple: We expect to ask natural language questions about software and receive plain language answers grounded in the source of truth. From the perspective of the agent handling those questions, the intended access pattern is to reproduce the search behavior of coding agents to find what it needs. The LLMs used by coding agents are heavily trained to explore code with shell commands, likelsor find, grep or ripgrep, cat, head, tail, and so on. Claude Code's built-in tools, such as Glob and Grep, provide similar functions.

I chose to go with the grain of how models are trained. So my intended access pattern for Sourcerer was to reproduce the names, inputs, and outputs of shell commands as ES|QL tools in Agent Builder. That way, the Agent Builder harness would allow an LLM to use its trained intuition to achieve similar results as a frontier harness, like Claude Code or Codex, without having to fill the LLM's limited context window with instructions for using a different search interface. This was the main context engineering problem to solve with Sourcerer. Solving it would enable faster searches that scale across many repositories at once, allowing an agent to answer questions about deployments in which many different versioned software projects work together.

Index design

I decided on three index templates to fulfill this access pattern:

• sourcerer-refs: Each document indexes the high-level metadata for a single Git reference or ref identified by its unique commit hash, which can have a tag name or branch name associated with it. A ref represents an entire snapshot of a repository at a point in time. This is a small index. The agent mainly uses this to discover the repositories and snapshots that are available to search.

• sourcerer-files: Each document indexes the metadata for a single file of a given ref. This is a larger index. The agent mainly uses this to navigate files and directories using ls semantics.

• sourcerer-lines: Each document indexes the contents of a single, numbered line of code for a given file of a given ref. Yes, every line of code becomes a document. This is the largest index (but perhaps not as large as you might expect). The agent mainly uses this to search and view code using grep and cat semantics.

The design is almost entirely denormalized. Each index has the same namespacing fields for fast, joinless filtering. Each indexed ref stores all files and lines from its commit snapshot, rather than storing diffs and reconstructing them at search time or storing unique files and lines with a mutable array of ref names associated with each. These choices trade duplicative storage (the cheapest compute resource) for faster searches and less segment merging pressure.

Namespacing

All three indices use four fields to namespace the ref, file, or line of code:

• git.host: A Git hosting provider (for example, github, gitlab).

• git.org: An account name (for example, elastic).

• git.repo: A Git repository (for example, elasticsearch, kibana).

• git.commit: A commit hash, stored as the full 40-character SHA-1 digest for integrity.

The document _id hashes for files and lines are also namespaced by {git.host}, {git.org}, {git.repo}, and {git.commit} to allow for idempotent indexing. That means you can safely rerun an indexing job without duplicating any documents.

Likewise, the index names are namespaced with the same semantics, using tildes (~) as a reliable separator since it's a disallowed character in Git repository names and organization names:

• sourcerer-v*-files~{git.host}~{git.org}~{git.repo}

• sourcerer-v*-lines~{git.host}~{git.org}~{git.repo}

This namespace convention has many benefits:

• Agents can quickly narrow the search space for refs and files, along with lines of code, by these common scoping fields, keeping searches fast and focused.

• The semantics reflect common permission boundaries. You can reproduce the access policies of your Git hosting provider by implementing your choice of index-level security and/or document-level security based on host or organization or based on repository.

• You can instantly delete indices for a whole repository or organization, or for a host, without an expensive _delete_by_query.

• The index names are future proofed for different levels of granularity. Sourcerer might eventually allow indexing code by {git.host}, {git.host}~{git.org}, or {git.host}~{git.org}~{git.repo}~{git.commit} for selective shard sizing optimizations. The query syntax would be unaffected because they target index aliases (sourcerer-files and sourcerer-lines), not individual indices.

Settings

Three index settings help to optimize storage costs and search speed:

• Index sorting gives faster searches and better compression at the cost of reduced indexing throughput. Each index sorts documents on disk by git.host, git.org, git.repo, git.commit. File and line documents are further sorted by file.path, and line documents are further sorted by line.number.

• Synthetic _source discards _source and instead reconstructs it as needed when reindexing. None of the queries access _source, which makes it dead weight. Enabling this setting reclaims ~50% storage space in the files and lines indices. While it requires an Enterprise license, enabling it on a non-licensed deployment won’t prevent the index from being created; instead the setting will be ignored.

• best_compression is a fallback for Elastic deployments that lack an Enterprise license to use synthetic _source. It provides decent compression for _source (~11% storage savings by my observations) in exchange for a modest tax on indexing throughput (~15% slower), while search speeds are essentially unaffected because the queries don't fetch _source.

I use index aliases to support zero-downtime upgrades when reindexing to a new schema.

Mappings

The indices mainly use keyword fields. They facilitate efficient filtering with basic wildcard support and aggregations, along with optimal storage usage and indexing throughput.

The line.content field is indexed both as a wildcard field and as a text field with tokenization and similarity settings tuned for code search. This gives agents the option to search code using familiar and effective grep-like regular expressions on the wildcard field or using the inverted index of the text field to return only the highest ranking matched lines to reduce token usage and increase search speed. Both options are remarkably fast and scalable, taking milliseconds to finish in most cases.

Shards

While shards are becoming less relevant with the rise of stateless platforms like Elastic Cloud Serverless, I want the solution to accommodate all deployment modes of Elasticsearch. So I've given attention to the effect of the index design on the number and size of primary shards. Based on my observations, I expect that most users won’t need to give much attention to shards.

By default, Elasticsearch enforces a soft limit of 1,000 shards per data node (including replicas). That means this solution will hit a soft limit of just under 250 repositories indexed per data node, because each repository is written to a files index and a lines index, each with one primary shard and one replica. Additionally, there’s a conventional best practice of limiting shard sizes to ~50GB, which affects how many refs you can index per repository. Both of these limits can be pushed a bit. But they reveal that this solution design really is optimized for its intended use case of searching the commit snapshots of supported, deployed software. You wouldn't use Sourcerer to index every repository on the Internet, and you shouldn't use it to index every ephemeral development branch. Plus, you should decide how many refs are worth retaining for each repository.

Repository-level granularity of indices appears to strike the right balance of shard counts and shard sizes. For reference, Kibana is one of the largest repositories on GitHub (source). I observed its shard size to be a manageable 60GB–75GB when retaining only the latest patch release for every major and minor version release from v6.0.0 to v9.5.0. That's great coverage for the Elastic deployments we see in the wild. If that's one of the largest repositories out there, you can expect just about any other repository to fit in a single shard as long as you have a reasonable retention policy, which I discuss in the next section (“Pruning”).

Pruning

By default, Sourcerer retains everything you index. Pruning lets you delete old refs to prevent unbounded growth. You can define ref retention policies based on the age of refs and the number of refs indexed in the repo. You can also define these policies based on the number of semantic versions indexed in the repo at any level of granularity for major, minor, patch, build, and prerelease versions. Some common configurations are to retain only the latest commit of the default branch or the most recent patch release tag for every major and minor version release tag.

Elastic Agent Builder tools

With the index design in place, we can review the tools that query those indices.

ES|QL tools in Agent Builder are parameterized ES|QL queries with descriptions to guide the agent's use of them. Sourcerer has tools for several purposes: repo discovery, file discovery, code search, and code display. These tools reproduce the names, inputs, and outputs of shell commands that coding agents prefer to use when exploring code, making them intuitive enough for the LLM to use with minimal instructions passed into its context window.

Repo discovery

These are typically the first tools that the agent calls. Unlike most coding agents, which search within a single repository on a filesystem, Sourcerer is aware that its search space likely has multiple repositories and versions, and so its first step is to decide which repos and refs to scope its searches to.

• sourcerer.repos.list: Lists the repos that are available to search.

• sourcerer.repos.search: Lists the repos whose file contents best match a given query.

• sourcerer.refs.list: Lists the repos and refs that are available to search.

File discovery

All file discovery tools support glob matching (* and **) on file paths.

• sourcerer.files.ls: Lists files and directories that match a given pattern.

• sourcerer.files.tree: Lists files and directories that match a given pattern in a tree-like format.

• sourcerer.files.wc: Counts lines, words, characters, bytes, and longest lines for each matching file.

Code search

All file code search tools support glob matching (* and **) on file paths.

• sourcerer.code.grep: Searches lines of code using RLIKE on a wildcard field for rapid execution of regular expressions.

• sourcerer.code.search: Searches lines of code using MATCH on a text field that has been tuned for code search.

Code retrieval

All file code retrieval tools concatenate the desired lines of any matching file and return them as a single, contiguous block of code in grep -n format, which is a format preferred by coding agents, including Claude Code's built-in Grep tool. This lets the agent see a faithful representation of file contents with line-level attribution for precise citations, without requiring the agent to reconstruct the contents or infer line numbers through reasoning.

To illustrate, here's how the sourcerer.files.head tool formats the first five lines of Kibana's README.md file, reconstructed from five documents from the lines index:

All file code retrieval tools support glob matching (* and **) on file paths. Agents typically use these tools to display the contents of a single file.

• sourcerer.files.cat: Concatenates and displays all lines for each matching file.

• sourcerer.files.head: Concatenates and displays the first n lines for each matching file.

• sourcerer.files.tail: Concatenates and displays the last n lines for each matching file.

• sourcerer.files.read_lines: Concatenates and displays the range of lines between two given line numbers for each matching file.

Agent Builder skills

With the tools implemented, we can review the skills that guide the agent's proper use of them. Agents typically invoke the follow skills in this order:

• sourcerer-repo-discovery: Guides the agent in discovering and selecting the repositories that are available to search for a given prompt. While this is typically the first skill an agent invokes, the agent might return to it when tracing dependencies from other repositories or when answering questions that span multiple repositories or versions.

• sourcerer-ref-resolution: Guides the agent in resolving the names of tags or branches to their unique, immutable commit hashes. This lets the agent reliably filter its searches to a single commit snapshot.

• sourcerer-code-search: Guides the agent in exploring code, with basic best practices on when and how to use the available tools for maximum efficiency.

• sourcerer-code-citations: Guides the agent in citing files, directories, lines of code, and ranges of lines of code. Sourcerer auto-generates more specific citation skills for each major Git hosting provider, so that its citation links conform to the URL formats of each respective host.

Agent system prompt

The final packaging of the agent comes with a system prompt that succinctly describes the agent's role and its high-level instructions. The configure file that defines the system prompt also defines the tools and skills that are made available to the agent, so that it can only execute what we permit it to execute.

Sourcerer CLI

The Sourcerer CLI assists with setup and indexing, along with pruning to keep operations simple. Configuration is managed through a sourcerer.yml configuration file.

• sourcerer setup: Idempotently loads the index templates and Agent Builder configurations, in addition to Kibana dashboards. This is typically a one-time operation and takes a few seconds.

• sourcerer index: Checks for new refs that match patterns defined in sourcerer.yml and then idempotently indexes them, skipping any refs that have already been indexed or that qualify for pruning based on retention policies. It calls git to clone repos and to list remote refs, as well as to  check out refs.

• sourcerer prune: Checks the retention policies for any refs that qualify for pruning and then deletes them from all three indices using _delete_by_query.

You can easily schedule indexing and pruning using external schedulers, like cron. For our internal use at Elastic, we maintain sourcerer.yml files in a private Git repository and schedule indexing and pruning with GitHub Actions. I prefer to index code frequently, while pruning outside of normal working hours to prevent agents from suddenly losing context in the middle of a conversation.

Feature license summary

For transparency, here’s a summary of the license levels for all non–open source software (non-OSS) features referenced in this solution design.

Subscription features:

• Agent Builder (optional; you can query the indices from a different harness using the Elasticsearch API).

• Synthetic _source (optional).

• Document-level security (optional).

Free features, proprietary to Elastic (not OSS as defined by the Open Source Initiative [OSI]):

• ES|QL.

• wildcard field.

A subscription gives you the magic of "everything just works" with Agent Builder, along with resource optimizations, finer security permissions, and platform support. Without a subscription, you can still index and prune code with the Sourcerer CLI and search the code using the Elasticsearch API.

Conclusion

Sourcerer demonstrates that Elasticsearch + Agent Builder is an exceptional solution for agentic enterprise code intelligence, as evidenced in many ways:

• Accurate: Sourcerer's agentic code retrieval quality is at parity with frontier coding agents as demonstrated by its performance on the academic benchmark SWE-Explore.

• Fast: Sourcerer's query speed ranges from milliseconds in a single repository to multiple seconds across a billion lines of code. Indexing is also much faster than could be achieved with vector embedding generations.

• Scalable: Elasticsearch sharding enables horizontal scaling, making it possible to search the current and historical states of an entire enterprise software estate. Alternatively, the stateless architecture of Serverless naturally scales without having to plan shards.

• Resilient: Elasticsearch replication enables high availability to keep the agent operational 24/7. Likewise, the stateless architecture of Serverless naturally provides high availability.

• Polyglot: Sourcerer's indexing and retrieval methods are completely language-agnostic and tolerant of malformed code.

• Secure: Elasticsearch document-level security enforces access policies for humans and agents at the organization, repository, and commit levels.

• Efficient: Sourcerer demonstrates an efficient use of storage, memory, compute, and token consumption for its intended use case.

• Manageable: The Sourcerer CLI, and its use of native git commands and Elastic REST APIs, makes it easy to get started with and operate, as well as schedule.

• Universal: Organizations from all industries build software, and Git is the source of truth for ~85% of them (source). Sourcerer has value to all of these organizations. 

Sourcerer is currently less than two months old, and the benchmark results suggest that there’s room to improve recall and token efficiency, along with task duration. I'll continue to work on this project in the near future.

Try it yourself

Sourcerer depends on Elasticsearch and Kibana. You can get started with those in a couple ways:

• Elastic Cloud (includes an Enterprise trial).

• Local setup with Docker (includes an Enterprise trial).

Then you can get started with Sourcerer.

I built this with Agent Builder. What will you build?

References

Sen, S. (2026). Is Grep All You Need? How Agent Harnesses Reshape Agentic Search. arXiv. https://arxiv.org/abs/2605.15184

Zhang, S. (2026). SWE-Explore: Benchmarking How Coding Agents Explore Repositories. arXiv. https://arxiv.org/abs/2606.07297

Appendices

Appendix A. SWE-Explore benchmark configuration

Benchmark environment:

• Sourcerer version: v1.0.0 (commit hash: 26d2e84e3f9e4c1e1598a48532eba9a739465c3e)

• Elastic Cloud Hosted (ECH):

• Region: GCP - Los Angeles (us-west2)

• CPU Optimized hardware (c4a-highcpu)

• 3x Elasticsearch data nodes (each with 16GiB RAM, 8 vCPUs)

• 2x Kibana instances (each with 2GiB RAM)

• Elastic stack version: v9.5.0 (commit hash: dbedef007f580447413782705acb9afec41f945b)

• Claude Code version: 2.1.202

• LLM: GPT-5.4

Indices as reported by GET /_cat/indices:

Benchmark task prompt:

Table 6. This table compares Claude Code's task prompt used in the original paper and Sourcerer's task prompt used in this benchmark run. The highlights indicate where Sourcerer's task prompt differed from Claude Code's: red indicates an instruction from Claude Code's task prompt that wasn't used by Sourcerer's, and green indicates an instruction that was unique to Sourcerer's task prompt.

Appendix B. Code search speed and scalability benchmark configuration

Benchmark environment:

• Sourcerer version: v2.0.0 (commit hash: dcc373b3bf7b24168e00ed7b172aea7227f705f9)

• Elastic Cloud Hosted (ECH) for ES|QL:

• Region: GCP - Los Angeles (us-west2)

• CPU Optimized hardware (c4a-highcpu)

• 2x Elasticsearch data nodes (each with 16GiB RAM, 8 vCPUs)

• 1x Kibana instances (2GiB RAM)

• Elastic stack version: v9.5.0 (commit hash: 8d4246a64bc255212407b1b313fe402391299c88)

• Virtual machine for ripgrep and grep:

• Region: GCP - Los Angeles (us-west2-a)

• CPU Optimized hardware (c4a-highcpu-8-lssd)

• 16GiB RAM

• Image: projects/ubuntu-os-cloud/global/images/ubuntu-2404-noble-arm64-v20260717

• Provisioned IOPS: 3300

• Provisioned throughput: 215

Indices as reported by GET /_cat/indices:

Cluster settings adjusted to avoid truncating the returned match count on frequent patterns:

Regular expression used by ripgrep and grep:

• DiskBBQ: .*[dD][iI][sS][kK][-_]?[bB][bB][qQ].*

• XContentType: .*[xX][cC][oO][nN][tT][eE][nN][tT][tT][yY][pP][eE].*

ES|QL query syntax used for sourcerer.code.grep:

ES|QL query syntax used for sourcerer.code.search:

ES|QL query parameters used in all tests:

• git_host="github"

• git_org="elastic"

• git_repo="elasticsearch"

• n=1000000

ES|QL query parameters used for specific tests:

• Corpus with single commit: git_commit="45f6a06b1b441b41fe711059b8720013173e7c89"

• Corpus with all commits: git_commit="*"

• sourcerer.code.grep pattern for rare pattern (DiskBBQ): regex=".*[dD][iI][sS][kK][-_]?[bB][bB][qQ].*"

• sourcerer.code.grep pattern for common pattern (XContentType): regex=".*[xX][cC][oO][nN][tT][eE][nN][tT][tT][yY][pP][eE].*"

• sourcerer.code.search pattern for common pattern (DiskBBQ): q="DiskBBQ"

• sourcerer.code.search pattern for common pattern (XContentType): q="XContentType"

n was set high enough to recover the true, uncapped match count for each pattern (605 and 1,041 matches for DiskBBQ;  7,999 and 290,662 matches for XContentType), confirmed in each case by comparing hits_returned against ripgrep's and grep's own match counts on the same underlying data.

Command syntax used for ripgrep :

rg -n --no-ignore -j 6

Command syntax used for grep:

grep -E -r -n --binary-files=without-match --exclude-dir=.git

Directories of each cloned repository snapshot and the approximate total sizes of their nonbinary files tracked by Git, which is the search space of ripgrep and grep:

What you'll discover

In this post, you'll learn how to:

• Build judgment lists from click data to evaluate and improve relevance.

• Apply basic search tuning (field weights, boosts, query rules) informed by analytics.

• Create rank features from behavioral signals, like popularity and conversion rate.

• Understand LTR and how click data becomes training data.

• Close the feedback loop between analytics and relevance improvement.

What you'll need

• Search analytics data from the previous blogs in this series (search, click, and conversion spans in Elastic).

• An Elasticsearch index with product data (the reference project includes sample data with rank features).

• Familiarity with Elasticsearch queries (BM25 [Elasticsearch's default text scoring algorithm], rank_feature, function scores).

Search relevance techniques at a glance

Technique

Effort

What it improves

When to use

Field weight tuning

Low

Relevance for attribute-rich queries

First step; quick wins

Query rules

Low–medium

Specific high-value queries

Known bad results for specific terms

Rank features

Medium

Blending behavioral signals with text score

Popularity, conversion, freshness boosts

LTR

High

Systematic ranking from click data

When you have ≥5k labeled query-doc pairs

From search analytics to search relevance improvements

Over the past three posts, you built a full instrumentation pipeline, including search spans with search.* attributes and click tracking with position data and click-through rate (CTR) / Mean Reciprocal Rank (MRR) metrics. This pipeline also includes conversion spans tying searches to revenue. And it all sits in traces-generic.otel-default, queryable with Elasticsearch Query Language (ES|QL).

Now you have dashboards, and you know that your CTR is 28%. You also know which queries generate revenue and which ones users abandon. Plus, you can tell your product manager exactly where the funnel leaks.

Now what?

The real value of search analytics is using behavioral data to improve relevance and close feedback loops. It’s also important to make search learn from its users. Your experience tells you that measurement is necessary but not sufficient, and a dashboard that shows poor ranking doesn't improve that ranking. 

This post covers four practical areas:

1. Judgment lists: The foundation for evaluating and improving relevance.

2. Basic search tuning: Field weights, boosts, and query rules informed by analytics.

3. Rank features for personalization: Feeding behavioral signals back into ranking.

4. LTR: Training machine learning (ML) models on click data to optimize ranking automatically.

Each one draws directly from the search.* attributes you're already collecting, and they don’t require any new instrumentation.

What are judgment lists?

Before diving into specific techniques, it's worth understanding the concept that ties them all together: judgment lists.

A judgment list is a set of query-document pairs with relevance grades: For a given query, how relevant is each document? They look like this:

Query

Document

Grade

Label

"wireless headphones"

SKU-001 (Sony WH-1000XM5)

3

Highly relevant

"wireless headphones"

SKU-042 (AirPods Max)

2

Relevant

"wireless headphones"

SKU-099 (Wired earbuds)

0

Not relevant

Judgment lists serve three purposes:

• Evaluating current relevance. Given a set of queries and known-good results, how well does your search rank them? Metrics like Normalized Discounted Cumulative Gain (NDCG) and the Rank Eval API use judgment lists to score your ranking quality. This gives you a baseline before making changes.

• Measuring the impact of changes. When you adjust field weights or add synonyms, judgment lists tell you whether the change helped or hurt. The same is true when you change boosting rules. Run the same evaluation before and after: If NDCG went up, the change improved relevance for those queries.

• Training LTR models. LTR algorithms need labeled training data, such as, For this query, these documents are relevant and those aren't. Judgment lists are that training data.

Manual vs. automated judgment lists

Traditionally, judgment lists are created by human assessors who manually rate documents for a set of test queries. This works well for a small number of high-value queries (your top 50 searches, for example), but it doesn't scale. A large catalog with thousands of distinct queries and frequent inventory changes makes manual assessment impractical.

This is where your click data becomes valuable. Every click is an implicit relevance judgment; that is, a signal that for a given query, a given document was relevant enough to engage with. Conversions are even stronger signals. By aggregating this data, you can build judgment lists automatically from real user behavior, at a scale that manual assessment can't match.

The trade-off is noise. Clicks are influenced by position bias (users click higher-ranked results more often regardless of relevance) and presentation effects. They’re also influenced by accidental clicks. We'll cover techniques to handle this noise later in this post. But the key insight is that approaches like personalization and LTR lean toward automated judgment lists because it's not scalable to manually create lists for every query or user segment, nor for every inventory change.

The judgment lists guide on Search Labs covers the concept in depth, including how to structure lists for evaluation with the Rank Eval API.

How search analytics identify search relevance problems

The most immediate use of your analytics data is to identify and fix specific relevance problems. This only requires using data to direct manual improvements, no personalization or ML.

Finding problem queries with search analytics

The ES|QL queries from the click in your application give you per-query CTR and MRR. Sort by search volume descending and CTR ascending to find your highest-impact relevance failures:

This is the CTR-by-query query re-sorted to surface problems first. The WHERE searches > 5 filter removes one-off queries that would dominate the low-CTR list with small sample noise.

• Zero-CTR queries with results.  Your ranking is returning content, but none of it’s compelling. These queries often benefit from synonym expansion. You can also improve them with boosting rules or pinned results.

• Low-CTR, high-volume queries. These are the biggest relevance investment opportunities, and they affect the most users.

• Low-MRR, high-CTR queries. Users find what they need but have to scroll for it. In these instances, the relevant documents exist, but they're ranked wrong.

Try it: Close the loop in the reference project

If you have click data in the reference project(python generate_traffic.py --blog 3 --sessions 100), you can run the problem-query query above against traces-generic.otel-default and observe your lowest-CTR queries.

The reference project's app.py already uses rank_feature boosting: rank_features.popularity, rank_features.conversion_rate, rank_features.margin_score, and rank_features.freshness are indexed on every product and blended into the BM25 score at query time. To see the effect of adjusting a boost:

1. Open reference/app.py, and find the "should" clause in _build_search_query().

2. Change the "boost" value on rank_features.popularity from 2 to 5.

3. Restart the server (python app.py) and rerun generate_traffic.py --blog 3 --sessions 50.

4. Compare the top-queries and click position distribution in ES|QL before and after.

This is informed manual tuning using the behavioral data you collected, not ML. That's the pattern for the rest of this post: First, ES|QL tells you what's wrong. Then, configuration changes and, eventually, LTR models fix it.

Field weights, boosts, and scoring for search relevance

The simplest tuning lever in Elasticsearch is adjusting how different fields contribute to the relevance score. A multi_match query across title, description, and brand fields can weight the title higher because a match there is usually more relevant. Your analytics data tells you where these weights are wrong: If users consistently click products with titles that don't match the query but with descriptions that do, your title boost may be too aggressive.

Beyond field weights, you can incorporate business metrics directly into scoring. A common example is boosting by profit margin; that is, products with higher margins rank slightly higher when relevance scores are similar. The rank_feature field type is designed for exactly this: Index a numeric signal (margin, popularity, recency) alongside each document, and the rank_feature query blends it with text relevance at query time. For more complex scoring combinations, like decay functions, weighted field values, and scripts, the function_score query gives you full control. The boosting by profit and popularity guide on Search Labs walks through this in detail, including the trade-offs between additive and multiplicative boosting approaches.

Query rules for targeted search relevance fixes

For targeted interventions, query rules let you pin, boost, or exclude specific results for specific queries, and no model training is required. They're the search equivalent of a manual override.

Your analytics data tells you exactly where to apply them. When you see a zero-CTR query, like "returns policy", that consistently returns product results instead of the returns page, you can pin the returns page at position 1 for that query. If you notice a high-revenue query where the best-selling product appears at position 4, you can boost it.

Query rules are valuable precisely because they're simple. They solve known problems immediately while you build toward more sophisticated approaches. The query rules tutorial on Search Labs walks through the setup.

Building judgment lists from click data

The basic tuning above handles individual problems. To improve relevance systematically, you need judgment lists, and your click data can build them automatically.

Converting click data into relevance grades

The simplest approach is to count clicks per query-document pair and assign graded relevance:

This gives you a table of (query, document, click_count, avg_position) tuples. The grading step maps click counts to relevance levels:

Click count

Suggested grade

Label

0

0

Not relevant (never clicked for this query)

1

1

Marginally relevant

2–3

2

Relevant

4+

3

Highly relevant

The thresholds depend on your traffic volume. For a high-traffic site, you might need 10+ clicks before calling something "highly relevant." For lower traffic, even two or three clicks is a meaningful signal. The point is that click frequency across users is a stronger signal than any single click.

You can strengthen judgments further by incorporating conversion data, a document that gets clicked and added to cart is a stronger relevance signal than one that gets clicked and abandoned:

A document with five clicks and three cart additions is a stronger candidate for grade 3 than one with five clicks and zero cart additions.

Handling position bias in click data

There's a problem with raw click counts: position bias. Users click position 1 more often because they see it first, not necessarily because it's the most relevant result. Blog 3 introduced this concept when we discussed click position distribution and referenced the foundational work by Joachims et al. (2005).

Position bias matters for judgment lists because it means raw click data over-weights whatever the current ranking happens to surface first. If you train an LTR model on biased judgments, it learns to replicate the existing ranking, which defeats the purpose.

Two practical approaches to handle this:

• Position-normalized click rates. Instead of raw click counts, calculate click-through rate per position. A document clicked 3 out of 10 times when shown at position 5 is arguably more relevant than one clicked 5 out of 10 times at position 1. Position 5 gets less visibility, so a higher click rate there is a stronger relevance signal.

• Skip-above heuristics. When a user clicks position 3 but skips positions 1 and 2, those skipped documents are implicitly judged "less relevant" for that query. This is the core insight from Joachims et al.: Skipped-above results provide negative training signals. You can extract these pairs from your click data:

Queries where the minimum click position is greater than 1 are sessions where the user skipped the top result(s). The documents at those skipped positions, for those queries, are candidates for grade 0 in your judgment list, because the user saw them and chose something lower.

For production judgment list generation, the LTR tutorial on Search Labs walks through the complete pipeline. The training LTR models with user behavior data guide covers the specific workflow of deriving training data from click-through behavior, including the Clicks Over Expected Clicks (COEC) algorithm for debiasing. And the LTR Jupyter notebooks provide working Python code for feature extraction and model training.

Rank features for search relevance personalization

Basic tuning applies the same ranking to every user. Personalization means adapting results based on who’s searching (their history, preferences, or segment). Rank features are the most accessible way to do this in Elasticsearch.

Building behavioral signals from click streams

You can aggregate your click and conversion data at the document level to produce signals that reflect overall or segment-specific popularity:

These document-level signals (click popularity, cart rate, and conversion rate) are indexed as rank_feature fields on each product document. The workflow:

1. Run the ES|QL query above periodically (daily or weekly).

2. Write the results back to a click_popularity or conversion_rate rank_feature field on each product document.

3. Use a rank_feature query to blend text relevance with behavioral signals.

The rank_feature query applies a saturation function by default: Initial popularity gains matter most, diminishing as values get large. This prevents a single viral product from dominating all queries. You can tune the function's pivot point to control how much influence the feature has relative to text relevance.

Personalizing rank features by user segment

The query above produces global popularity; that is, it’s the same for every user. Personalization comes from segmenting these signals. If you're tracking enduser.pseudo.id or user.id, you can compute features per user cohort:

• Category affinity: How often does this user click products in "electronics" versus "clothing"?

• Price sensitivity: Does this user tend to click and convert on higher- or lower-priced items?

• Brand preference: Which brands does this user engage with most?

These become additional rank features, applied at query time based on who's searching. The personalized search with LTR guide on Search Labs walks through training per-user ranking models. For a lighter approach, the cohort-aware ranking guide shows how to use multiplicative boosting to personalize at the segment level without ML, just analytics-derived weights applied to rank features at query time.

Combining multiple signals with the linear retriever

When you're blending text relevance with semantic search and behavioral rank features, you need a way to combine them. Elasticsearch's linear retriever gives you precise control over how different query types contribute to the final ranking. It computes a weighted sum of normalized scores, so you can say text relevance matters 60%, semantic similarity 30%, popularity 10% and adjust those weights based on your analytics. This is unlike Reciprocal Rank Fusion (RRF), which only considers relative rank positions.

This is particularly useful for personalization because you can vary the weights per user segment. A returning customer might get more weight on purchase history, while a first-time visitor gets more weight on global popularity.

If you're also using semantic search with embedding models, the Elastic Inference Service (EIS) now offers GPU-accelerated embedding generation, including multilingual models, directly within Elastic Cloud, making it straightforward to add a semantic retriever alongside your text and behavioral signals.

How Learn To Rank uses click streams to optimize ranking

Rank features handle individual signals, but LTR handles all of them at once. It trains an ML model to combine features (like text relevance, popularity, CTR, recency, margin, or user affinity) into a single ranking function optimized for your users.

What you need for Learn To Rank in Elasticsearch

Elasticsearch has supported native Learning to Rank since version 8.12 as an Enterprise subscription feature. The pipeline:

1. Judgment list. Query-document pairs with relevance grades (built from click data, as above).

2. Feature extraction. Numeric signals for each query-document pair (BM25 score, popularity, CTR, recency, price, margin, user segment features).

3. Model training. Typically XGBoost or LambdaMART, trained on your judgment list with features.

4. Deployment. Upload the trained model to Elasticsearch via Eland, and use it as a rescorer.

The click and conversion data from this series feeds steps 1 and 2, and the judgment list is your training labels. The document-level features from the rank features section above (click popularity, conversion rate, margin) are additional features alongside text relevance scores.

Why automated judgment lists from click data scale better

This is where the scalability argument for automated judgment lists becomes concrete. A manually curated judgment list might cover your top 100 queries well, but personalized ranking needs judgment data across thousands of queries and multiple user segments. You can't hire assessors to rate results for "wireless headphones" separately for electronics enthusiasts, budget shoppers, and professional audio engineers.

Automated judgment lists from click data scale to every query your users actually run and update as inventory and user behavior change. They can also be segmented by cohort. The training LTR models with user behavior data guide demonstrates this end-to-end workflow, showing how to go from raw click events to trained ranking models.

For teams looking to close this loop even further, the agentic autotuning approach demonstrates using an AI agent to continuously monitor search quality and generate judgment lists from user interactions. It automatically retrains LTR models, turning the feedback loop into an autonomous system.

What Learn To Rank requires: Traffic, pipeline, and evaluation

LTR requires enough traffic to generate meaningful judgment lists and engineering time to build and maintain the pipeline. It also requires ongoing evaluation to ensure that the model improves over time. But for search applications with sufficient volume, it's the most effective way to make ranking learn from user behavior. The LTR introduction on Search Labs covers the full scope of what's involved.

Elasticsearch Relevance Studio: Visual search relevance tuning

Between query rules (manual, targeted) and LTR (ML, systemic), there's a middle ground: visual relevance tuning. Elasticsearch Relevance Studio is a tool for comparing and tuning search configurations side by side. It lets you adjust boost values, field weights, and query structures while seeing the results update in real time.

Analytics data, especially CTR and MRR, tells you which queries to focus on. Start from a ranked list of problem queries (the low-CTR, high-volume queries from the ES|QL analysis above), and work through them systematically in Relevance Studio, instead of guessing which searches need tuning.

The workflow:

1. Identify problem queries. Run the CTR-by-query and MRR-by-query analyses.

2. Open those queries in Relevance Studio. See current results alongside the tuned version.

3. Adjust field weights and boosting. Experiment with configuration changes.

4. Evaluate with judgment lists. Use the Rank Eval API to confirm that the change improves NDCG for your test queries.

5. Measure the impact in production. Rerun the analytics after deploying changes, and compare CTR / MRR.

This before-and-after loop is where analytics and tuning connect. With the data analytics you can prioritize the queries that affect the most users and verify that changes actually helped. Without this data, tuning is guesswork, and you're adjusting weights without knowing which queries matter or how to measure improvement. 

If you are looking at tuning and evaluating relevance you should also investigate the Elasticsearch Relevance Studio project which lets you directly compare search strategies. 

Closing the loop between search analytics and search relevance

Search relevance improvement follows a consistent loop: measure quality with ES|QL analytics, apply changes (query rules, field weights, rank features or LTR models), then measure again.

The key insight is that the same instrumentation that measures search quality also generates the data to improve it. Your search spans produce the queries to analyze, and your click spans produce the judgment data for evaluation and LTR training. Plus, your conversion spans tell you which improvements matter most to the business.

Measuring search relevance improvements after changes

After deploying a change, like a new query rule or updated field weights, or deploying an LTR model, you need to know whether it helped:

Metric

What it measures

How to compare

CTR trend

Whether more users are clicking results after the change

Run the CTR-by-query ES|QL query for the week before and after; compare per-query percentages

MRR trend

Whether clicks are shifting toward higher-ranked positions

Compare mean reciprocal rank per query before and after using the MRR-by-query analysis

Conversion rate trend

Whether more clicks are turning into purchases or cart additions

Run the click-to-conversion ES|QL query for both periods; compare cart rate percentages

Revenue per query

Whether the business impact of the change is positive

Compare revenue attributed to affected queries via conversion spans before and after deployment

Run the same ES|QL queries before and after. If your change was a query rule for "laptop bag", compare that query's CTR and MRR from the week before to the week after.

A/B testing with experiment attributes

The feature_flag.key attribute  is designed for exactly this. Route a percentage of traffic to a new ranking configuration, and set feature_flag.key to the experiment name and optionally feature_flag.result.variant to the variant (for example, "control" or "variant-boost-v2"). Then propagate it to click spans the same way you propagate `query_id`, and compare metrics between groups:

This gives you an A/B comparison of CTR by experiment variant. You don’t need a separate experimentation platform for basic comparisons, although you’ll want a proper framework for statistical rigor on sample sizes and significance.

Monitoring for search relevance regressions

Once you've identified your high-value queries, that is, the ones that drive revenue and have been tuned for good engagement, you need to protect them. Relevance regressions on your top 20 revenue-generating queries are business-critical incidents.

This is where monitoring comes in. In the next blog in the series, we'll cover setting up alerts on these metrics: CTR drops on high-revenue queries and MRR regressions after deployments. We’ll also cover conversion rate anomalies. The same ES|QL queries that power your analytics dashboards can drive alerting rules; the feedback loop covers measurement and improvement. It also provides operational protection.

Search relevance improvement roadmap: Week 1 to Quarter 1

Here's a practical starting point. You don't need to implement LTR on day one. The approaches build on each other:

• Week 1: Evaluate and fix known problems. Run the CTR-by-query analysis. Find your zero-CTR queries with high search volume, and fix the worst ones with synonyms or pinned results via query rules. You could also use adjusted field weights. Use judgment lists (even a small manual one for your top queries) to confirm that the changes improve NDCG before deploying. You should get immediate, targeted impact.

• Month 1: Add behavioral rank features. Extract document-level click popularity and conversion rates from your analytics. Index them as rank_feature fields, and blend behavioral signals with text relevance using the linear retriever or `rank_feature` queries. Then consider simple business boosts like margin. This lifts baseline quality across all queries without manual intervention per query.

• Quarter 1: Automate with LTR. Once you have enough click data (typically several weeks of production traffic), build judgment lists automatically from click and conversion data. Train an LTR model that combines text features, behavioral features, and business features, and then deploy it as a rescorer. The ranking now learns from user behavior and improves as you collect more data.

At each stage, measure the impact with the same metrics. CTR and MRR are your scorecards, as is conversion rate. If a change doesn't move them, it didn't help, regardless of how sophisticated the approach.

What's next: Search reliability engineering and SLOs

We've covered the full sequence: instrument search, measure quality, track conversions, and improve relevance. The missing piece is making sure it all keeps working.

Next, we close the series with search reliability engineering, including Service Level Objectives (SLOs) for search quality and alerting on metric regressions, along with operational dashboards that catch problems before users notice them. The same metrics you've been building become the basis for search health monitoring, turning your analytics pipeline into a reliability system.

Get started

Working code

• Reference project: Working code for the entire blog series; clone, configure, and run.

What is metric temporality in OpenTelemetry?

Metrics stores usually receive client-side, pre-aggregated metrics. For example, if an application records request response times, it won’t send each individual response time as a single data point to your metrics back end. Instead, the application (or rather the OTel SDK) pre-aggregates those raw response times into counters or histograms. These pre-aggregated values are then exported at a periodic interval, dramatically reducing the number of data points. Temporality is about how this pre-aggregation works. There are two temporality models: cumulative and delta.

Cumulative temporality in OTel metrics

With cumulative temporality, each data point represents the totalamount of change in the metric value since the process started. Values monotonically increase, with occasional reset to 0 (for example, when the process restarts).

Take a counter tracking the total CPU time consumed by a Java Virtual Machine (JVM):

Timestamp

Value

Meaning

10:01

12.4s

12.4s total CPU time since start

10:02

13.1s

13.1s total CPU time since start

10:03

13.9s

13.9s total CPU time since start

To compute the rate of change between 10:01 and 10:02, we subtract: 13.1 - 12.4 = 0.7s of CPU time was consumed in that interval. Dividing by the time range of the interval gives us the rate. This is the default temporality for counters in both Prometheus and OTel.

Delta temporality in OTel metrics

With delta temporality, each data point represents the change since the last measurement. Values are independent of each other. In other words, after each export, the OTel SDK resets all values for all series.

The same raw observations from the cumulative example above would look as follows with delta temporality.

Timestamp

Value

Meaning

10:01

0.5s

0.5s of CPU time in this interval

10:02

0.7s

0.7s of CPU time in this interval

10:03

0.8s

0.8s of CPU time in this interval

To compute the rate or increase, we can use the value directly, without any subtraction.

Trade-offs between cumulative and delta OpenTelemetry metrics

Both temporalities have practical trade-offs:

• Resilience to data loss: Cumulative counters are self-describing: If you miss an export, the next data point still gives you the correct total. Delta values are incremental, so a lost data point means that the corresponding increase is lost.

• Metric producer memory footprint: For cumulative temporality, the OTel SDKs need to keep a state for every series in memory. For delta temporality, the footprint is much lower. There, the SDKs only need to keep track of counters or histograms which changed since the last export. If there are a lot of counters or histograms and many of them don’t increase each period, this difference can be quite substantial.

• Aggregation across restarts: Cumulative counters require reset detection logic, which in edge cases can fail: If the metric value decreases, it’s detected as a reset. We assume that the application was restarted and the counter started from 0 again. This can be missed if the first reported counter value after the restart is higher than before the restart. A concrete example:

• The service consumes 1 second CPU time and restarts.

• After the restart, the service performs a CPU-intensive task and consumes 2 seconds of CPU time before the metric is exported again.

• The metric back end just sees 1 followed by 2 as the metric value. It never observes a decrease and therefore misses the reset.

Delta values don't have this problem since each value is independent.

If you’re using histograms, the trade-offs have an even bigger effect:

Trade-off

Cumulative

Delta

Histogram size

Buckets accumulate across exports, consuming more storage

Buckets reset each export, producing smaller histograms

Min/max accuracy

Approximated from buckets for custom time ranges (tracked values represent extremes since process start)

Exact per-export minimum and maximum values

Query performance

Faster: only the first and last value in a time range plus resets are needed

Slower: all histograms in the queried range must be combined

OpenTelemetry supports both models and lets you choose per SDK via the OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE environment variable.

Why native temporality support eliminates OTel Collector workarounds

Prometheus and most other metrics back ends pick a side: All metrics have to be either cumulative or delta. Elasticsearch previously followed that pattern, too, with native storage of cumulative counters and delta histograms, and workarounds for everything else. Delta counters were stored as gauges, functional but without native counter semantics for rate queries. And cumulative histograms were unsupported.

One workaround for unsupported temporalities is to configure your metric producers (for example, OTel SDKs) to produce data with the temporality that your back end supports. In large-scale deployments, this can be a very challenging task. And sometimes this isn’t even possible (for example, if you consume OTLP metrics from third-party services).

Another workaround is to convert the temporality prior to ingestion. In the OTel Collector, you would typically use the cumulative-to-delta processor, which comes with a big warning sign about statefulness. The conversion is inherently stateful, requiring ordered delivery of metric series to the same collector and persisted state across restarts. In practice, it works, but at scale, it comes with a lot of deployment headaches.

With Elasticsearch 9.5, you can skip the conversion pipeline entirely. Elasticsearch natively stores and queries metric data with both temporalities. It doesn’t require any stateful conversion required or explicit configuration of your OTel SDKs.

Demo: ingesting cumulative and delta OTel metrics side by side

To demonstrate the temporality support, we'll reuse a demo setup from our OTel histogram metrics ES|QL blog post: a Java Renaissance benchmark instrumented with the OTel Java agent. The twist this time: We run two instances of the benchmark, each configured with a different temporality:

• renaissance-delta: Exports metrics with delta temporality.

• renaissance-cumulative: Exports metrics with cumulative temporality.

Both instances report the same metrics under the same service name renaissance, but with different service.instance.id values. Here’s the relevant section of the docker-compose.yml that can be found in the companion code:

To run the demo yourself, you'll also have to fill out the managed OTLP endpoint URL and the corresponding API key:

After starting the demo with docker compose up --build, both instances will start reporting metrics to Elasticsearch.

Querying OTel counter metrics with ES|QL and PromQL

Let's query the first few raw data points of jvm.cpu.time for both instances to see the different temporalities in action:

This gives us the first five data points for each service instance:

The benchmark consumes CPU at a nearly constant rate. This is directly visible based on the delta temporality data: The values are nearly constant between exports. In contrast, the cumulative temporality values grow over time, as they represent the total CPU usage of the benchmark instance.

Now let's have a look at how to properly query this metric using PromQL:

The screenshot shows that both benchmark instances consume a nearly constant of 1 to 1.2 number of CPU cores with some variance. This query works because we made our rate implementation respect the temporality: Every time series (so every service instance in our case) stores the temporality as a metric dimension. The rate implementation looks at this dimension and interprets the data accordingly: For delta temporality, values are summed up; for cumulative temporality, a difference computation is done. This all happens automatically in the background, without requiring any changes to your queries.

We’ve adapted rate, increase, and irate to work this way. The same applies when using those functions in ES|QL TS queries:

Because Elasticsearch tracks the temporality as a dimension, you can have multiple series with different temporalities for the same metric, just like in the demo use case. Aggregating across series also works as expected, because at that point rate, increase, or irate already took care of normalizing the data:

Querying OTel histogram metrics across temporalities

Metric temporality applies to histograms in the same way it applies to counters: histogram buckets are effectively a set of counters, each tracking values in a specific range.As in our histogram demo, we use exponential histograms, where bucket boundaries adapt automatically to minimize relative error.

Due to this similarity, histograms can also be cumulative or delta. Either the counter per bucket is reset after each metric export or the cumulative count carries over between exports.

Let's query the median major garbage collection (GC) duration for our benchmark instances, which is a histogram metric:

Or the equivalent ES|QL query:

Again, both queries will automatically load the temporality per series and interpret the histograms accordingly. In PromQL, this is handled by the increase function. Note that in ES|QL, you don't explicitly call increase on histograms. The TS command automatically handles the temporality-aware merging of histograms when you use aggregation functions, like PERCENTILE, MEDIAN, or AVG.

How Elasticsearch stores metric temporality in TSDB

Elasticsearch's time series database (TSDB) stores metric temporality in a dedicated dimension field on each document. The index.time_series.temporality_field index setting lets you specify which field carries the temporality information. The field must be a keyword field with time_series_dimension: true and the permissible values "delta" or "cumulative".

As soon as this setting is present on a time series index, ES|QL and PromQL will load the corresponding field when performing temporality-dependent aggregations. If the field isn’t present or has no value on a document, we fall back to defaults based on the type of the corresponding metric: counters default to cumulative, and histograms default to delta. This matches the historical behavior and ensures existing queries and existing data continue to work without changes.

When you ingest metrics via the OTLP endpoint, Elasticsearch automatically adds a temporality dimension field to each document, populated from the OTLP AggregationTemporality metadata. For custom (neither OTLP nor Prometheus remote write) ingestion, you’ll have to manually set up the index.time_series.temporality_field setting and populate your temporality dimension.

The temporality is also respected during downsampling: As it’s a dimension, it’s preserved automatically and used to compute the aggregate values.

Getting started with mixed-temporality OTel metrics in Elasticsearch

With Elasticsearch 9.5, cumulative versus delta is no longer a decision you have to get correct at the start. Ingest both temporalities side by side, even for the same metric name, and let ES|QL and PromQL handle the rest. You can switch between both without having to touch your queries. For more details, see the metric temporality documentation.

Why OAuth is better than API keys for MCP server authentication

With OAuth, tokens are short-lived credentials that expire on their own, so a leaked token is a narrowing window rather than a standing grant. Every token traces back to an explicit consent: which user, which client, which time.

In contrast, an API key is a long-lived credential. It lives in a configuration file on the machine that runs the agent, working for whoever uses it until someone rotates or deletes it. That model is manageable for a CI pipeline you wrote and deployed for your team. It gets uncomfortable when the key is held by an AI agent that assembles its own requests, retrieves untrusted content that may contain injected instructions, and sometimes passes context to sub-agents.

The failure mode is familiar from every credential-leak postmortem: the key ends up somewhere it shouldn't (a log file, a prompt), and from that moment anyone who has the key can access everything its creator could. The audit trail doesn't help much: API key logs tell you that a key with a given name did something, but not who authorized the client that used it or when.

Attribute

OAuth 2.1

API Key

Credential lifetime

Short-lived, auto-refreshing

Long-lived until manually rotated

Audit trail

User, client and timestamp per connection

Key name only

Revocation

Per connection, immediate

Requires key rotation

Blast radius

Single client connection

Everything the key creator can access

How to set up OAuth 2.1 for the Elasticsearch MCP server

Setup is a one-time step per project:

1. In your Elastic Cloud Serverless project, open Agent Builder → Tools library → MCP clients → Create MCP client (OAuth). This gives you a client ID and the MCP server URL.

2. Add the server to your MCP host. In Cursor or Claude Desktop, the configuration file entry looks like this:

3. The first time the agent calls a tool, the host opens your browser on an Elastic Cloud consent screen. You sign in with your normal Elastic Cloud credentials and see what the agent is requesting access to.

4. Click Authorize. This creates an application connection between you, your machine, and your project, and Elastic Cloud issues the agent a short-lived access token.

From there, your agent has access to your project’s data. The mechanics are invisible. When the access token expires, the host refreshes it using a refresh token with a 30-day rolling expiry, so you are not re-authenticating every hour. If you stop trusting a connection, open the application connections page in Elastic Cloud and revoke the connection. Both tokens die immediately, and nothing else about your account changes.

The agent acts with the permissions of the user who consented. Calls to Agent Builder tools, such as ES|QL queries, Workflows, and Streams, are evaluated based on your role. If you want an agent with less access than your own, authorize the connection as a user with a tighter role, or keep using a scoped API key for that workload. 

How to manage and revoke MCP server connections in Elastic Cloud

Org owners can list every active application connection across an organization or a specific project: which MCP hosts were authorized, by whom, and when. Revocation is per connection, so cutting off one misbehaving agent does not disturb the authorizing user's account or any other integration. There is no shared credential to rotate and no blast radius beyond the one client.

This is the practical difference for teams. An OAuth app connection records who consented, to which client, and when. An API key log shows that agent-key-3 queried an index, but not who authorized that agent. 

What's next for OAuth authentication across the Elastic API

Agent Builder is the first Elastic surface behind OAuth, and the same authorization model will carry to the rest of the Elastic API surface, including Elasticsearch and Elastic Cloud management, as we work toward a single Elastic MCP endpoint.

To try it now, open Agent Builder in an Elastic Cloud Serverless project and create an OAuth client, or start with the documentation. If you don't have a project yet, you can start a free trial.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

• Geometry-preserving Embeddings: jina-embeddings-v5-omni sits atop the foundation laid by jina-embeddings-v5-text. That original text embedding space is completely unchanged, with its geometry left wholly intact.

• Locked: Synonymous with "frozen." All of the towers in this architecture have their weights locked. 

• Aligned: Aligning the other modalities with the text model's vector space, allowing for cross-modal comparison.

• TOwers: Modality component that converts one type of raw input into vectors.

The model comes in two variants: small and nano. The former has more parameters (1.57 billion) than the latter (0.95 billion), but functionally their architectures are nearly identical. For the sake of brevity, we mostly focus on jina-embeddings-v5-omni-small in this article.

What are vectors, towers and frozen encoders?

jina-embeddings-v5-omni relies on three core machine learning (ML) concepts: vectors, towers, and frozen weights. Here's what each means; feel free to skip ahead if you're already familiar with them. 

How vectors represent data in embedding models

How does AI understand abstract concepts? Can a machine comprehend what "ice cream" is? Does it understand that "chocolate fudge" and "rocky road" have more in common with each other than "sorbet"? The answer, surprisingly, is yes. The mechanism that makes it possible is the vector.

Take a look at this diagram.

If you tried visualizing a way to organize all ice cream flavors, you may end up with something like this: a simplex (triangle) with three vertices, each representing a base flavor. Each flavor is closer to or farther from each vertex, depending on how much of the corresponding base makes up that particular flavor. Chocolate ice cream is all chocolate, so it hugs the top vertex. Vanilla has a similar affinity for the bottom-left vertex. But cookies and cream is roughly a 50/50 mix, so it's about equidistant from both. Neapolitan blends all three, so it sits at the center of the triangle.

This is functionally how vectors work. These flavors get funneled into an ML model (more specifically known as an embedding model) that will then generate coordinates for each of these flavors along this simplex. By measuring the distance between coordinates, software can parse how related or unrelated two flavors are. It's easy to see that "chocolate" and "chocolate brownie fudge" are closely related flavors because they sit close to one another, but "strawberry" and "cookies and cream" are far away from each other, so we can infer that they aren’t similar.

It won't be quite this simple though. Rather than three labeled points, real embeddings run along hundreds or even thousands of dimensions. Nor will they have nice, human-readable labels; the vertex markers are something that only the model understands. The benefit, though, is that we can graph basically anything like this.

To clarify some jargon: These coordinates = vectors = embeddings. For the rest of this article, we use these terms interchangeably.

Much like how embeddings go by many different names, so too do the models that create them.

What is a tower in multimodal embedding models?

The term tower comes from Contrastive Language-Image Pre-training (CLIP), a model released by OpenAI in 2021 that was one of the first to learn a shared embedding space across text and images. In CLIP, each modality is handled by a completely separate model. On an architecture schematic, these models look like towers standing side by side, each taking one type of input and producing vectors in a shared space. The name stuck, and you'll see it used broadly across multimodal ML. 

In the context of jina-embeddings-v5-omni, the word is used a bit more loosely. Its architecture doesn't have true parallel towers in the CLIP sense. All modalities ultimately funnel into a single central text model, rather than sitting as equals beside it. 

With that caveat in place: A tower (or modality component) is a pipeline that converts one type of raw input into vectors. A text tower vectorizes strings, and a vision tower generates image embeddings. An audio tower does the same for sound.

Why freeze a tower instead of training it?

If we want multimodal capabilities, could we Frankenstein multiple towers that handle each of those inputs together into one model? The issue with this approach is that vectors from different models aren’t intelligible to each other. Images will exist in one vector space and audio in another, for example. This means that we have no ability to compare across different modalities. Think of the vectors outputted from one model as existing in their own language. Let's say our audio tower outputs Spanish and our image tower outputs English. Conceptually, the vectors can be describing the same things, but downstream tools that try to make use of these embeddings are "monolingual," so we're out of luck.

The CLIP-style approach is to take several towers and train them together, letting them all reshape each other until their outputs agree. It's sort of like having Spanish and English speakers try to communicate for long enough that they eventually all start speaking Spanglish.

This approach works, but it has a side effect: Towers that you already had working get remodeled in the process. Any embeddings that they produced before are now incompatible with any that are produced by the older version of the tower. If you had a text tower and generated 100 million embeddings with it, you would now need to re-embed all of those strings.

To combat this issue, you can freeze certain towers. This locks their weights, which are the knobs and dials that influence how they behave. This way, training never changes them. What you train instead is a small projector that translates one tower's output into another tower's language. Many different models train some portion of projectors and towers while leaving some others frozen. What makes jina-embeddings-v5-omniunique is that we froze every encoder and trained only the projectors and delimiter tokens. By the end of this article, you’ll understand exactly how that works.

You can think of these frozen and trainable components as clusters of neurons or isolated regions of the mind, and the whole embedding model (jina-embeddings-v5-omni) as the entire brain. Between these learning-capable components sit fixed math operations, such as merging, squashing, selecting, and rescaling numbers on their way from one tower to the next. They have no weights, so there’s nothing in them to freeze or train. 

With that groundwork laid, here’s the full architecture.

How the jina embeddings multimodal architecture works

The architecture routes all modalities through frozen encoders and small trainable projectors into a single shared text embedding space.

Why the text tower is the backbone

The first thing to notice here is the general flow of data in this diagram. Image, video, and audio inputs all (eventually) end up in the same space as text inputs. Why is the architecture set up like this?

jina-embeddings-v5-omni builds on top of jina-embeddings-v5-text: It retains its text-processing backbone and extends it with pretrained vision and audio components.

This has two main advantages:

1. The text model is already state of the art for its size and what it does, and it stays completely untouched. There’s no need to fix what isn't broken. We also don't need to re-embed anything we already embedded with jina-embeddings-v5-text.

2. A single shared vector space is what makes cross-modal search work at all. Because image, audio, and text all resolve to vectors in the same geometry, you can query an image with text, or audio with text, and compare them directly with cosine similarity (a similarity measure based on the angle between two vectors). If each modality lived in its own separate space, those comparisons would be meaningless.

How vision and audio encoders feed into the text model

The first step in building upon the foundation set by jina-embeddings-v5-text is integrating vision and audio encoders into this architecture. In this case, we use the existing Qwen3.5 vision encoders and the Qwen2.5-Omni audio encoder, which themselves have been adapted from SigLIP2 and Whisper-large-v3, respectively. They’re ultimately what’s responsible for generating raw vectors for all vision- and audio-based data. The emphasis is on raw here, since much transformation still needs to be done afterward.

How the vision encoder processes images

In the case of images, we’re borrowing more from Qwen than just the encoder. Additional plumbing inherited from Qwen is attached to the output of the encoder. Let's walk through what comes out of the encoder and how the inherited downstream components transform that output.

Vision encoder (frozen)

Let's use an image as our primary example, since the visual component of video is basically identical. Take this image of a banana split.

Rather than generating one single, clean vector embedding for this image, the vision encoder breaks the image up into 14-pixel by 14-pixel sections and generates a tiny patch token (basically a mini-vector) for each of these sections.

While still inside of the vision encoder, each token looks at every other token that makes up the image and pulls in information from the ones relevant to it, updating its own vector based on that context. This process allows us to preserve fine-grained details. Now, instead of one single vector for the whole image, we have multiple, smaller vectors that represent particulars of the whole dish.

LayerNorm (frozen)

LayerNorm rescales each patch's numbers so they sit in a consistent range before anything else touches them. It stops some patches from being wildly larger than others and drowning out the rest.

2x2 merge 

We saw in the vision encoder section that we split up the image into small, 14-pixel by 14-pixel squares. However, processing this many patch tokens will become expensive downstream. For this reason, the 2x2 merge operation consolidates four patch tokens into one. The squares are now 28 pixels by 28 pixels. The corresponding patch tokens are similarly consolidated.

fc_vision_1 (frozen) 

fc_vision_1 is the first of two matrix multiplies; it mixes the merged patch numbers into a new set. This layer is inherited from Qwen and left as is.

GELU 

Gaussian Error Linear Unit (GELU) is a gate applied to each number. It lets useful signals through and squashes the rest toward zero. It’s the one nonlinear step and is ultimately what lets the two fc_vision layers together learn shapes that a single flat multiply couldn't.

fc_vision_2 (trainable) 

As mentioned earlier, trying to put Qwen image embeddings directly into the same vector space as Jina text embeddings would be like trying to include a Spanish sentence in an English novel. Outside of its native context, its meaning is totally lost.

That is what the trainable projector fc_vision_2 is here to fix. It learns to translate the image embedding into something that jina-embeddings-v5-text can understand. For this reason, you can think of a trainable projector as a translator.

The emphasis belongs on trainable: This is the first component we’ve encountered in our walkthrough so far that isn’t frozen. Many components of this architecture are frozen, meaning that their weights are locked and never change during training, but fc_vision_2 is one of the few parts that actually gets updated, because it has to learn how to translate Qwen's image dimensions into a form that lands meaningfully inside Jina's text space.

× 4

By now, you may have noticed that fc_vision_2 has a "× 4" marked on the bottom, along with fc_audio, both encoders, and the ″Embedding text″ section within jina-embeddings-v5-text. In this case, it represents four different instances of fc_vision_2, each optimized and tuned to one of four slightly different tasks, outlined below.

Task

What it facilitates

Example user input

Example end result

Note

Retrieval

Finds a similar match for the input (that is, standard Google search)

"melting ice cream" as a string/text

Picture of a fallen ice cream cone on asphalt

Text-matching

Judges how similar inputs are

A text string "melting ice cream" and an image of a fallen ice cream cone on asphalt

Score judging how similar the two inputs are

The name of this task is a bit of a misnomer. It's called text-matching, but it works for any modality, not only text.

Clustering

Groups data into clusters

A large array of ice cream images

Lets the user discover natural groups, like "sundaes" and "popsicles"

Classification

Places data into predefined buckets

Two string labels: "melting" and "intact", along with a large array of ice cream images

Sorts the array of ice cream images into the two provided categories, based on their proximity in vector space to the label embeddings

To clarify, the "Example end result" column is the takeaway after some additional math and processing happens once the output vector is generated. The point is that  jina-embeddings-v5-omni only generates vectors. Those vectors take on a mildly different form to optimize for the selected task type.

These task types are also explicitly outlined in the Low-Rank Adaptation (LoRA) component of the architecture diagram, which we'll cover in a moment.

How the audio encoder processes sound

Before we go any deeper into the trenches of our model architecture, let's back up and see how the audio-oriented path differs and how it stays the same. If you were able to follow along during the vision section, the audio portion will be a breeze. We have no extra inherited plumbing from Qwen this time, only the audio encoder and one trainable projector.

Audio encoder (frozen)

Before audio can enter the encoder, it needs to be converted into a form that the encoder can work with. Raw audio is a one-dimensional wave, which isn't particularly useful to a neural network on its own. Instead, the audio is first transformed into a mel spectrogram: a 2D representation that maps frequency against time, weighted to emphasize the frequency ranges that the human ear is most sensitive to. Think of it as a visual fingerprint of the sound. Below is a mel spectrogram of a person saying, "I scream, you scream, we all scream for ice cream."

That spectrogram is then sliced into fixed-length 40ms chunks, analogous to how the vision encoder breaks an image into 14×14 pixel-tiles. The encoder then produces one token per chunk.

This matters for the same reason it does in the vision pipeline: Fine-grained detail would otherwise be lost. A single embedding for the full phrase "I scream, you scream, we all scream for ice cream" would smear everything into one blurry vector. Slicing it into short chunks keeps each fragment of sound intact, so the model can later distinguish "scream" from "cream" rather than collapsing them into an average. The same applies to music, animal noises, environmental sounds, and more.

fc_audio (trainable)

Once the encoder has produced its tokens, fc_audio performs the same translation role that fc_vision_2 does for images: It projects each audio token from the encoder's 1280-dimensional output space into jina-embeddings-v5-text’s hidden dimension (1024 for small, 768 for nano). Like fc_vision_2, it carries a "× 4" in the architecture diagram, meaning that there are four instances, each optimized for a specific task type (retrieval, text-matching, clustering, and classification).

Delimiters (trainable)

We need to convey to jina-embeddings-v5-text that this embedding represents an unexpected modality. We can do this pretty easily with delimiters. In the world of HTML, this looks like:

The first and last <p> tag conveys that everything in between them is a paragraph element.

In our case, things are a bit more complicated. For audio, the architecture diagram shows the delimiters as <aud_start> and <aud_end>, but that isn't quite accurate. The delimiters themselves are actually vectors rather than hard-coded strings.

Each task type has its own pair of delimiter vectors (hence the "4 × special tokens"). These vectors are identical every time rather than being different for each audio embedding. So a task type of retrieval with an audio input type always gets start delimiter vector X and end delimiter vector Y; text-match with audio input always gets a start delimiter vector A and an end delimiter vector B, and so on.

This is necessary because the Transformer layers component only understands vectors. So, by the time the audio embedding makes its way there, it looks like:

The same pattern applies to images:

Video is where this gets interesting. Up to 32 evenly spaced-out frames are pulled from the video and fed into the vision encoder. 

Rather than producing one delimiter-wrapped segment, each sampled frame gets its own <vis_start> / <vis_end> wrapper, and these per-frame segments are concatenated into one long sequence:

This is what makes multi-frame video work: Rather than averaging frames or treating them separately, the transformer receives the whole video as one token stream, allowing its attention to relate tokens across frames. This means that earlier frames can inform how later ones are interpreted. If the video does have an audio component, it's pulled out and fed into the audio encoder and ultimately prepended to the frame sequence.

The transformer layers component then processes this entire concatenated sequence as a single input.

Jina text transformer layers (frozen) 

We’ve generated patch tokens for our images, videos, and audio files. We’ve also wrapped them inside of vector delimiters, all for the sake of having them understood by these layers. They’ll allow each patch token to examine the other patch tokens and determine whether they need to update themselves based on the surrounding context. I know what you're thinking:

Didn't we already do this inside of the encoder? We split up the image into 14-pixel by 14-pixel sections and generated patch tokens for each section, and then the encoder updated each patch token based on surrounding context within the same image.

And you're right! We did. But there's a key difference now.

Originally, that recontextualization ran on the attention of Qwen's vision encoder. The operation within the Transformer layers runs the frozen Jina text transformer's attention. It’s the same operation with different learned parameters, so it transforms the tokens differently.

Think of it like a move: The projector is the flight and the moving trucks. It physically relocates you from vision land to text land, landing you in the right city and even the right neighborhood. The transformer's attention is what happens after you've unpacked. You're already home; you spend the next few weeks figuring out exactly where you fit, meeting the neighbors, finding your bearings, and adjusting your exact spot based on who's actually around you. You did the macro move already. This is the micro fine-tuning.

Lastly, the <vis_end_vector_delimiter> will absorb all the information from the patch tokens it wraps.

LoRA (frozen) 

LoRA is a way of fine-tuning an existing model without completely retraining it. It’s a small set of extra adjustment knobs bolted onto the transformer that nudges its behavior to optimize for one of the specific tasks (such as retrieval or classification).

Last-token pooling 

Since <vis_end_vector_delimiter> absorbed all the other patch tokens into itself, we don't need to consider anything except it, so we throw the rest away. It acts as a stand-alone embedding that represents a summary of the whole.

L2 normalization 

<vis_end_vector_delimiter> could be any length now, which is no good. This step shrinks or stretches it so its length is exactly 1, without changing the direction it points. This is tidying so that comparing it to other vectors later is a fair, clean angle comparison. It changes only the vector’s scale, not what it means.

Enough about ice cream

As we've journeyed our way through jina-embeddings-v5-omni, we’ve been careful to outline which components are frozen and which ones are trainable. By now, you may have noticed that every single tower has been frozen. In fact, only the small projectors (translators) and delimiter tokens have been trainable. We dubbed this architecture pattern GELATO. This makes the entire training process significantly cheaper.

To be clear, we didn't invent the concept of frozen towers. Prior work on Locked-image Tuning (LiT), VISTA, and Multi-modAl Retrieval model via Visual modulE pLugin (MARVEL) froze one side or the other. What no one had done before GELATO was push the idea to its limit: text, image, video, and audio all in one model with every encoder frozen. The only trained pieces are a single projector layer per modality and a handful of delimiter tokens.

But is it any good?

Benchmark results: jina embeddings vs. other multimodal embedding models

There's no point in building out a model and releasing it if you don't even know if it's any good, especially compared to the competition. That's why we have benchmarks and evaluation frameworks. The benchmarks that jina-embeddings-v5-omni was run against are Massive Image Embedding Benchmark (MIEB), Massive Audio Embedding Benchmark (MAEB), Massive Multimodal Embedding Benchmark–Video (MMEB-Video), and Massive Multilingual Text Embedding Benchmark (MMTEB).

As for the models we compare against, it's important not to make apples and oranges comparisons. For that reason, we’re specifically using open-weight omni-style models with support for the same media types:

• LanguageBind

• Omni-Embed-Nemotron-3B

• LCO-Embedding-Omni-3B

• LCO-Embedding-Omni-7B

Evaluation

The table above is sorted by parameter count. Models with the fewest parameters are clustered at the top, and models with the most parameters hug the bottom. This context is important because performance alone isn't the final variable. If a $100 ice cream with gold flakes and the finest dairy milk tastes the same as (or worse than) the average gallon of ice cream that you can buy from your local grocery store, it would be silly to buy it, because you're paying a massive premium for nothing.

A similar situation is unfolding here. jina-embeddings-v5-omni-small and nano outperform every other model on text, despite ranking low to middle in terms of parameter count.

Audio performance is strong, as well. jina-embeddings-v5-omni-small and nano beat out all other models except those from LCO, which they both trail by about 2 to 3 points.

The gap shrinks when considering image performance, particularly with jina-embeddings-v5-omni-small. It beats both LanguageBind and Omni-Embed-Nemotron-3B. It lags less than a point behind both LCO models, despite the fact that they have 4.70 billion and 8.93 billion, respectively, compared to Jina’s 1.57 billion.

Video is the weakest performer for our models, though even in that case jina-embeddings-v5-omni-small still beats Omni-Embed-Nemotron-3B, which has three times as many parameters. Ultimately, when these scores are averaged out, you get the following rankings:

Model

Number of parameters (B)

Average score

LCO-Embedding-Omni-7B

8.93

54.43

jina-embeddings-v5-omni-small

1.57

54.04

LCO-Embedding-Omni-3B

4.70

53.83

jina-embeddings-v5-omni-nano

0.95

47.49

Omni-Embed-Nemotron-3B

4.70

41.21

LanguageBind

1.14

35.82

LCO-Embedding-Omni-7B has nearly six times the number of parameters as jina-embeddings-v5-omni-small but barely squeaks past it in average performance.

One benchmark deserves a special callout for anyone building search or retrieval augmented generation (RAG): visual document retrieval, measured on the ViDoRe benchmark. 

Here, jina-embeddings-v5-omni-small scores 79.25 using only 0.92 billion active text-and-image parameters, ahead of LCO-Embedding-Omni-3B (78.24) and within striking distance of LCO-Embedding-Omni-7B (80.32), a model nearly 10 times its size on that path. nano matches that exact same 79.25 score with just 0.31 billion active parameters. The larger Omni-Embed-Nemotron-3B does take the top spot at 85.64, but it carries roughly five times the active parameters of jina-embeddings-v5-omni-small, so our models remain the most parameter-efficient of the group. If your workload is retrieving pages of documents by their layout and text, this is the number to weigh.

Limitations of frozen-tower multimodal embeddings

GELATO's frozen-tower design delivers strong results at low training cost, but it comes with trade-offs worth naming plainly. As already mentioned, the most consistent weak spot is video. jina-embeddings-v5-omni-small trails the LCO models on video, and moment retrieval (locating a specific event within a clip) is the weakest subtask of all. This is partly structural, since each frame produces its own token set before everything is concatenated and pooled into a single final embedding. Packing that much information into one embedding means that the early dimensions carry a heavier load, so video embeddings degrade faster than image embeddings when truncated to smaller sizes.

Audio has its own gap. While retrieval and classification scores are competitive, audio clustering is the weakest audio subtask (6.13 for jina-embeddings-v5-omni-small).

Cross-modal audio–text retrieval trails LCO-Omni-7B by 11–15 percentage points, a larger gap than the 6–7 points seen on the image–text (I-T) pair. The fc_audio projector is the natural next target for additional trainable parameters, suggesting the audio–text (A-T) alignment path has more room to grow than the multilayer vision pipeline. 

How multimodal embeddings distribute in vector space

We've already discussed performance via benchmarks, but what about how the actual embeddings are distributed in vector space? How does that tangibly differ from model to model?

In the above illustration, data from video clips is funneled into each model and graphed in vector space. It’s then compressed down to two dimensions via the Uniform Manifold Approximation and Projection (UMAP) method for easy visualization. Each modality corresponds to a different component of the video:

Modality

Component of the video

Image

Frame from the middle of the video

Video

The full video

Audio

Audio track from the video

Text

Description of the video

Immediately, some interesting patterns stand out.

Our models and the LCO models seem to have different modalities all mixed together, while LanguageBind and Omni-Embed-Nemotron-3B seem to lean more toward having their embeddings separated by modality.

Our models and the LCO models exhibit interleaved geometry for these vectors. This means that different modalities aren't clearly separated in vector space, but instead intermingle in similar areas. This is less pronounced with Omni-Embed-Nemotron-3B, since only image and video seem to occupy a similar space.

LanguageBind is fully separated, with different modalities occupying entirely different spaces. This is known as the modality-gap pattern.

So which one is better? In practice, interleaved geometry tends to be the more useful of the two, and it’s worth noticing that the strongest models in our benchmarks (ours and LCO's) all exhibit it. However, there are trade-offs.

Interleaved geometry excels at cross-modal retrieval, since everything is jumbled up together in the vector space and, therefore, much closer. It's easier to find a matching picture for the text "strawberry ice cream" when the text and image embeddings sit so close together in vector space.

When you're trying to do a same-modality task though, the image that was so conveniently within reach is now in the way. However, in practice, this is easily mitigated by metadata filtering on something like a “modality” field. 

No such workaround exists for the issues inherent to models that exhibit the modality-gap pattern. It’s easy to find another video of syrup poured on ice cream, since all the videos are sitting together in isolation. But having modalities confined into clusters like that makes finding an accompanying image much harder.

Why this architecture?

GELATO gives a lot of performance for very little training. You keep all your towers as they are and train only small projectors and delimiter tokens, which is significantly cheaper than the alternatives. To put concrete numbers on "cheaper": for jina-embeddings-v5-omni-small, training just the vision projector updates 4.20 million parameters instead of the 920.6 million a full fine-tune would touch. At the same 15,000-step budget, that projector-only run finishes about 1.8 times faster and peaks at 7.52 GiB of GPU memory instead of 12.96 GiB. The audio path shows an even wider gap, with projector-only training running 3.2 to 3.9 times faster than full training. But how did we conclude this was the way to go? We used a process known as ablation.

Ablation is when you remove or change one piece of a system to see how much it actually mattered. Imagine you've been working on an ice cream recipe. Every time you make a tweak, like doubling the milk, swapping brown sugar for white, using vanilla beans instead of extract, or taking out the chocolate chunks, that's ablation.

Ablation in ML functions much in the same way. It asks whether removing, rearranging, freezing, or unfreezing certain components makes the whole system more, less, or equally as performant. In this case, we’re particularly interested in whether unfreezing certain components, and in what order, may affect performance. We conducted five ablation studies on the Qwen3.5 vision stack. The results are measured in mean nDCG@10 (normalized Discounted Cumulative Gain), a standard score for ranking quality where higher is better.

Overall, nearly every ablation study yielded basically identical results, except for case #3, which performed terribly. Before we explain why, let's look at the equivalent ablation diagram for audio.

In this instance, ablation case #2 performs the worst. Do you see the commonality between the worst performer here and the worst performer among the vision ablations? Across both modalities, the same rule holds: If you unfreeze the encoder before the projector has been trained, you’ll see worse performance.

For both modalities, we ultimately chose ablation case #1 for the final architecture. Both had relatively high scores. In vision's case, the configurations that edged out case #1 did so by margins too small to justify their added training stages and extra per-task artifacts. A similar story unfolds for audio, with case #3 beating out case #1 by a small margin but requiring more per-task artifacts.

Ablation validates the GELATO approach: It's cheaper and nearly identical in quality to train a dedicated translator (rather than the speaker).

Summary: why frozen encoders make multimodal embeddings cheaper

Rather than expensively retraining multiple towers to achieve multimodal capabilities, GELATO allows us to minimize cost by freezing our already functioning towers and training only small projectors to translate embeddings. These embeddings get funneled into jina-embeddings-v5-text, ultimately allowing all the output vectors to exist in the same, interleaved geometry. We can now compare text, audio, images, and video at a fraction of the cost of the competition. 

Both jina-embeddings-v5-omni-small and jina-embeddings-v5-omni-nano are open-weight for personal use and available now. You can download them from the Jina AI collection on Hugging Face and start generating multimodal embeddings today, or read the full technical report for the complete set of benchmarks and ablations.

How to configure frozen_after in data stream lifecycle

frozen_after sits at the top level of the lifecycle, next to data_retention and downsampling.

That's the whole feature, at the API level. Indices in my-data-stream stay on hot for 30 days, then move to frozen for the remaining 60. After 90 days they're deleted, and the backing snapshot goes with them.

It composes with the rest of the lifecycle, including downsampling:

Same options in an index template:

The order of values is enforced: frozen_after has to be less than data_retention and greater than any downsampling.after. The API rejects configurations that don't make physical sense.

Where frozen tier data is stored: the default snapshot repository

Frozen tier data is held as partially-mounted searchable snapshots, which means DLM needs a snapshot repository to write into. Rather than make you choose a repository per lifecycle, 9.5 introduces a cluster-level default snapshot repository.

DLM uses this repository for every frozen tier index in the cluster. On Elastic Cloud Hosted (ECH), the default is pre-populated with found-snapshots so existing clusters work out of the box. You can change it to a repository you control if you'd rather keep your frozen data in a bucket you own (useful if you want object versioning, lifecycle backups to Glacier, or anything else that needs bucket-level access). Wherever you can set frozen_after in Kibana, the UI shows the current default repository inline and links to the place to change it, so you can see where frozen data will be written.

If the cluster doesn't have a default repository configured, you can still write a lifecycle with frozen_after. The API accepts it but returns a warning:

The data stays on hot until a default repository is configured and exists. The same logic applies if the cluster lacks a valid Enterprise license. Errors are visible in the data stream lifecycle status API for that stream.

Configuring frozen_after in Kibana

Kibana in 9.5 lets you set frozen_after and the default snapshot repository from the UI. In Streams, the Retention tab shows the frozen phase on the lifecycle timeline alongside hot and any downsampling steps. Click the timeline to open the data lifecycle flyout, set frozen_after, and see the timeline update before you save. Index Management's Data Streams page opens the same flyout.

How frozen tier conversion works in data stream lifecycle

When a backing index ages past frozen_after, DLM walks through five steps in order:

1. Clone. Mark the index read-only and clone it to a zero-replica copy so the original stays available while conversion runs.

2. Force merge. Merge the clone to a single segment. On completion a cluster state marker is written; duplicate force-merge requests (for instance after a master failover) are deduplicated, so a restart doesn't repeat the work.

3. Snapshot. Write the merged clone to the default repository, and record the snapshot name in cluster state on success. If a stalled snapshot from a previous attempt is detected, DLM deletes it and re-runs the step.

4. Mount. Create a partially-mounted searchable snapshot index from the snapshot.

5. Swap and delete.Once the mounted index's shards are fully allocated, atomically swap it in for the original in the data stream, then delete the original. The swap is atomic, so query results don't see a data volume dip during the transition.

On failure, DLM retries from the last successful step on the next run.

Each step is idempotent, and the cluster state markers make sure work already done isn't repeated after a master failover. Throttling caps concurrent conversions so a lifecycle change covering thousands of indices doesn't overwhelm the cluster. Errors surface in the data stream lifecycle status API and roll up to the lifecycle health indicator.

Scope and limitations of frozen_after

• Data indices only. frozen_after applies to a data stream's data indices. Failure store indices don't currently support the frozen tier and continue to be governed by their own data_retention setting.

• Enterprise license required. Frozen tier in DLM is implemented with searchable snapshots and requires an Enterprise license. You can write frozen_after on any license, but data won't move to frozen until the license is valid.

• Serverless ignores the field. In Elastic Cloud Serverless, frozen_after is accepted but ignored - Serverless manages tiering on your behalf. Built-in templates may include the field, so we don't reject it, but the step is skipped.

Getting started with frozen_after

1. In Kibana, open Streams or Index Management and choose a data stream backed by data stream lifecycle.

2. Open the Edit data lifecycle flyout, set a frozen-after value, and save. The lifecycle timeline shows the new phase.

3. On a self-managed cluster, set repositories.default_repository to a repository you control. On ECH, found-snapshots is already configured if you want zero setup.

4. For declarative workflows, write the same configuration into your index templates so new data streams pick up the lifecycle automatically.

Learn more

• Data stream lifecycle

• Frozen tier overview

• Searchable snapshots

• Manage data retention for Streams

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

While AI indices will be included in future Stack releases, today we recommend using Serverless.

How it works: AI Index, Kibana Workflows, and the query-ki skill

Building context in this walkthrough has three moving parts:

1. An AI Index, where KIs live. It's a regular Elasticsearch index or data stream with a specific naming convention triggering component templates to configure the right mappings automatically.

2. Kibana Workflows, which read from your data sources, run an LLM to structure content into KIs, and write those KIs into the AI Index.

3. A query-ki skill, a skill that queries KIs directly from the AI Index using ES|QL, and that a chat agent can call as a tool.

Prerequisites

This tutorial assumes you have:

1. An Elasticsearch Serverless project. You can sign up for a trial if you don't have one.

2. An API key to access your Elasticsearch project. 

Create sample indices for agent routing

First, we’ll need some sources. Sources can be data that already exists in your Elasticsearch indices, or external data accessed via connectors or ES|QL data sources. 

For this blog, we’ll create some indices with example data. We’ll start with an example using three datasets: BEIR/fiqa (financial), beir-nfcorpus (biomedical/nutrition), and beir-scifact (scientific fact-checking). Each index is populated with its own _meta.description. 

Here are the mappings we define for these indices:

Then, using the above convenience scripts, load a handful of documents into each index with the _bulk API.

Now imagine an agent with a question and the indices we’ve just created. The agent has no idea which one is relevant at the start. Without pre-computed context, it either performs exploratory lookups (mappings, test searches) to figure out which source to use, or searches all three and hopes the merged results contain something useful. Either approach costs tokens, and if you multiply that inefficiency across every query an agent makes, it adds up.

Create your AI Index

Before generating any KIs, you need an index to store them. We call this an AI Index.

The naming convention is what triggers automatic configuration. Any index whose name starts with ai-index-idx- is a regular index; ai-index-ds- is a data stream. You’ll want to choose data streams for observability use cases, time series data, and when recency is important. Conversely, standard indices are a good choice for static data that will exist for a long while, where recency is not as much of a concern, and may need to occasionally be updated on demand. This naming convention is required for AI indices. 

When Elasticsearch sees the ai-index- prefixes, it automatically applies component templates that configure the right mappings and settings.

Creating an AI Index is a single call:

To see exactly what the component templates applied, inspect the mappings:

The response shows the fields every AI Index gets out of the box:

title, description, and content are each a text field with a .semantic sub-field of type semantic_text, supporting hybrid retrieval.

Data stream indices (ai-index-ds-*) additionally carry a default 90-day data retention policy. This blog uses a standard index (ai-index-idx-*).

Index Metadata as a Knowledge Indicator

The target use case for this example is how the query-index-metadata-ki skill can route an agent to the correct Elasticsearch index, even when index or field names are vague. This reduces mistakes from choosing the wrong index or formulating queries based on incomplete schema exploration.

Since we're creating KIs for our own indices, we can give the LLM a head start: annotate index mappings with human-written _meta.description content. The workflow generates better KIs with more context to work from.

To address this, we'll manually create a Kibana Workflow that profiles each index and writes routing KIs into the AI Index. The workflow chains four steps:

Step

Type

What it does

get_mapping

elasticsearch.request

Read the mapping, including _meta.description and per-field descriptions.

sample_docs

elasticsearch.search

Pull a few real documents so the profile reflects actual value shapes.

profile_index

ai.agent

Generate a structured index profile as structured output.

sink_index_ki

elasticsearch.bulk

Write the profile into the AI Index as a KI.

Paste the following YAML into the Workflows editor:

Let's walk through what this workflow does. We loop over three specified indices with a foreach loop. For each:

1. get_mapping fetches the Elasticsearch index mappings, including any _meta.description annotations we added earlier.

2. sample_docs pulls 3 real documents. Concrete examples give the LLM much better signal than schema alone.

3. profile_index calls ai.agent with the index name, mappings, and sample documents. The LLM returns structured output describing the index's purpose, key fields, and an example ES|QL query showing how to use it.

4. sink_index_ki writes the result into the AI Index as a KI of type index_metadata_entry, keyed on the index name so re-runs are idempotent.

A few things to point out: 

• This workflow hard-codes a specific set of indices. In practice, you could derive the list from an index pattern or a dynamic source. 

• The foreach loop also runs iterations sequentially, which is fine for this guide but slow in production because each iteration involves an LLM call. For scale, use workflow.executeAsync or native parallel support. The cheat sheet has tips on both.

• In the profile_index step, the agent prompt is the special sauce. This is what shapes the accuracy and usefulness of the KIs. 

• Using ai.prompt can improve workflow efficiency (and cost) if you don’t need to load other tools. 

• Cost can be controlled in multiple ways. Richer prompts and structured output often result in higher token utilization, and of course the model you choose significantly impacts total costs. The Elastic Inference Service (EIS) can be a great playground to test different models against the profile_index’s ai.agent step to compare how different models stack up against each other when generating KIs. 

Query your AI Index to verify Knowledge Indicators

Once the beir-index-profile-ki workflow runs, query the AI Index directly in the Discover tab using the following ES|QL query to confirm what got written:

This will result in the following output: 

Build a portable skill to retrieve AI agent context

Retrieval is a critical component in an AI index. A KI is a document in the AI Index, and finding one is a single ES|QL query. We package that query as a small, portable skill so any agent can call it, regardless of the harness it runs in.

We write the skill as a SKILL.md: a YAML header with a name and description, followed by markdown instructions. This is the same Agent Skills format that many harnesses, including Claude Code, LangChain's Deep Agents, and others, load directly. 

The harness reads the header content up front, and only pulls in the full instructions when a question matches the description. The one thing the skill asks of the harness is a way to run ES|QL against Elasticsearch.

Here is a sample query-index-metadata-ki skill:

Let’s break down what the skill is doing: 

• We’re defining index-metadata-entry as a KI type/use case.

• We’re performing a hybrid ES|QL search on our AI indices, filtering by the appropriate type using RRF as the default method to fuse results.

• The KI results will directly ground the agent’s answer when determining what indices are relevant to the query.

Because the skill is just instructions plus a query, it travels wherever your agent does. You can point the same file at a Kibana Workflow agent, Claude Code, LangChain Deep Agents, or any other harness without changing a line of it.

Connect your AI Index to an agent harness 

We want to demonstrate how you can use AI indices to query your data with any harness. For these examples, we’ll use LangChain Deep Agents and an OpenAI-compatible key, but any other agent harness can be easily substituted in, including Elastic Agent Builder.

First, let’s create a baseline to see how an agent will perform without using KIs:

Here’s a modified example that could run the same agent, but now with the ability to search AI indices to return KIs: 

This agent will always query the KI indices to get the answer.

How much do Knowledge Indicators reduce agent token usage?

Since we’re using agents, the results of these scripts are non-deterministic. However, when I ran these results against the query Is there scientific evidence that vitamin D supplementation prevents cancer?, both agents led to the same conclusion, but they took different paths to get there: 

Baseline (No AI Index)

With AI Index

Total tool calls

12

8

read_file calls

0

2

get_mapping calls

3

0

esql_query calls

9

6 

Total indices queried

2 (bounced between beir-scifact and beir-nfcorpus)

1 (beir-nfcorpus)

Tokens consumed

167,763

92,711

Latency

39.58s

36.15s

Answer

Grounded, correct

Grounded, correct

The KI answers were both grounded and correct, but an interesting datapoint is the fact that the overall tool usage and token utilization was smaller when using KIs (latency was roughly equivalent). Here’s how both paths went, side by side: 

Run the full AI Index pipeline in Serverless

This walkthrough offered a deep dive into the build-it-yourself version of AI indices and KIs. In production, you wouldn't hand-write these workflows; a setup agent would generate them, and a feedback loop would refine KIs from the agent's own traces. But the primitives are exactly what you just used: extract KIs with a workflow, store them in an AI Index, and retrieve them with a skill.

Managing context is key to a relevant and efficient agentic search system, and AI indices are a way to manage this context with the full power of the Elastic stack. Try it out in Serverless and let us know what you think in our Discuss forums or the #stack-kibana channel in our Community Slack! 

Before adding precomputed context, the Support team at Elastic used an agent for case investigation, root cause analysis (RCA), triage, and related work. To answer a question, the agent had to discover relevant indices, inspect schemas, issue several queries, reconcile conflicting updates, and assemble a response. Questions ranged from narrow state lookups to multisource investigations:

• What is the current status and priority of support case 1234567?

• After client ABC’s cluster migrated certificates, it returned Secure Sockets Layer (SSL) handshake and certificate_unknown errors. What caused the failure, and how should it be fixed? Which knowledge base articles are relevant?

• Client ABC’s cluster stopped processing indexing requests and returned 'rejected execution of primary operation' errors. What was the root cause, and how was service restored?

The next question about the same case or about a related case often required much of the same orientation and synthesis work. That repetition consumed input tokens and added latency. It also increased the chance of invalid queries or wrong source selection.

Earlier work on precomputed context provided a starting point. It described extracting useful context ahead of time and storing it as structured Knowledge Indicators (KIs). Then it explained how an agent retrieves those compact records before scanning raw documents.

For this support use case, we wanted to understand what context was worth preparing in advance and how the choice of context affected response efficiency and reliability. We also wanted to know what retrieval rules were needed when an agent could use several KI types. This article follows that design process and reports the findings.

Starting with index profiles

We began with index profile KIs. A workflow read each support index mapping and sampled a few documents. It generated a compact profile describing the index’s purpose, the questions it could answer, important fields, exclusions, verified joins, time coverage, example Elasticsearch Query Language (ES|QL) queries, and more. A representative KI document with selected fields looked like this:

The profiles helped the agent choose a source and form a query. They reduced orientation work, especially when index or field names were unclear, but they didn’t remove the main cost of a case investigation. After selecting the indices, the agent still had to gather the case record and reconstruct the conversation. They also still had to follow links to engineering and knowledge sources and then reconcile the evidence. Routing was useful, but the repeated work was synthesis.

Changing the unit of context

That observation led to case-level KIs. The workflow materializes one evidence-aware snapshot per eligible case. It gathers the case record, recent material conversation events, linked engineering issues and comments, and linked knowledge articles. The resulting KI document contains a stable case identifier and source references. It also contains tags and freshness metadata.

The design became more detailed as we worked through support specific nuances. Support conversations contain provisional theories, later corrections, administrative status changes, and occasionally separate incidents inside one case. A single fluent summary can flatten those distinctions. The case schema therefore records incident phases, a hypothesis ledger with confirmed, inferred, contradicted, and unresolved states, root cause status and confidence, technical outcome, reusable learning, and more.

Since large language models (LLMs) can hallucinate some of these details, we added deterministic checks after generation. For example:

• An unresolved root cause must be empty and have low confidence. 

• An inferred root cause cannot have high confidence. 

• A case marked closed in a customer relationship management (CRM) system is evaluated separately from whether the technical issue was resolved. 

These checks repair structural contradictions without requiring another model call to reinterpret the evidence. A representative KI document with selected fields looked like this:

Generating KIs only when they’re useful

After a few iterations, we realized that generating a KI for every case adds cost and noise. Many cases have little usable evidence. In some, there’s only a short intake message; in others, there’s no substantive feed or acknowledgement from a support engineer. So we added gates and filters to the workflow. For example:

• A case proceeds when it has linked engineering evidence or at least two material conversational events, including evidence of support participation.

• The workflow compares the stored KI watermark with the newest timestamp across the case and its material feed. An unchanged case reuses its existing KI instead of regenerating it every time.

• The workflow limits corpus selection by focusing generation on recent cases or those otherwise likely to be queried, especially when the source corpus is large.

• The workflow also refreshes the case KI when linked sources can change independently, rather than relying on an incomplete watermark.

Deferred cases remain available through raw lookup and can be reconsidered after new activity arrives. Provenance is part of the stored KI: References point back to linked sources, and the output records what was missing or unverifiable. The KI shortens routine investigation, while raw records remain available for current status, complete history, attachments, and evidence outside the generated snapshot.

The retrieval skill is part of the system

While workflows generate KIs, the agent requires retrieval guidance to use them effectively. To address this, we developed a retrieval skill which provides ES|QL templates for querying KIs stored in the index and guidance on when to use each retrieval method. For instance, a known case number is searched lexically because it’s an exact identifier. Questions about similar cases or precedents use hybrid lexical and semantic retrieval, while questions about official knowledge, engineering issues, comments, or other non-case entities begin with an index profile and then query the selected raw source.

The skill also assigns a clear role to each KI family. An index profile provides routing and field guidance, while a case KI provides a bounded snapshot of the evidence relevant to one case. If no case KI is available, the agent treats that as a cache miss and follows the index profile guidance back to the underlying data, rather than assuming that the available context is complete.

That distinction matters during a typical case lookup. For a question such as What is the current priority of case 1234567?, the agent retrieves the case KI by case _id to understand the investigation and its supporting evidence. It then checks the live case record for values that can change, including status and priority, along with updated_at. If the live record is newer than the KI’s last update, the live value takes precedence. The KI remains useful for durable evidence, while the source record remains the reference for current state. The next eligible refresh rebuilds the KI using timestamps from the case and its material feed, in addition to linked sources.

This separation was shaped by an evaluation failure. In an earlier version, the agent used the case KI to plan retrieval but issued a raw source query using an inferred schema instead of the fields supplied by the index profile. The query failed, causing retries and additional model work. Later versions of the skill made the precedence rules, source roles, field guidance, and recovery steps for mapping errors explicit.

What we observed 

We evaluated the agent using 20 distinct questions, with approximately 10 for each workflow: case investigation/postmortem summarization and shorter technical support Q&A across several source types. We used three trials per question to measure variation from run to run. The examples in this post were tested on an Elastic Cloud Hosted deployment running Elasticsearch 9.4.2.

Different agent configurations used the same answering model and comparable tool conditions. They varied only in whether the agent had access to KIs and, if so, which ones: index-profile KIs, case-level KIs, or both. We measured factuality against expected answers, input and output tokens, latency, and tool execution failures. A separate LLM judge scored factuality.

Case-level KIs produced the clearest signal of operational efficiency in both workflows. Relative to the raw index baseline, observed input token use and latency changed as follows:

Workflow

Input tokens

Latency

Case investigation

About 58% lower

About 40% lower

Technical support Q&A

About 43% lower

About 17% lower

We noticed that index profiles shortened source orientation but left consolidation to the agent, whereas case-level KIs supplied a compact evidence bundle aligned with the requested output. That led to fewer raw queries and reduced the opportunity for schema- and partition-related tool errors.

Factuality varied by workflow, but we didn’t observe a statistically significant reduction in this evaluation.

A note on interpreting these results: The evaluation was performed on a small dataset, which is common and useful early in the agent development lifecycle when large labeled evaluation datasets aren’t yet available. We therefore treat the results as directional signals for comparing variants. This mirrors strategies outlined in engineering blogs by Elastic and other companies, such as Anthropic: Start with a small number of tasks drawn from real failures, and then expand the suite as effects become smaller and the product matures.

Why case-level KIs fit support work

The case-level strategy matched the shape of support investigation work in several ways:

• A case number is a stable retrieval anchor.

• The expensive operation is repeated consolidation across the same source relationships.

• Much of the evidence used for explanation is durable, while volatile states can be verified separately.

• The KI schema mirrors the type of questions an engineer asks during case summary and RCA work.

• Maturity and freshness controls limit generation to cases with enough evidence and likely reuse.

Index profiles still have value. They help with source discovery, schema orientation, non-case questions, and fallback. For case investigation, they leave the cross-source reconstruction inside the response path. Case-level KIs remove part of that recurring work, which is the main reason we expected lower token use and latency in this domain.

The combined strategy exposed a composition problem

While evaluating the responses, we made a counterintuitive observation: Providing both index profiles and case KIs didn’t improve on using case KIs alone. Inspection of the trace showed that the agent understood some of the case context but lacked a dependable rule for composing the two KI families. It used a case-level KI for planning but then ignored the index profile’s schema guidance when querying raw data, leading to failures and suboptimal responses.

This resulted in a practical lesson for improving guidance in the retrieval skill: Context sources need roles and precedence. The agent must know which source can support an answer, which source only routes to evidence, when verification is required, and what to do after a cache miss or mapping error. Two individually useful context types can create additional work when those contracts are implicit.

What we learned about precomputed context

In this support workflow, the most useful unit of context was a case and the evidence connected to it: the case record, conversation, linked engineering work, and relevant knowledge. Preparing that evidence ahead of time meant that the agent didn’t have to reconstruct the same relationships for every response. Compared with the raw index baseline, the case-level approach was associated with lower observed input token use and latency, while factuality didn’t show a statistically significant reduction.

Because case state can change, the system still checks source data for values, such as priority and case status, and falls back to raw data when the available evidence doesn’t justify a case-level KI. Next, we plan to test how well this design holds with support data from external systems, such as Salesforce, and to identify any adjustments needed.

How a search is executed in Elasticsearch

Let’s start with some Elasticsearch basics. When a search request lands on an Elasticsearch node, that node is called the coordinating nodefor the search. By default, any Elasticsearch node can act as a coordinating node. The coordinating node determines which shards need to be searched based on the index or indices specified by the search. The nodes that those shards live on are called data nodes. A coordinating node can also be a data node, as it may host shards that are relevant to the search.

Elasticsearch performs the search in two primary phases: the query phase (also known as the scatter phase) and the fetch phase(also known as the gather phase). The query phase is responsible for going to the data nodes and executing the query on each shard. Each shard responds with a set of document IDs (just the IDs, no data) and an associated score for each. These results are reduced. Next, the fetch phase goes back out to the data nodes to fetch the document _source (the data). 

What exactly is a reductionin Elasticsearch? A reduction turns per-shard results from the query phase into a single merged result for the client. Suppose a search asks for the top five hits in a three-shard index, according to some relevance score. Elasticsearch must then get the top five hits from each targeted shard. Why? Because it’s possible that one shard contains the global top five, or, more likely, that the top five docs are spread across shards. 

If three shards are being searched, the coordinating node will have 15 (docID, score) pairs after the query phase. These results are reduced: The documents with the top five scores are kept and the rest thrown away. Then the fetch phase reaches back out to the data nodes to get the documents’ _source data, which Elasticsearch then responds with.

shard1_results = [(id: 231, score: 0.871), (id: 445, score: 0.812), (id: 88, score: 0.754), (id: 312, score: 0.701), (id: 567, score: 0.643)]

shard2_results = [(id: 847, score: 0.921), (id: 76,  score: 0.843), (id: 125, score: 0.783), (id: 438, score: 0.729), (id: 590, score: 0.668)]

shard3_results = [(id: 512, score: 0.887), (id: 289, score: 0.798), (id: 74,  score: 0.741), (id: 631, score: 0.682), (id: 405, score: 0.619)]

// Take the top five scores from above (that’s the “reduction”) reduced_result = [(id: 847, score: 0.921), (id: 512, score: 0.887), (id: 231, score: 0.871), (id: 76, score: 0.843), (id: 445, score: 0.812)]

What is the batched query phase?

Without batching (shard fan-out)

To understand the batched query phase, we must first understand how the query phase worked without batching. The following diagram represents a three-node Elasticsearch cluster with 12 index shards. Suppose a client search request lands on Node 1. That makes Node 1 the coordinating node for the search.

The coordinating node first fans out to all shards, performing the client’s query on each shard. Each shard query is facilitated by a transport request, which is a protocol used for communication between Elasticsearch nodes. Notice that in this diagram, each shard gets its own transport request. Node 1 holds shards itself, but no network request is needed to query those shards.

After querying the shards, the coordinating node must reduce them, as described in the previous section. At this point, the coordinating node is holding 12 shard results. Once it reduces them all, it can proceed with the fetch phase and then respond back to the client.

With batching

So what does the batched query phase change? The batched query phase first takes effect before dispatching the shard query transport requests. Now the coordinating node batches the shards it needs to query for each data node and requests them all at once.

Notice that only one transport request was made to Node 2 and one to Node 3. This is the first change in the batched query phase. Each transport request induces some overhead, so this change alone buys us our first performance improvement, reducing latency and CPU cycles spent on overhead.

The second change in the batched query phase has to do with reducing shard results. Now partial reductions occur on the data nodes themselves. For instance, after querying shards 2, 5, 8, and 11, Node 2 then reduces those results. The coordinating node, upon receiving partially reduced results from Node 2 and Node 3, must then perform afinal reduction. This is the second major enhancement we get from the batched query phase: the spreading out of reduction work across the data nodes. This reduces memory pressure on the coordinating node, which we’ll see measured later in the benchmarking section.

The benefits of batching

Fan-out (no batching) is how search worked for a very long time. It has advantages: each shard is a separate request that returns and can be retried independently from all the others. With many shards involved, though, each shard request causes overhead due to many round trips going between the coordinating node and the data nodes.

Also, the coordinating node doesn’t have enough information to be able to determine the pace at which to send requests to each data node. It sends a maximum of five concurrent requests per data node by default, where five is a bit of a magic number which allows for some parallelism, while at the same time preventing a single query from taking over an entire data node. At the same time, data nodes also don't make distinctions between the different shard requests they receive, for instance based on what parent search request they belong to. In reality, if all shard requests involving a search request are presented in one batch to each data node, the data node can then look at its internal state and adapt its pace dynamically, removing the five concurrent shard requests artifical limit mentioned above.

The batched query phase results in several benefits, including:

• Increased efficiency, thanks to fewer round trips: less CPU spent on transport overhead and fewer bytes going through the transport layer.

• Spreading out the load of reductions: the coordinating node was previously the bottleneck for reductions, and now data nodes share the work.

• Better resource usage: we may be able to better max out the data node’s CPUs.

The average search against many shards can now be served much quicker, with lower latency and higher throughput.

Batched query phase benchmarks: Latency and memory usage

To measure the benefits of the batched query phase, we ran a couple of benchmarks. The first displays the benefit of reduced transport overhead. This benchmark was built off the “many-shards-quantitative” nightly benchmark. It runs on a three-node Elasticsearch cluster, running batches of searches targeting 1,000, 5,000, and 20,000 shards. The queries are match_all queries with size: 0. That means querying the shards themselves is effectively a no-op. This benchmark is meant to isolate the work of coordinating the search across the cluster.

We ran the same benchmark with and without the batched query phase (by toggling the cluster setting search.batched_query_phase). At 5,000 shards, the batched query phase makes searches twice as fast. By 20,000 shards, it’s 2.2x faster. These results are a ceiling for what a realistic workload can expect to benefit; any real query work at the shard level will dilute the overall result. However, the gains are real, and the coordination work of your queries will benefit as shown. Note that this first benchmark did not attempt to measure any gain brought by spreading reductions across data nodes, as opposed to performing them only on the coordinating node.

Next, we benchmarked the benefits of the batched query phase on large reductions. In our benchmark, we ran a large terms aggregation over a data set called http_logs (which can be found in our rally-tracks repo). This data set has 247 million documents, which we indexed in seven indices each with 100 shards, again in a three-node Elasticsearch cluster. We ran a single terms aggregation query for the clientip field with size: 1000 and shard_size: 50000. That means we’re asking for the top 1,000 terms, although each shard will return 50,000 buckets to be reduced to that 1,000. Requesting so many terms from each shard increases precision, but has a cost in terms of memory usage and reduction overhead, which helps highlight the gain provided by spreading incremental reductions across data nodes.

The memory used is measured by something called a circuit breaker, which accounts for memory usage in Elasticsearch. This field can be found in the /_nodes/stats response under nodes.<node_id>.breakers.request.estimated_size_in_bytes. It estimates the memory in use for processing in-flight search requests.

The memory benchmark shows how the work of reductions is now spread across the cluster. In blue is the memory used with search.batched_query_phase: false. We can see that the node elasticsearch-0 is the coordinating node, as it bears the entire reduction load itself. This effect is no longer the case with search.batched_query_phase: true in red. The node elasticsearch-1 is the coordinating node, but it uses far less memory. That’s because the data nodes elasticsearch-0 and elasticsearch-2 do reductions themselves, reducing the results before they’re returned to the coordinating node.

Summary

Batching shard queries per data node reduces transport overhead and allows each data node to perform partial results reduction locally. The result is lower latency for searches targeting many shards and better distribution of work across the cluster for reduction-heavy workloads. At the same time, the new model works well for all scenarios and opens the door for potential future enhancements which we’re excited to pursue, such as:

• Optimize out-of-the-box resource usage by introspecting data nodes' activity and adjusting the pace accordingly, which would replace the five concurrent shard requests per data node artifical limit.

• Assign priorities to search requests and let each data node process shard requests accordingly.

• Reduce the number of roundtrips further by folding the `can_match` phase into the query phase. Can match is a separate "batched" roundtrip used to shortcut the query against shards that can't possibly match based on index statistics. Now that the query phase is batched, both rounds can be executed in one go.

• Discontinue support for minimize roundtrips in cross-cluster searches in favour of batched query execution; in hindsight, minimize roundtrips achieves a similar goal but is specific to cross-cluster execution, while batched execution is applicable to every search. 

We wrote about the architecture behind consolidating those shapes in Why Elasticsearch is becoming a columnar database. This post is about the other side of the ledger. What does the split actually cost, and what does "search and analytics on one platform" mean in terms clear enough to hold up in a proof of concept?

What running five data engines actually costs

Anyone who has kept two sets of books for the same business knows where the hours go. Writing the second ledger is quick, but making the two agree is what takes the week.

Each engine needs its own ingest path, so the same events get parsed and shipped twice. That gives you two sets of failure modes and two backlogs to drain when a broker slows down. It also gives you drift: A field rename lands in one copy before the other, and for a while, the two systems disagree about the same hour of data. Reconciling that disagreement is real engineering work that rarely appears in the business case.

Each engine also brings a query language, and the syntax is the small part. The cost is everything written in that language, including dashboards, alert rules, saved queries, runbooks, and the operational knowledge of the person on call this week. Two languages means two of all of it, maintained in parallel.

Then there’s correlation. You find the failing request in the log aggregator. You move to the warehouse to chart how often it happened this week and then to the metrics store to check whether the host was saturated at the time. Each of those moves is a join performed by hand, by a person under time pressure, and every one of them adds minutes to the incident.

Retention compounds all of this. Each system gets its own lifecycle policy, so the cheap system ends up keeping data that the expensive one dropped weeks ago. When you finally need the history, it lives in the engine that cannot answer your question quickly.

Why search and analytics were split across two systems

The split was a reasonable response to a real constraint. Document engines and columnar engines were built to answer different questions. A document engine is good at finding the records that match a query and ranking them by relevance, and a columnar engine is good at reading three columns out of 50 and aggregating them across billions of rows. For years, running both was the reasonable answer, because no single system was credible at both jobs. The introduction of a full columnar engine in Elasticsearch 9.5 makes the split optional rather than necessary. For the full history and what changed, check out the columnar post.

What one platform means in practice

Three things have to be true before "one platform" means anything.

1. The first is one query language. Elasticsearch Query Language (ES|QL) runs across logs, metrics, traces, security events, and documents, which means a single skill set and one set of dashboards. Because it’s one language rather than a federation of several, queries compose: A time series aggregation can sit in the same query as LOOKUP JOIN or INLINE STATS, which systems built around PromQL alone cannot do.

2. The second is one storage substrate. Doc values, the column store that has been inside Elasticsearch since 2013, is what every index mode reads and writes underneath. Various modes tune that substrate for different shapes of data. Elasticsearch's time series engine (TSDB) went fully columnar in 9.4, which is the clearest evidence so far that the approach works. Columnar Mode and Columnar Logs, both in technical preview in Elasticsearch 9.5, extend the same treatment to analytical and log-shaped data.

3. The third is one operational story. It’s the same cluster, with the same APIs, integrations, agents, access control, backup, and upgrade path that you already run.

Several index modes share one column store, and one language queries all of them inside a single cluster. That’s a narrower claim than "one engine for everything," and it’s the one that holds up when someone tests it.

When is a specialized database still the right choice?

A specialized system will always win a specialized benchmark. If your workload is one shape with a single query pattern, there’s a purpose-built engine that beats us on it, and we would rather say so than pretend otherwise. The argument for consolidation applies to data that spans more than one shape, which describes most production estates.

That said, being the general-purpose platform doesn’t mean settling for second place on every shape. We’ve climbed this curve once already. TSDB has stored metrics since Elasticsearch 8.7, and the early work concentrated on storage efficiency rather than on competing with dedicated metrics stores.

A run of releases from Elasticsearch 9.1 through 9.4 then turned it into a columnar metrics engine: OpenTelemetry (OTel) metrics now land at 3.75 bytes per data point, down from 25 a year earlier, which is 2.5x less storage than Prometheus and 2x less than ClickHouse. Gauge average and counter rate queries run up to 30x faster than Prometheus and Mimir, and on the high cardinality benchmark, Elasticsearch scans four hours of data across half a million time series in under two seconds, where the other systems needed more than 30 seconds. The full methodology and per-query results are in how we rebuilt Elasticsearch as a leading columnar metrics datastore.

Columnar Mode is at the start of that same curve. Technical preview lands in Elasticsearch 9.5 and general availability (GA) in 9.6, with phased improvements to storage, ingest, and query performance in the releases that follow. Metrics took a year of that work to get where they are, and we expect logs and analytical data to follow the same path rather than a shorter one.

Which workloads should stay on the document modes

Some workloads should stay exactly where they are. 

1. Search-first applications, like product catalogs and knowledge bases, where the answer is the 10 most relevant documents, are what the existing document modes do well, and none of those modes are deprecated. 

2. Frequent individual document updates - these workloads suit the document modes, too. 

3. Nested data models - The same is true of data models that genuinely depend on nested structure, because Columnar Mode flattens fields into key/value pairs and doesn’t support the nested field type. In Elasticsearch 9.5, semantic_text and dense_vector fields aren’t available in Columnar Mode; a columnar profile for vector retrieval comes later.

Adoption is per index and opt-in. Existing indices continue to behave as they do today, and your APIs don’t change. Plus, your dashboards don’t break.

Where Columnar Mode is today: Elasticsearch 9.5 preview, 9.6 GA

Storage and performance numbers are coming in a separate technical deep dive, and the public roadmap issues for Columnar Mode and Columnar Logs are the right place to tell us what your workload needs.

If you want a real number for your own five-tool tax, start by counting two things: ingest pipelines carrying the same events to more than one destination, and dashboards that answer the same question in two different query languages. In most estates, the second number is the one that surprises people.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Static ID lists vs. dynamic filtering with ES|QL's WHERE clause

Static ID list filtering

Dynamic filtering with WHERE IN subquery

Run query A to find the IDs you care about

Outer query asks the main question

Copy the values by hand

Subquery builds the filter list from live data

Paste them into a static WHERE IN list

WHERE field IN (subquery) applies the filter live

Run query B with the hardcoded list

Results stay current with the data

Repeat the whole process when the data changes

No copied list to maintain

Figure 1. The old copy-paste loop collapses into a single dynamic filter.

How the WHERE IN subquery replaces static filter lists

Traditional IN filtering is still useful when the list is small and static:

But many real investigations don’t start with a tidy list. They start with a question: Which users are suspicious, which hosts are noisy, which services are failing, or which accounts crossed a threshold?

That’s where the WHERE IN subquery becomes useful. The list is produced by ES|QL rather than typed by hand.

WHERE IN subquery example: filtering logs by suspicious users

Read it like this: Show me log events for users who appear in the list of users with at least 10 failed login attempts.

The outer query asks the main question, and the subquery builds the dynamic filter list, eliminating the need to copy and paste.

The subquery can target a different index or index pattern from the outer query.

Filtering without subqueries: the manual ID copy workflow

Imagine that you want to inspect traffic for the top failing services. First you run:

Then you copy the five service names and paste them into another query:

That’s fine once, but less fine when the top five change every hour.

Dynamic filtering with a WHERE IN subquery

The subquery finds the top failing services from the last two days, and the outer query returns the log events for those services. One query builds the full picture.

Excluding values with NOT IN subqueries in ES|QL

Sometimes the interesting question is about what doesn’t belong:

That pattern is useful for exclusion checks, gap analysis, and workflows that ask for the things outside an approved or expected set.

Nested subquery chains in ES|QL's WHERE clause

An IN subquery replaces the literal value list with a query in parentheses. The inner query runs first and returns a single column, and the outer WHERE filters against it. Because that inner query is a full pipeline, it can contain its own IN subquery, which lets you express a chain of lookups that would otherwise require three separate queries and two rounds of copy-paste.

Read it inside out. The innermost query finds priority regions, and the middle query finds customers in those regions, while the outer query sums revenue for those customers. Each layer is a normal ES|QL pipeline, so each one can filter, aggregate, or sort on its own before handing a clean column up to the layer above.

Combining WHERE IN subqueries with AND and OR conditions

Because an IN subquery is a Boolean condition, it composes with AND and OR like any other predicate. You can require membership in two independent sets or accept membership in either:

The AND combination finds orders placed by VIP customers for products that are being discontinued. Swap AND for OR, and you get orders that match either condition. Each subquery runs its own pipeline, so the two sets are computed independently and then combined by the Boolean operator.

Merging multiple indices into one WHERE IN subquery

The query inside an IN subquery is a full pipeline, so its FROM command can reference more than one subquery. Each branch runs its own pipeline, and the FROM command merges the rows from all branches into one result set. KEEP host_id selects the single column that the outer filter needs. This is useful when the values you want to filter against live in several indices with different schemas. For more details on how subqueries in the FROM command handle indices with different schemas, see Three indices walk into a FROM clause: ES|QL subqueries in Elasticsearch.

The IN subquery combines matching host IDs from three indices, prod, staging, and edge, into one value list. The outer query then counts alerts for any host in that combined set. Adding a fourth source means adding one more branch, with no change to the outer query.

Using an Elasticsearch subquery inside each FROM branch

A FROM subquery gives each index its own branch with its own WHERE, and that WHERE can use an IN subquery. This is how you apply the same dynamic filter across several indices that each have their own schema.

Each branch filters its index down to VIP customers before the two branches combine, so the final aggregation runs over a single normalized set of rows.

When to use ES|QL WHERE IN subqueries

• Investigations that start by finding risky users, hosts, accounts, or services.

• Operational dashboards where the interesting entities change over time.

• Top-N follow-up queries, such as events for the five noisiest services.

• Set comparison workflows, especially with NOT IN.

• Queries that would otherwise need glue code just to pass values from one step to the next.

Requirements and constraints for WHERE IN subqueries

• Return exactly one column from the IN subquery.

• Use KEEP at the end of the subquery so the comparison field is obvious.

• Make sure the outer field and the subquery field have compatible types.

• If the subquery uses SORT, add an explicit LIMIT, as unbounded SORT isn’t supported in ES|QL yet.

• Use this for membership filtering. If you need columns from both sides, a JOIN` may be the better tool.

Why ES|QL dynamic filtering replaces manual query workflows

The WHERE IN subquery turns a manual workflow into a declarative one. You can let one query build the filter for another query directly inside the WHERE command, instead of asking ES|QL for a list, copying it somewhere else, and hoping it stays fresh. 

Your WHERE clause now has a better way to handle Filter this by whatever that query finds.

If you've been exploring ES|QL, you might have noticed that a subquery used to feel like it had exactly one front door: FROM. And it made sense. Most of the time, queries start with a simple directive to go fetch documents from an index. In an earlier post, we taught the WHERE command a new trick: IN subqueries. And before that, subqueries showed up in the FROM command to combine data sources. In both instances, every subquery started the same way, with FROM.

But not every useful query starts with a bulk document fetch. Sometimes you need to evaluate time-series semantics. Other times, you just need to whip up a tiny inline row for testing. Sometimes, the absolute best input to a filter is a dynamic query that builds the list for you on the fly, rather than a static, hard-coded list.

Source command

Reads from

Best for

Requirements

FROM

Indexes, data streams, aliases, views

Stored document lookups, live filter lists

None (works with any index)

TS

Time-series data streams

Metric aggregations with counter-reset handling

TSDS with index.mode: time_series

ROW

Inline literal values

Test cases, seed values, synthetic placeholders

None (no index needed)

The FROM, TS and ROW source commands are generally available (GA) in Elasticsearch 9.5, while the WHERE IN subquery remains in technical preview in 9.5.

How ES|QL subquery source commands work

Think of subqueries as having two specific placements and three different engines.

Where subqueries go: FROM and WHERE placements

• Inside the FROM command: The subquery is an independent result source, contributing its rows to the outer query. Fields that exist in one source but not the other are gracefully filled with null values.

• Inside the WHERE command: The IN subquery contributes dynamic values to be used as a predicate or filter.

Three source commands for starting a subquery

Door #1: FROM (the classic door)

FROM is the familiar workhorse. You use it when the subquery should read stored data from indices, data streams, aliases, or views. One of its best use cases is the "stop-copy-pasting-IDs" pattern. Instead of running one query, manually copying the output values, and pasting them into the filter of another query, the subquery becomes your live filter list.

Door #2: ROW (the tiny door)

ROW is the lightweight option that requires absolutely no index setup. It allows you to build rows completely out of literal, inline values. This makes ROW useful for small seed values, test cases, allow/deny lists, or one-off "what if?" scenarios. In the query below, ROW is the perfect way to staple a synthetic sentinel row or placeholder directly onto real data.

Door #3: TS (the time-series door)

The TS command targets time-series data streams and enables time-series aggregation functions. Why not just use FROM for metrics? TS is uniquely optimized for time-series data, and it natively handles tricky scenarios, like counter resets on process restarts and uneven metric publish intervals. Using TS as a subquery lets you filter your application logs by what your metrics are saying. For example, imagine asking ES|QL: Show me log events only from clusters whose peak metric throughput crossed 800:

In this scenario, the filter is reacting to live metric behavior rather than being hard-coded.

Combining FROM, TS and ROW in one query

Because subquery placements and source commands are independent, you can freely mix and match them. You can throw all three doors into a single FROM union to generate a cohesive table containing real logs, a live metrics summary, and a synthetic row.

ES|QL subquery constraints

A few constraints to keep in mind before using subqueries:

• IN subqueries demand one column: If a subquery feeds an IN operator, it must project exactly one column. Use the KEEP command to make that explicitly clear.

• TS requires a time series data stream (TSDS): The TS command only works on data stored in a TSDS, which uses index.mode: time_series.

The takeaway: when to use each source command

Subqueries in ES|QL provide a structured way to compose queries by using the results of one query as the input to another. Choosing the appropriate source command (FROM, ROW, or TS) lets you combine data and generate inline values. It also lets you filter dynamically without duplicating query logic. For more details and additional examples, see the ES|QL subqueries documentation.

• Subquery in FROM command

• Subquery in WHERE command

What you'll discover

In this post, you'll learn how to:

• Add conversion tracking (add-to-cart and purchase spans) with search.* attributes that tie back to the originating search.

• Build a full search-to-revenue funnel: search → click → add-to-cart → purchase.

• Write Elasticsearch Query Language (ES|QL) queries to calculate conversion rates, revenue per query, and average order value.

• Identify where users drop off and which team should own each drop-off point.

• Attribute revenue to specific search queries for prioritizing relevance work.

What you'll need

• Click tracking from Blog 3 (search + click spans flowing to Elastic via OTel-native ingestion).

• Backend endpoints for add-to-cart and checkout events (example code provided).

• Basic understanding of ecommerce conversion funnels.

Why search revenue attribution matters

Your product manager walks into a meeting and asks, "Which searches are driving the most revenue?"

You can tell them what users search for (Blog 2) and what they click (Blog 3). But you can't tell them what they buy. The gap between "clicked a result" and "purchased a product" is where the business case for search investment lives, and right now it's invisible.

This post closes the loop. By adding two more span types (add-to-cart and purchase), you get a full funnel from search to revenue, built on the same search.* attributes and ES|QL queries you've been using since Blog 2.

What search conversion tracking answers for your team

Here's what conversion tracking lets you answer and why each question matters to different people on your team.

• Which searches drive revenue? This is the product manager's question. When you can attribute dollars to specific queries, you can prioritize relevance work by business impact. A query with mediocre click-through rate (CTR) but high conversion value is more important than a high-CTR query that never leads to a purchase.

• Where do users drop off? The funnel from search to purchase has four stages: search, click, add-to-cart, and purchase. Each drop-off points to a different problem. High click-to-cart drop-off suggests that product pages aren't convincing. High cart-to-purchase drop-off is checkout friction rather than a search problem. Knowing where users abandon tells you which team should fix it.

• Which queries to protect? Once you know that "laptop bag" generates $12,000/month in attributed revenue, you treat it differently. Any relevance change that touches high-revenue queries gets extra scrutiny. You can set up monitoring (Blog 6, coming soon) to alert when conversion rates drop for your top-earning searches.

Two new instrumentation points total about 20 lines of code, and they follow the same pattern as you’ve used before. You add attributes to spans, and query them with ES|QL.

Following along with code? The reference project has conversion tracking ready to enable. Uncomment the Blog 4 sections in app.py and app.js, restart, and then generate traffic with python generate_traffic.py --blog 4.

The four-stage search conversion funnel

Before we write any code, here's the shape of what we're building. Each stage is an instrumentation point, and each creates spans in traces-generic.otel-default:

The thread running through the entire chain is search.query_id. The same identifier you derived from the trace ID in Blog 2 and used to link clicks to searches in Blog 3 now carries through to add-to-cart and purchase events. This is what makes revenue attribution possible: You can trace a purchase back to the search that started the journey.

Add conversion tracking spans with OpenTelemetry

Add-to-cart span instrumentation

When a user adds a product to their cart from a search results page (or from a product detail page they reached via search), you create a cart.add span. This captures the moment that intent turns into action.

This follows the same pattern as click tracking in Blog 3; that is, an independent span linked to the originating search via query_id. The new attributes are cart.quantity and cart.price which let you aggregate revenue at query time.

Purchase span instrumentation

When the user completes checkout, you create a checkout.complete span. This is the revenue event, the one that answers the product manager's question.

• Spans capture errors automatically. This is a side benefit of using OTel spans for conversion events. If an add-to-cart or checkout call throws an unhandled exception, the span's status is automatically set to ERROR and the exception details are recorded. These errors are business-critical (a broken checkout flow means lost revenue), and they show up immediately in Elastic APM's error tracking, service maps, and alerting. You get conversion analytics and operational monitoring from the same instrumentation, with no extra code.

• checkout.total_amount is the revenue number. This is what you'll aggregate in ES|QL to get revenue-by-query. It represents the order total, not the price of a single item.

• search.query_id is conditional. Not every purchase originates from search. Users browse categories and follow promotional links. Then they return to their cart days later. The if event.query_id: guard ensures that you only attribute purchases to search when there's a genuine connection. Purchases without a query_id still get recorded; they just don't appear in search attribution queries.

• search.query is set on both cart and purchase spans. This is a deliberate denormalization. You could join back to searches via query_id to get the query text, but while ES|QL does support LOOKUP JOIN, it’s likely not the right choice here due to the need to optimize the lookup index. By putting the query text directly on conversion spans, your revenue-by-query and cart-by-query queries are single, straightforward aggregations.

Span attributes for cart and purchase events

Add-to-cart attributes:

Attribute

Type

Required

Purpose

search.action

string

yes

"add_to_cart"

search.result_click_id

string

yes

Product document ID

search.result_click_position

int

yes

Position in results when added

search.query_id

string

yes

Links to originating search

enduser.pseudo.id

string

yes

Client/device identifier (OTel Semantic Conventions  [SemConv])

cart.quantity

int

yes

Quantity added

cart.price

float

optional

Product price

search.query

string

recommended

The search query text (for cart-by-query analysis)

Purchase attributes:

Attribute

Type

Required

Purpose

search.action

string

yes

"purchase"

checkout.order_id

string

yes

Unique order identifier

checkout.total_amount

float

yes

Order total

checkout.item_count

int

yes

Number of items purchased

enduser.pseudo.id

string

yes

Client/device identifier (OTel SemConv)

search.query_id

string

recommended

Links to originating search (last search in journey)

search.query

string

recommended

The search query text (for revenue-by-query analysis)

These follow the same search.* namespace from Blogs 2 and 3, with new cart.* and checkout.* prefixes for conversion-specific data. With OTel-native ingestion, all attributes are stored under attributes.* with dot notation preserved. That means no more mapping strings to labels.* and numbers to numeric_labels.*. A search.query attribute is queryable as attributes.search.query. Likewise, checkout.total_amount is queryable as attributes.checkout.total_amount.

User identity: connecting search to purchase across sessions

You'll notice that enduser.pseudo.id appears on every span type in this series. It's the minimum identity level; that is, a persistent identifier stored in the browser's localStorage that ties events to a device across sessions.

For conversion tracking, identity becomes more important. You need to connect a search on Monday to a purchase on Tuesday, or correlate cart additions across tabs. Our schema supports three identity levels, aligned with OTel semantic conventions:

Attribute

Persistence

Purpose

enduser.pseudo.id

Permanent (localStorage)

Device/browser identifier (OTel SemConv)

session.id

Per-visit (sessionStorage)

Groups events within a single visit (OTel SemConv)

user.id

Account (auth system)

Authenticated user (OTel SemConv)

For the funnel queries in this post, enduser.pseudo.id is sufficient. It links the journey from search to purchase within a browser. If your users authenticate, adding user.id enables cross-device attribution (searched on mobile, purchased on desktop) and richer personalization. session.id helps disambiguate when the same client has multiple active sessions.

All three are optional on interaction spans. Start with enduser.pseudo.id, and add the others when your use case requires them. The important thing is consistency: Use the same identifiers across search, click, and conversion spans so the joins work.

Frontend integration

The front end needs to propagate query_id through the user journey. When the user clicks a search result, you already have query_id from the search response (Blog 3). The key is carrying it forward.

The front end sends client_id as an HTTP field name; and the back end maps it to the OTel semantic convention enduser.pseudo.id when setting span attributes. The query_id propagation is the critical piece. Store it alongside the product in the cart data structure so it survives navigation between pages. We'll discuss the design challenges of this in the attribution section below.

Using the reference project? The reference app's frontend/app.js wires up add-to-cart buttons, but the checkout flow isn’t implemented in the browser UI. It goes through the traffic generator. To simulate conversion events, run: python generate_traffic.py --blog 4 --sessions 100. This sends a realistic mix of searches, clicks, cart additions, and purchases to all three backend endpoints.

Verify that conversion events are arriving

Before building funnel queries, confirm that both span types are flowing to Elastic:

Add-to-cart events:

Purchase events:

If these return rows, you have the full funnel instrumented. If not, check the same things as always: OpenTelemetry Protocol (OTLP) endpoint, auth token, and span export.

Funnel analysis with ES|QL

Count search, click, cart and purchase events in a single query

You can count all four funnel stages in a single ES|QL query using the same COUNT(CASE(...))pattern from the CTR query in Blog 3:

Example output:

searches

clicked

carts

purchases

click_rate

cart_rate

purchase_rate

146

41

28

12

28.1%

19.2%

8.2%

You get all four counts and three conversion rates from a single query. The WHERE clause pulls all four span types into one result set, and COUNT(CASE(...)) counts each type separately. This is the same technique that made the CTR query in Blog 3 so clean.

Notice that we use search.first_clickfor the click stage rather than counting all click events. This gives you the number of searches that received at least one click (the same definition used for CTR in Blog 3). Without this, a search with three clicks would inflate the click count to three while only counting as one search, making the funnel numbers misleading. Each stage now represents a unique progression: how many searches happened, how many of those got clicked, how many led to a cart addition, and how many resulted in a purchase.

You can turn this into a funnel visualization using Kibana Lens. Run each stage count as a separate ES|QL query, save them as dashboard panels, and arrange them as a horizontal bar chart with the four stages on the y-axis and counts on the x-axis. The drop-off at each step becomes immediately visible.

The dashboard above (from the first blog in the series) shows this in practice: The Conversion Funnel panel in the lower left uses a horizontal bar chart to visualize the four stages, and the Top Queries by Revenue table alongside it shows revenue attribution. You can build these directly from the ES|QL queries in this post.

What drop-off rates tell you and who owns each bottleneck

Each transition in the funnel tells you something specific:

• Search to click (CTR): You measured this in Blog 3. Low CTR means that results aren't compelling. This is a relevance problem.

• Click to cart: The user engaged with a result but didn't add it to their cart. This could mean that the product page isn't persuasive or the price isn't competitive. It could also mean that the item was out of stock. It's often not a search problem. It’s possible that the search worked (the user clicked), but something downstream lost them.

• Cart to purchase: The user committed to buying but didn't complete checkout. Complicated forms, unexpected shipping costs, and payment issues cause this checkout friction. This is almost never a search problem, but it's useful to know where the funnel leaks so you don't waste time optimizing relevance when checkout is the bottleneck.

The diagnostic pattern is straightforward:

Drop-off point

Likely cause

Who owns it

Search to click

Relevance / ranking

Search team

Click to cart

Product page / pricing / availability

Product / merchandising

Cart to purchase

Checkout UX / payment / shipping

Checkout / growth

This is one of the most valuable things that full-funnel data gives you: the ability to point at the right problem. When the VP asks, "Why aren't searches converting?", you can show whether the bottleneck is relevance, product pages, or checkout.

Revenue attribution by search query

Here's the query your product manager actually wants:

This gives you a ranked list of queries by the revenue they generated. The search.query attribute on purchase spans (the denormalization we discussed earlier) makes this a single aggregation query, without joins or subqueries.

How to act on search revenue  data

• Protect high-revenue queries. If "laptop bag" generates the most revenue, any relevance change that affects that query gets extra scrutiny. You might add it to a regression test suite or pin specific results with query rules. You might even set up an alert when its conversion rate drops (Blog 6).

• Prioritize relevance investment. The queries at the top of this list are where relevance improvements have the most business impact. A 10% CTR improvement on a query that generates $500/month in revenue is worth more than a 50% improvement on one that generates $20.

• Identify missed opportunities. Cross-reference with the top-queries data from Blog 2. A query with high search volume but no purchase attribution is either a browsing query (informational intent) or a conversion gap worth investigating.

Top revenue queries: where to focus relevance investment

To focus your relevance team's efforts, pull the top revenue-generating queries:

Cross-reference this with the per-query CTR from Blog 3. A query with high revenue but low CTR is underperforming; even small relevance improvements have outsized business impact. A query with high CTR but no purchase attribution might be informational (such as users browsing but not buying). The queries at the top of both lists deserve the most attention from your relevance team.

Average order value by search query

Average order value (AOV) tells you how much purchases are worth, in addition to which queries convert to those purchases. Queries with high AOV are your premium-intent searches; ranking improvements there have the biggest per-purchase impact:

A query with high AOV but low volume is a different opportunity than high volume + low AOV. The first means premium intent from a small audience (consider featured results or dedicated landing pages); the second means broad reach with budget buyers (price sensitivity may be limiting conversion more than relevance).

Search revenue attribution: limitations and workarounds

Revenue attribution from search is valuable, but it's imperfect. Understanding the limitations helps you set appropriate expectations and design around them.

Last-touch attribution in multi-search journeys

Users rarely search once and buy. A typical journey might look like:

1. Search "laptop bag": Browse results and click a few.

2. Search "laptop bag leather": Refine the search.

3. Search "laptop sleeve 15 inch": Try a different angle.

4. Add to cart from the third search's results.

5. Purchase.

With the instrumentation above, this purchase attributes to the third search, the one whose query_id was on the cart item. The first two searches contributed to the journey but get no credit.

This is last-touch attribution, and it's the simplest model that works within a single query_id linkage. It's not perfect, but it's concrete and unambiguous. The alternative, that is,tracking every query_id in a user's session and distributing credit, adds significant complexity to both instrumentation and analysis.

For most teams, last-touch is a good starting point. If you need multi-touch attribution later, the raw data is there. You can query all searches and clicks for a given client_id within a time window and reconstruct the full journey:

This reconstructs a user's full search journey in chronological order. It's useful for debugging individual sessions, even if you don't build automated multi-touch attribution.

Cross-session attribution limits with query_id

A user searches for "wireless headphones" on Monday, clicks a few results, leaves, and comes back on Wednesday to buy. The query_id from Monday's search is long gone, since it was a property of that specific search request.

This is a fundamental limitation of query_id-based attribution. It works within a session (or more precisely, within the scope where the front end retains the query_id). It doesn't work across sessions.

For cross-session attribution, you'd need a different approach, typically a user-level event store where you associate product views, cart additions, and purchases with a persistent user ID and then look back in time to find the originating search. That's a more complex analytics pipeline and is outside the scope of what we're building here.

The practical impact is that your search-attributed revenue will be an undercount. Some purchases that were genuinely influenced by search won't carry a query_id. This is fine for relative comparisons (such as, Which queries generate more revenue than others?), even if the absolute numbers are conservative.

How to persist query_id from search to checkout

A few practical decisions affect how far your query_id propagation reaches:

• Store query_id in the cart. When a user adds a product to their cart, persist the query_id alongside the item. This way, even if the user navigates away and comes back to checkout later (within the same session), the attribution survives.

• Don't overwrite query_id on re-search. If a user adds a product from search A and then searches again and adds another product from search B, each cart item should keep its own query_id. The purchase event carries the last search's query_id as a summary, but per-item attribution gives you richer data.

• Accept the limitations. Not every purchase will have search attribution. Direct navigation, category browsing, promotional links, and returning customers who go straight to their cart will all produce purchases without a query_id. That's correct behavior, not missing data.

Spans vs. log events for conversion tracking

Conversion events can emit both an OTel span and a UBI-compatible log event, the same dual-signal pattern used for click tracking in Blog 3. The log event is actually richer for conversions. It can contain the full items list with per-item query_id attribution, which doesn't map cleanly to flat span attributes.

The span gives you the simple, aggregatable view (total revenue by query), and the log gives you the detailed, per-item view (which specific products from which specific searches). For the funnel queries in this post, spans are sufficient. If you need item-level attribution analysis, the log events in logs-generic.otel-default have the detail you need, and ES|QL queries them the same way, just against a different index pattern.

What's next: turning conversion data into relevance improvements

We now have the complete instrumentation picture, with four span types: search (Blog 2), search.result.click (Blog 3), cart.add, and checkout.complete (this post). These capture the full user journey from query to purchase. Every span lives in traces-generic.otel-default, and every metric is queryable with ES|QL. The search.query_id thread ties the entire funnel together.

But measuring the funnel is only half the story. The real payoff is using this data to make search better.

In a later blog in this series, we take everything we've built and turn it into relevance improvements. Click positions and conversion data become judgment lists for Learning To Rank, and per-query CTR and revenue become rank features for boosting. Plus, high-revenue queries get protective monitoring. And tools like Elasticsearch Relevance Studio give you a visual interface for tuning the searches that matter most, using exactly the data you're now collecting.

The instrumentation you've built in Blogs 2–4 is a feedback loop, beyond analytics: Measure, improve, and measure again.

Get started with search conversion tracking

• Reference project: Working code for the entire blog series; clone, configure, and run.

• Elastic Distribution of OpenTelemetry for Python: EDOT for Python.

• OpenTelemetry with Elastic: How to send OTel data to Elastic APM.

• ES|QL documentation: Query language reference.

• UBI Standard: Reference schema for search event structure.

• Query rules: Pin, boost, or exclude results for specific queries.

Why dense vectors create off-heap memory pressure that heap charts miss

Semantic search and kNN rely on dense_vector fields. Elasticsearch stores much of that data in off-heap memory. It’s related to how the Java Virtual Machine (JVM) operates, but it isn’t the same thing as heap usage.

In production, the heap versus off-heap split shows up in a familiar pattern:

• Heap looks fine for weeks, while the dense vector off-heap footprint quietly grows.

• Heap circuit breakers stay quiet or only spike late because the pressure sits outside the JVM.

• kNN search and bulk ingest slow down, queues build, and nothing on the dashboard points at vector RAM as the cause.

Heap limits protect Java allocations. They don’t tell you whether vector off-heap footprint still fits the RAM envelope that your deployment actually runs in. AutoOps already watches cluster health broadly; Vector memory pressure adds a focused read for vector-heavy nodes when memory and load signals line up.

How AutoOps measures vector RAM, heap, and headroom

AutoOps works from the same node stats metrics you already use for Stack Monitoring. For each node, it tracks three derived numbers:

Symbol

Meaning

Source (typical)

Chart (see below)

V

Vector off-heap footprint

indices.dense_vector.off_heap.total_size_bytes

First, green line

A

Available RAM in the product view

Delta between os.mem.total_in_bytes and os.mem.used_in_bytes

Second, green line

H

Headroom

A − V (headroom_bytes)

First, blue line

H > 0 means there’s a modeled runway: Vector use still fits comfortably in that accounting. H ≤ 0 means that you’re in a compression regime: Vector footprint (V) meets or exceeds the free RAM (A) picture that AutoOps can align in telemetry. On small tiers, that can be common under load. The insight emphasizes trends, growth in vector off-heap footprint, and corroborating stress, not a single negative snapshot.

AutoOps also tracks a compression regime flag (fraction of recent samples where H ≤ 0), so brief flickers don’t dominate the story (see third chart below):

How vector memory pressure detection works: Expansion and compression

Vector memory pressure is a single HIGH severity AutoOps event, which adapts to the compression regime:

• Expansion (H > 0): Emphasize shrinking headroom, hence a growing vector off-heap footprint.

• Compression (H ≤ 0): Emphasize ΔV, heap context, I/O, and latency. "Hours until H hits zero" isn’t the main narrative when headroom is already gone.

The detector requires three layers before it fires:

1. Memory carriers: Compression regime, shrinking headroom, or sustained growth in vector off-heap footprint.

2. Operational latch: Search or indexing latency versus rolling baselines, filesystem read stress (paired with latency or heap), indexing throttle, thread-pool queues or rejections, segment creep, or heap circuit breakers when paired with other stress, as circuit breakers alone don’t provide enough evidence to be escalated without corroborating stress.

3. Heap hot: Heap usage elevated versus a 24-hour rolling median on that node, so compression alone on a calm heap doesn’t fire the insight.

That pairing is intentional. Vector pressure without load might be capacity planning, and load without vector pressure might be a different root cause. Together, vector memory, operational stress, and heap heat surface the vector RAM story when the node is actually in trouble, not on every compressed mapping while the heap stays normal.

Validation: Memory pressure detection on a 4 GiB node under kNN load

We stress-tested vector memory pressure detection on 4 GiB Elastic Cloud Hosted deployments with throttled dense-vector ingest (~2,000 docs per minute) and steady kNN search (~8 queries per second). Across Hierarchical Navigable Small World (HNSW), Better Binary Quantization (BBQ) HNSW, and DiskBBQ mapping profiles over 24–48 hours:

• Vector off-heap footprint grew from near zero to about 6 GiB on the tightest runs (more than 4.7 million vectors indexed) in both HNSW test runs.

• Nodes spent most of each run in compression (H ≤ 0), which is expected when vector footprint exceeds total RAM in this model.

• Vector memory pressure stayed off while heap held near 50%, even with compression and pool stress building.

• On both HNSW and BBQ HNSW, the insight fired once heap climbed past ~75% with memory compression and thread-pool queue stress, roughly an hour before heap neared saturation. Node out of memory (OOM) and circuit breakers followed in the same window, as did slow search/indexing. As we can see on the dashboard below, performance drops drastically due to corroborating stress toward the end of the test run:

• On DiskBBQ, where compression was chronic but heap stayed normal, the insight didn’t fire,since storage rather than memory was the limiting factor. Disk and watermark signals are the right path to follow for that profile. As we can see on the screenshot below, all indicators stayed steady with constant performance throughout the test, even though we filled up the disk with more than 70 million vectors on the same instance type:

That timing is the point: Operators get a vector-first read tied to real RAM stress, with subsystem context, rather than an alert on every compressed index or only a red heap chart after the node is fighting on every front.

What to do when AutoOps raises vector memory pressure

Here’s the insight that AutoOps now raises when it detects vector memory pressure:

Recommendations in the product map to concrete actions:

1. Reduce vector footprint where quality allows: Fewer dimensions, quantized mappings, archive or split indices, reindex with a leaner mapping.

2. Tune kNN load: Lower num_candidates, reduce concurrent query rate, narrow filtered kNN where possible.

3. Consider DiskBBQ when HNSW in RAM is the bottleneck (evaluate recall/latency trade-offs for your use case). If you’re already on DiskBBQ and the heap memory is calm, treat disk and watermark insights as the primary signals. Note that DiskBBQ requires an Enterprise license.

4. Right-size RAM when vector off-heap footprint (V) trends up and headroom stays tight.

AutoOps links affected nodes and summarizes regime and stress in plain language. Treat it as “act now, rather than waiting for red on every chart.”

Where AutoOps vector memory pressure monitoring is available

Vector memory pressure is available wherever AutoOps runs against Elasticsearch 9.2+, including:

• Elastic Cloud Hosted (ECH).

• Elastic Cloud Serverless (coming soon).

• Self-managed via Cloud Connect.

AutoOps is included at all subscription levels for supported deployment types and doesn’t consume ECUs on ECH.

How MATCH and TO_TEXT enable full-text search on any ES|QL expression

Let's start with a query that was impossible in Elasticsearch 9.4, which uses the EVAL command:

In this example, summary has no mapping or analyzer configuration. It’s also not associated with any inverted index. It exists only for the lifetime of this query, but now you can search it anyway. Two additions make this work.

First, MATCH now accepts any expression as its first argument, not just a mapped field. That includes columns produced by EVAL and function results used inline. It also includes unmapped fields loaded directly from the original document. Furthermore, all data types normally accepted by MATCH are supported in this new use case.

The second part of this is the new TO_TEXT function, which is the first ES|QL conversion function that produces output of type text. Until now, text columns could only come from indexed mapped fields, and all strings produced by ES|QL expressions were keyword values rather than text. The distinction matters because MATCH treats the two differently: text values are analyzed, while keyword values are compared exactly, mirroring how a MATCH query on an indexed keyword field rewrites to a term query. TO_TEXT(x) is how you tell ES|QL: treat this string as full text.

This ships as a technical preview in Elasticsearch 9.5, and as such, it has some limitations:

• It’s currently filtering only. A MATCH on an expression doesn't contribute to the relevance score yet; only matches on indexed fields affect the score.

• Querying options like fuzziness and others aren't yet supported when matching an expression.

• Runtime text is analyzed with the standard analyzer. This isn’t configurable yet.

Work is underway to address these limitations.

Why use full-text search instead of LIKE or RLIKE in ES|QL?

ES|QL already had two ways to search strings without an index: LIKE (wildcard patterns) and RLIKE (regular expressions). Both work on any string expression, so it's fair to ask what MATCH adds. The answer is analysis, a more advanced form of search which uses techniques such as stemming and synonyms. It also uses stopword handling.

LIKE is simple substring matching, without any understanding of the words that comprise a string. Say, for example, you're looking for log messages about a fox:

This misses "Fox spotted near the henhouse" due to the capitalization, while matching "Outfoxed by the competition", which isn't about a fox at all. It fails in both directions, with false negatives on capitalization and false positives on substrings buried inside other words.

Regular expressions can patch the case problem, but the word-boundary problem gets ugly fast. Something like:

And even that's not right yet. It misses a fox at the end of a sentence followed by ! or ?, and it says nothing about tabs, quotes, or parentheses. Each fix makes the pattern longer, and the next person to read the query has to reverse-engineer what it’s actually doing.

MATCH makes the problem go away, because it runs both the query and the value through an analyzer, which tokenizes text into lowercase terms and then matches term against term:

This query will match values like "The quick brown fox" and "FOX spotted near the henhouse" but not "Outfoxed by the competition" or “FOXTROT protocol enabled", regardless of any punctuation surrounding the words. Of course, this all works for multi-term queries, like MATCH(TO_TEXT(message), "brown fox"), too, just the way you’d expect it to.

Work is underway to enable the use of the 36 dedicated language analyzers, with support for natural languages on data that was never indexed or mapped.

Full-text search use cases for unindexed and unmapped data

The examples above searched values computed from mapped fields. . The more interesting use cases for ES|QL MATCH on expressions involve data that was never searchable at all. Let's walk through a few.

How to search unmapped fields in ES|QL without adding a mapping

Sometimes you deliberately leave a field out of your mappings, such as a verbose stack trace or a raw request payload. You might even leave out a debug blob. Indexing one of these would cost disk and heap space on every document and wouldn’t be worth it for a field you might query once a quarter.

That decision has always been final, because unmapped fields were invisible to queries entirely. In Elasticsearch 9.5, you can use SET unmapped_fields="load" to make ES|QL load unmapped fields directly from the source document as keywords. Follow that up by wrapping it in TO_TEXT, and now you can run full-text search on it:

Here, stack_trace was never mapped. Every value is fetched from the original documents and analyzed on the fly. They’re matched row by row. That’s real work, and it will never be as fast as an inverted index lookup. But now, that field you didn't index is no longer unsearchable. You get to keep the mapping small for the everyday case and still answer the once-a-quarter question when it matters.

Full-text search on a keyword field without reindexing

Keyword fields can do a lot. They give you exact matching, fast aggregations, and sorting, which is why so many fields end up mapped that way. But mappings are decided when data arrives, and it’s easy to end up in a situation where you want to do something different with your data than you had originally intended. Maybe product_name was mapped as a keyword because the dashboards aggregate on it, and then, after receiving a year’s worth of product data, someone wants to be able to search within product_name values.

The old answer was to change the mapping to text (or add a multi-field) and reindex everything. This can be both time-consuming and costly, and in many cases, users simply won’t want to bother with it. The new answer is one function call:

TO_TEXT converts the keyword values to text on the fly, so MATCH analyzes them instead of comparing them exactly. This allows you to query a keyword field without creating a mapping or reindexing the source document. If the search becomes an everyday query, indexing the field as text is still the right long-term move, but TO_TEXT gets you an answer today, without any extra work.

Searching the same field across indices with different mappings

ES|QL can span many indices, and the same field doesn't need to always look the same in all of them. When the same field has different types in different indices, ES|QL treats it as a union type, and a conversion function resolves the conflict. Let’s consider an example in which the message field has type text in this year's index template but was a keyword in last year's:

Every value is analyzed at query time, whether it came from the text index or the keyword one. The keyword values from the older indices are tokenized and lowercased like everything else, so "connection reset" finds “Connection RESET by peer”, no matter which index it lives in.

Another interesting case is when a field is mapped in only one index but also present (and unmapped) in the other:

There's a nuance worth calling out here. If error_details is mapped in logs-2026 but not in logs-2025, Elasticsearch cannot push this query down to Lucene, because the indices where the field is unmapped would silently return no matches. Instead, the planner notices that the field is potentially unmapped and evaluates the whole MATCH row by row, wherever the rows came from. You don't have to know which of your indices have the field mapped; the query just answers the question.

How ES|QL analyzes text at query time without an inverted index

When ES|QL plans a MATCH against an expression, it analyzes the query string once, up front, into a set of terms. How each row is then evaluated depends on the expression's type:

Expression type

Processing

Matching behavior

text (via TO_TEXT)

Analyzer tokenizes value into lowercase terms

Token-against-token comparison; a row matches if any token equals any query term (OR semantics)

keyword,ip, date, numeric

No analysis; query constant converted once to the native type

Exact comparison per row

Both paths bypass Lucene entirely and evaluate values row by row. The non-text path mirrors exactly what a match query does when pushed down to Lucene against those field types, so the semantics stay consistent regardless of whether your query hits an index.

An inverted-index lookup does its work at ingest time and never touches non-matching documents at query time. A runtime MATCH does that analysis at query time, for every row that reaches it. One is fast because the work already happened; the other is flexible because the data doesn't need to have been indexed at all.

What's next for ES|QL full-text search

Everything in this post is the first installment of a larger effort to make search in ES|QL work on anything, not just on what you indexed ahead of time. The limitations called out earlier are actively being worked on, and the roadmap goes further:

• Scoring. Runtime matches will contribute to _score, so you can sort by relevance even when the data was never indexed.

• MATCH_PHRASE on expressions. Already available in Elastic Cloud Serverless, and coming to the Elastic Stack in 9.6.

• Configurable analyzers. Analyzer support for MATCH and MATCH_PHRASE on expressions, enabling language analyzers, stemming, and synonyms at query time.

• Match options. Options like fuzziness and operator for runtime matches.

• Vector search. Generating embeddings per row and running k-nearest neighbors (kNN) on runtime dense_vector expressions, bringing semantic search to unindexed data, too.

Try ES|QL full-text search on expressions today

You can try runtime search today. It's available now in Elastic Cloud Serverless, where new ES|QL capabilities land first, and it ships as a technical preview in Elasticsearch 9.5. Start with the search functions reference, and check the ES|QL limitations page for the current boundaries. It's a technical preview because we want your feedback: If you search something that was never indexed and it surprises you, either positively or negatively, we'd love to hear about it.