A sorted ES|QL query over 346 million log documents

Using real-world Nginx logs, the benchmark searched 346 million docs across 432 GB of total index data. It returned 500 results and took ~500 seconds. That's more than eight minutes.

What’s happening under the hood?

Before this optimization, candidate rows were loaded, decoded, and evaluated by the user filter before TopN could determine which of them could no longer contribute to the final result.

How min-competitive TopN prunes work in ES|QL

TopN learns something useful while the query is still running. Once its heap is full, the worst value currently in the heap becomes a competitive threshold: that is, any row that cannot beat that value cannot make the final result.

ES|QL can feed that threshold back into earlier stages of execution. Two changes make this work:

1. Dynamic Lucene pushdown. As TopN finds better candidates, its competitive threshold improves. ES|QL pushes that threshold down to Lucene, allowing Lucene to skip candidates that cannot beat the current worst TopN value. This avoids loading, decoding, and evaluating rows that have no chance of making the result.

2. Shared threshold across drivers. ES|QL processes data in parallel across multiple drivers. If each driver only uses its own local threshold, one driver cannot benefit from better candidates discovered by another. Sharing the best-known competitive threshold means a strong result found by one driver immediately helps all the others prune their remaining work.

Together, these changes move TopN from a late-stage filter to something that shapes query execution much earlier in the pipeline.

How does the min-competitive TopN optimization work?

The query asks for the top three rows matching a wildcard filter, sorted in descending order, across three segments of 10 rows each. Without the optimization, all 30 rows are loaded, decoded, and evaluated against the filter before TopN sees any of them, at a cost of 30 row loads and 15 TopN operations.

With dynamic pushdown, ES|QL maintains the TopN heap as it scans. The heap fills at the end of the first segment with [98, 51, 47], and its worst entry is pushed down to Lucene as the competitive threshold. That cuts the second segment to four rows out of 10, which improves the heap to [98, 83, 77] and tightens the threshold again, leaving just two rows to load in the third segment. This produces the same three results, with 16 row loads and 10 TopN operations.

Sharing the TopN threshold across drivers

One driver may find excellent recent candidates immediately. Without coordination, another driver doesn't know that and continues working with a weak threshold. Sharing the global worst competitive value lets that discovery benefit all drivers.

Benchmark results on a local node and Elastic Cloud Serverless

The min-competitive TopN optimization ships as generally available (GA) in Elasticsearch 9.6. For real-world Nginx data benchmarks (same query, same dataset, p50 of 51 measurements, with 20 warmup runs discarded):

Local node

Serverless (GCP)

Before optimization

~500s

~520s

Production implementation

~5s

~14s

Runtime improvement

~99%

~97%

Speedup

100x

37x

Benchmark hardware

Local node

Serverless (GCP)

CPU

Apple M4

Small GCP host

RAM

48 GB

8 GB

Java Virtual Machine (JVM) heap

24 GB

4 GB

Storage

Local external TB5 NVMe SSD

Cloud storage

Shards

100

100

When does this query optimization technique help most?

The biggest gains happen when TopN becomes competitive early. If good candidates appear at the start of execution, the threshold rises quickly and subsequent scans get pruned aggressively. Three conditions amplify this:

1. N is small relative to the candidate population. Finding 500 rows out of 346 million documents gives the optimizer far more room to eliminate work than returning a large fraction of the matches does.

2. The post-Lucene work is expensive enough that skipping a row saves something measurable, like avoiding loading large strings, decompression, or wildcard evaluation.

3. Many candidates pass the coarse Lucene scan in the first place. If almost nothing matches the initial filter, there’s not much unnecessary work left to eliminate.

If matching rows are rare or if good candidates are discovered late, the threshold takes longer to become useful and the gain can be much smaller.

The current implementation applies to @timestamp sorted queries. Queries sorted by other fields don’t yet benefit from this optimization.

What's next for ES|QL query performance

The same idea has room to run in several directions, including:

• Other numeric sort fields. The underlying approach isn’t inherently limited to timestamps, so extending it to other numeric sort fields could benefit a much broader range of sorted TopN queries.

• Processing promising data first. How quickly the threshold becomes useful depends on how early TopN sees good candidates. Processing more promising data slices first could establish a strong threshold sooner, allowing more aggressive pruning of subsequent work.

• Early termination. As the threshold strengthens during execution, it becomes possible to recognize when remaining ranges cannot produce a better result and to stop processing them entirely, rather than running through to completion.

• Coordination across nodes. The current optimization shares competitive information across drivers within a node. Broader coordination across nodes or clusters could let strong candidates discovered in one part of a distributed query reduce work elsewhere.

• Late materialization and result streaming. Competitive filtering and late materialization attack the same problem from different angles. Competitive filtering eliminates rows that cannot reach the final TopN as early as possible, while a fetch phase can defer loading expensive field values until the set of likely winners is much smaller. Result streaming could extend this further. If the engine can determine that some leading results can no longer be displaced, those rows could be returned before the full query completes. For interactive investigations, getting the first useful result fast matters as much as total query runtime.

What this means for Elasticsearch performance tuning

The min-competitive TopN optimization in Elasticsearch 9.6 shows how much work can be avoided when information discovered during query execution is fed back into earlier stages. In our Nginx log benchmark, that turned a roughly 500-second query into a roughly 5-second query on a single node.

This is also a starting point. Extending competitive filtering to more sort fields, finding strong candidates earlier, terminating work sooner, and eventually coordinating these decisions across nodes offer opportunities to push the same idea further.

The earlier we know what can still win, the less work we need to do for everything that cannot.

We started from Text-to-ES Bench (ACL 2025), which measures how well large language models query Elasticsearch using Query DSL. Its models pair a DSL query with Python post-processing, and pandas assembles the multi-index answers. We wanted to see what changes when the model can join inside the query. LOOKUP JOIN joins indices natively, so a multi-index question becomes one statement that Elasticsearch executes end to end.

The dataset: BIRD benchmark, Mini-Dev set

Text-to-ES Bench sourced its questions from BIg Bench for LaRge-scale Database Grounded Text-to-SQL Evaluation (BIRD), a widely used text-to-SQL benchmark. We do the same, using BIRD's Mini-Dev set: 500 natural-language questions over real relational databases. Every task gives you three things:

1. A question in English: The prompt that the model translates.

2. A gold SQL query: The reference query that produces the answer.

3. The returned rows: The answer key that you grade against.

The part we rely on is that the ground truth is the rows, not the SQL. We never compare the generated query text against the gold SQL; a prediction is graded only on the rows that it returns. A query that doesn’t parse returns no rows, and no rows is a zero. "The names of the three drivers with the shortest average pit stop" should produce the same answer, whether you computed it in SQL, ES|QL, or by hand. So we keep BIRD's questions and BIRD's answer key and swap only the language that the model writes in.

Here’s one task, from the student_club database:

That [["Sacha", "Harrison", 866.25]] is the answer key. We also pass through BIRD's "evidence" hints (the full name refers to... line above) exactly as the dataset provides them; the questions are written assuming that you have them.

Getting BIRD into Elasticsearch

Each table becomes its own index, deliberately not denormalized. Flattening the schema in advance would quietly answer the hard part of the question for the model, and we would end up measuring our data modeling rather than its querying.

Indices on the right side of a LOOKUP JOIN must use lookup index mode:

And anything you join or group on needs to be a keyword rather than a text field, so text columns become keyword with an ignore_above guard. That keeps them usable in WHERE equality, STATS ... BY, LOOKUP JOIN, and LIKE, while staying under Lucene's term limit:

All in, that’s roughly 3.9 million documents across 75 indices, running on Elasticsearch 9.5. If you haven’t built a join like this before, this walkthrough of native joins in Elasticsearch covers the index-mode requirements in more depth.

How we prompted the models and scored ES|QL accuracy

The prompt mirrors BIRD's zero-shot protocol: a schema block, the evidence hint, the question, and an instruction to return only the query. The one deviation is that the schema is presented as Elasticsearch index mappings instead of CREATE TABLE DDL, because the target language is ES|QL.

Every query gets one attempt. That’s deliberate, because one call with one prompt isolates what the model knows from what a scaffold could recover.

We ran three prompt variants against four models:

1. base: Schema, evidence, question; whatever ES|QL the model already knows.

2. focused skill: The above, preceded by a compact subset of the ES|QL skill: its SKILL.md overview plus the language reference, the generation tips, and the query patterns. The parts that are irrelevant to relational queries, time series, PromQL, and full-text search are left out.

3. full skill: The same, but with the complete skill attached and every reference file included.

In both skill variants, the files are pasted into the prompt as a static block. A skill normally reaches the model through a trigger, an agent deciding it needs the reference and loading it. We cut that step out so the variable under test is the content of the reference.

The models tested were gpt-5.5, claude-opus-4-8, claude-sonnet-4-6, and gpt-5.4-mini. 

Scoring runs the generated ES|QL through the _query API and compares the result to BIRD's gold rows as a set. Row order, duplicate rows, column order, and extra returned columns are ignored. One standard: the query returns the right data.

We decided to ignore the presentation details because they say nothing about whether the model understood the question. Take the student_club task from above and a model answer that a strict tuple comparison rejects:

The model built a full_name where the gold SQL kept first_name and last_name apart. They produced the same data, same rows. Extra columns are the same class of problem and are more common; a model that answers KEEP atom_id, element when the question only asked for the element has still found the element.

How accurate is LLM-written ES|QL?

Execution accuracy, 500 questions per cell:

Model

base

focused skill

full skill

gpt-5.5

59.0%

48.8%

50.4%

claude-opus-4-8

40.4%

50.0%

49.0%

claude-sonnet-4-6

30.2%

39.2%

39.6%

gpt-5.4-mini

19.8%

30.4%

31.6%

Two things stand out:

1. The reference is worth about 10 points to every model except the strongest: Opus, Sonnet, and gpt-5.4-mini gain between 9.0 and 10.6 points from the focused skill; gpt-5.5 loses 10.2. It was already at 59.0% with nothing but the schema, and it misformed only 7 of 500 queries, so it had no grammar problem for a reference to fix, and handing it one cost it more than it could possibly win.

2. The gains are smaller than the error counts suggest: The skill nearly eliminated the biggest failure in the run: across all 6,000 queries, join errors fell from 939 to 354. Aggregate accuracy didn’t move. The reason shows up one column over, as a class of error that barely existed before: Found ambiguous reference went from 120 to 706. The errors weren’t fixed so much as renamed, and we take that apart in the next section. Documentation teaches a model the grammar of ES|QL; it doesn’t teach it the shape of your data.

Four ways that ES|QL generation breaks

The percentages below are out of the misses. A miss is any query that didn’t come back with the right data, whether it failed to run or it ran and returned the wrong rows. Getting the data right and the shape wrong doesn’t count.

Failure class

What to do about it

Join key names differ on each side

RENAME before the join, match foreign key names to primary key names, or use the 9.2 join predicate

SQL syntax the ES|QL parser rejects

Put a one-page syntax card in the prompt

Counting after a one-to-many LOOKUP JOIN

COUNT_DISTINCT on the entity, or aggregate before the join

The model guessed a value that isn't in your data

Sample rows and distinct values in the prompt

1. ES|QL LOOKUP JOIN needs the same field name on both sides

Join key naming is the largest failure class: 134 of Opus's 298 base misses, roughly 45% of them. SQL joins columns with different names, and BIRD's schemas rely on it (superhero.skin_colour_id joins to colour.id). ES|QL's bare join form does not. LOOKUP JOIN <index> ON <field> takes a single field name that must exist on both sides, closer to SQL's JOIN ... USING than to JOIN ... ON. Elasticsearch 9.2 added a second form that lifts this restriction, but only when the two keys have different names.  That condition is where our run went sideways, and we come back to it at the end of this section.

A model reaching for SQL join syntax writes this:

Give it the skill, and it learns the ON <field> form and then trips one step later, still reaching for the left-hand name (expense.link_to_budget points at budget.budget_id):

Across the run, 212 join queries failed to parse at all, the SQL-shaped ON a = b among them, and another 226 parsed but were rejected for the right-hand name. There are three ways out, depending on what you control.

1. Rename at query time. One line before the join:

One wrinkle: RENAME replaces the target column if the name you rename to already exists on the left. Joining superhero to colour is exactly that case, since both carry an id, so move the collision out of the way in the same command (renames apply left to right):

2. Fix it in the data model. Give foreign keys the same name as the primary key that they point at, and the problem disappears permanently. It’s cheap at design time, which is why we call it a modeling habit.

3. Use a join predicate (Elasticsearch 9.2 and later). Elasticsearch 9.2 added complex join predicates that compare differently named fields directly, the way that SQL taught you to:

Every name in a predicate must be unambiguous, so this form doesn’t rescue the case where the key already exists on both sides. That case is most of BIRD, and it’s where our edited skill backfired. Telling the models to prefer the predicate over RENAME dropped gpt-5.5's use of RENAME from 47% of its joins to 11%, and the error that RENAME was preventing showed up in its place:

Across the run, that error went from 120 to 706, while the join bucket fell from 939 to 354; almost exactly a wash. The 9.2 release write-up covers the new join forms.

2. SQL syntax the ES|QL parser rejects

The models that struggle most are the ones reaching for SQL habits. This is where gpt-5.4-mini lost 236 of its 500 base queries and Sonnet lost 181.

The model writes

ES|QL wants

WHERE x = 5

WHERE x == 5

WHERE name = 'Bob'

WHERE name == "Bob"

CASE WHEN x > 1 THEN 'a' ELSE 'b' END

CASE(x > 1, "a", "b")

COUNT(DISTINCT id)

COUNT_DISTINCT(id)

DIVIDE(a, b), YEAR(d)

These do not exist

A correlated subquery

Restructure with STATS and a join

Single quotes are the sneakiest of these, because in ES|QL, double quotes delimit strings and single quotes do not. Here Sonnet reached for both habits at once, a bare = and a single-quoted string, and the parser stopped at the quote:

This is the one category that in-context documentation helps with. Sonnet's 181 syntax errors became 86, and gpt-5.4-mini's 236 became 150. If you’re pointing a smaller model at ES|QL, a one-page syntax card is the single highest-return thing that you can put in the prompt. A full language reference isn’t better.

3. Counting across a one-to-many LOOKUP JOIN

LOOKUP JOIN fans out the way that a SQL join does; when a row on the left matches several rows in the lookup index, you get one output row per match. A model that forgets this counts the joined rows instead of the entity that the question asked about. This is around 32% of the runs-but-wrong bucket.

Join a member to their expenses, and you get one row per expense, so a plain count answers How many expenses, when the question asked How many members:

COUNT(*) here counts expense rows. The fix is to count the entity that you actually mean or to aggregate before the join rather than after:

4. The model guessed a value that isn't in your data

The fourth failure class is the model inventing a value that looks plausible and isn’t what’s stored.

BIRD's databases are full of these. A transaction type is stored as 'VYBER', not 'withdrawal'. Ask for withdrawals, and gpt-5.4-mini writes the English word, which matches nothing:

The query is valid ES|QL and runs cleanly. It just comes back empty, because nothing in that column says withdrawal. The same pattern shows up across the dataset:

• A lab result is 'negative', not '-' or false

• Dates frequently live as keyword strings rather than date fields, which breaks any date function that the model reaches for and accounts for roughly 17% to 24% of wrong results on its own

• Ranking questions confuse position with a stored rank column

A schema block cannot fix this, because a schema tells you that a column is a keyword and never tells you which keywords are in it. The fix is sample values in the prompt: a handful of rows or the distinct values of low-cardinality columns. This is the failure class where retrieval helps and documentation does not.

Conclusion: What this means for text-to-ES|QL in production

We draw three conclusions from the run:

1. The newest models already write good ES|QL: Single call, zero shot, nothing but a schema, and the strongest model still gets the data right on most questions, while almost never writing one that fails to run. ES|QL isn’t exotic to frontier models anymore.

2. A new language feature doesn’t automatically become a win: Elasticsearch 9.2's join predicate removes the restriction behind our largest failure bucket, and pointing the models at it did cut that bucket from 939 errors to 354. Accuracy didn’t move, because the models spent the savings on a new error. A feature only pays once the guidance says when not to use it.

3. Prompt additions pay off for every model except the strongest: A compact syntax reference is worth around 10 points to a smaller model, a fuller one doesn’t help more, and the frontier model needs neither. Give it advice it didn’t need, and you can take 10 points off it.

Roughly half of the failures throw a precise, actionable Elasticsearch error (Unknown column [gender_id] in right side of join tells you exactly what to fix), so a single retry with the error text appended should recover a large share of them when using an agent like Elastic Agent Builder. 

The other half fail silently on guessed values, and no retry helps; those need sample rows and value lookups in the prompt. Grammar is no longer the ceiling. Knowing your schema and your values is, and that’s a retrieval problem, a much more tractable one.

Resources

• Browse the benchmark harness and all 6,000 scored queries, or filter them in the live interactive report

• LOOKUP JOIN command reference, including multi-field joins and the predicate form

• The Elasticsearch 9.2 ES|QL release write-up for the new join forms

• Text-to-ES Bench (ACL 2025), the Query DSL benchmark that this work builds on

If tracing is on, the agent-builder-traces skill is already loaded on every agent in your space and there’s nothing to install. The first post in our series covered enabling OTel tracing and the out-of-the-box (OOTB) dashboards, along with threshold alerts. This post is about asking.

What is the agent-builder-traces skill?

agent-builder-traces is a built-in Agent Builder skill. It takes a natural-language question, generates an ES|QL query from it, executes that query against your trace index, and returns a plain-language summary. The index it targets is traces-agent_builder.otel-<space-id>, where <space-id> is the Kibana space that you’re working in. Using the exact space-scoped index pattern, rather than a wildcard, keeps data from other spaces out of your results.

Under the hood, the skill uses a single inline tool: agent-builder-traces.generate_esql. Instead of calling this tool directly, you ask a question and the agent calls the tool on your behalf, passing your question as the prompt for ES|QL generation. The tool resolves the current space's trace index, builds a query using the default model, executes it against Elasticsearch, and returns the result.

Which privacy settings control what your traces capture?

Several fields containing sensitive information are off by default. You can enable them in Gen AI Settings (Stack Management > AI Assistants), under the "Agent Builder Traces" section. Expand the Advanced privacy settings to find them.

Setting key

What it captures

agentBuilder:tracing:includeUserPrompts

User messages

agentBuilder:tracing:includeLlmResponses

Large language model (LLM) response text

agentBuilder:tracing:includeToolDetails

Tool call arguments and results

agentBuilder:tracing:includeSystemPrompt

The agent's system prompt

agentBuilder:tracing:includeRealNames

Real tool, agent, and conversation names (hashed when off)

agentBuilder:tracing:includeRealIds

Real conversation and workflow IDs (hashed when off)

agentBuilder:tracing:includeUserData

Real user IDs and usernames (hashed when off)

Why is my trace analysis query returning empty rows?

If the skill returns empty rows or tells you that a field is unavailable, check two things: whether the relevant setting is enabled in your configuration, and whether your query time window overlaps with any recorded spans. The skill reports exactly what the query returned; it doesn’t fabricate content when fields are empty..

What agent performance questions can you ask?

Examples of questions that you can ask include:

How many tokens have my agents used, by model?

Ask: How many input and output tokens have my agents used in the last 24 hours, broken down by model?

ES|QL: 

The TO_LONG() cast around each token field is required. These fields can surface as mixed integer and long types across index generations, and ES|QL’s SUM() needs an explicit numeric conversion before it can aggregate them. If you write your own queries against this index and hit unexpected type errors, this is usually why.

Which tools are failing most often?

Ask: What is the error rate for each tool over the last 7 days?

ES|QL: 

The tool error rate query aggregates across spans named ‘execute_tool’ and compares status.code == "Error" counts to total counts per tool name. The result shows which tools are failing most often. 

Which conversation turns are slowest?

Ask: Show me the 10 slowest conversation turns in the last hour.

ES|QL: 

The skill queries spans where span.name LIKE "invoke_agent *" and attributes.elastic.inference.span.kind == "CHAIN". These correspond to individual conversation turns, from the moment a user sends a message to when the agent returns a response. Durations are stored in nanoseconds, so the skill converts to seconds before sorting.

What have users been asking my agents?

Ask: What questions have users been asking in the last 30 minutes?

ES|QL: 

The recent-prompts query returns data only when agentBuilder:tracing:includeUserPrompts is set to true. When the setting is off, the attributes.gen_ai.input.messages field is empty and the skill will suggest that you check your Include User Prompts privacy setting.

When should you use Discover or Kibana Lens instead?

The skill targets one index pattern: traces-agent_builder.otel-<space-id>. Two scenarios fall outside that boundary.

Can the skill query my own application indices?

If you want to run ad hoc questions against data that you've indexed yourself, use a general data exploration skill or write ES|QL directly in Discover. The agent-builder-traces skill won’t query outside the Agent Builder traces index.

Can the skill build or edit dashboards?

The skill focuses on ad hoc queries and generating summary text. If your goal is to build a permanent visualization from your trace data, use Lens or the dashboard-management skill instead. We covered that specific process in depth in the first post in our series.

The skill sits alongside two other ways of watching your agents.

Tool

Best for

You use it when

agent-builder-traces skill

Ad hoc agent performance questions

You notice something and want an answer without leaving the chat

OOTB dashboards

Ongoing visibility

You want trends over time without asking anything

Threshold alerts

Automated monitoring

You define a condition once and get notified when it's breached

How to build evaluation pipelines from agent trace data

Spans in traces-agent_builder.otel-<space-id> capture real execution data from every conversation turn and LLM call running in your space. That record is the raw material for evaluation pipelines: automated checks on response quality and latency budgets per agent type, along with regression detection when you update a system prompt. The next post in our series covers how to build those eval loops from trace data that Agent Builder is already collecting.

Doc values were built to make aggregations, sorting, and grouping possible on a document engine, and they do that job well. Columnar Mode changes what the columns are for, and this post walks through the five properties that separate storing columns from being a columnar database.

Five properties separate a column store from a columnar database

Doc values were an optimization on top of _source

For most of the last decade, the original JSON document was the source of truth and the columns were a derived convenience. That ordering has consequences throughout the engine.

Because the engine could always fall back to _source, per-field storage was allowed to be lossy. Text fields had no doc values at all, since text could be reread from the stored document when needed. Even synthetic source, which reconstructs a document from its fields rather than storing a copy, sometimes reads from row-shaped structures to stay faithful to the JSON that arrived, with values that exceed ignore_above and fields that arrived unmapped going into stored fields.

The result is a clear contract; whatever JSON you send, you get back, and the columns accelerate everything else. For an engine whose job is to return your documents, that’s the right way round. Columnar Mode inverts it. Every field stores itself exactly once as doc values, doc values cannot be turned off, text fields get doc values, too, and the document is reconstructed from the columns when something asks for it.

How dictionary encoding handles high-cardinality data

Sorted doc values, the default for keyword fields, store a dictionary of distinct values plus one ordinal per document pointing into it. This is an excellent trade when values repeat. A host.name field drawn from a hundred machines, or a status code that’s almost always 200, compresses beautifully and groups quickly.

It works like the index cards in a warehouse. When 50 crates hold the same product, one card and 50 pointers beats writing the product name 50 times. When 50 crates hold the same product, you can store the product name in the index with the list of 50 crate IDs. When every crate holds something unique, you might as well just put the product name on the crates; the index will help you find what crate you want, but it won't save ink.

High-cardinality fields describe a lot of real data, including URLs and trace identifiers, along with message bodies. Columnar Mode, which skips the dictionary and compresses the values in blocks instead, uses binary doc values for high-cardinality strings. Which of the two a field gets isn’t something you configure. The engine decides per field, based on the values it sees, so each column is encoded for the data it actually holds instead of one default applied to every field. Pure columnar systems have long carried cardinality in their type system, but usually as something you declare, and you own the consequences when the data shifts underneath it. Here, it’s the engine's job.

Why every field builds an inverted index by default

By default, a keyword field also builds an inverted index and a numeric field also builds a BKD tree. That happens on every field because at write time the engine doesn’t know which capability you’ll want at read time, significantly increasing the footprint of each field. Those structures also have to be rebuilt during segment merges, which costs CPU exactly when ingest is heaviest.

Our time series engine (TSDB) is proof of what happens when you stop paying for capability that the workload doesn’t use. Replacing the indices on @timestamp and dimension fields with doc value skippers, which are sparse structures holding the minimum and maximum value for each block of documents, removed 10 bytes of the original 25 bytes per OpenTelemetry (OTel) data point. There was no measurable query regression on time range and dimension filters, and indexing CPU dropped by about 10%, as a bonus.

Columnar Mode generalizes that default. Fields aren’t indexed unless something needs them to be, with only text-mapped fields keeping their inverted index for fast free-text search.

Metadata fields like _id and _routing were row-shaped

The fields you never think about followed the same document-first design. The _id field was a stored field plus an inverted index. Custom _routing was a stored field. Sequence numbers were kept for optimistic concurrency control, regardless of whether a workload ever updated a document.

TSDB deals with all three. It synthesizes _id from the _tsid and @timestamp values that already identify a data point, using a segment-level bloom filter to catch duplicates, which removes 5 bytes per data point with no loss of functionality. It trims sequence numbers once replication no longer needs them, which removes 4 bytes. Add a codec block size increase from 128 to 512 elements for another 2 bytes, and those four changes contribute across versions 9.1 through 9.4 to the 21 bytes that took OTel metrics from 25 bytes per data point down to 3.75.

Columnar Mode makes those ideas general rather than metrics-specific. By general availability (GA), all metadata fields will store themselves as doc values, while we plan to follow up and add a sort id mode synthesizing the identifier from the index sort fields, in addition to derived fields that will generalize what _tsid does for time series to any set of fields.

Columnar query execution in the ES|QL compute engine

A column store only pays off if the engine reads it as columns. Aggregations inherited the document-at-a-time shape from search, which is the natural fit for an engine built around documents. Reading columns instead lets the engine hand a whole block of values to a single instruction, and that’s where the numbers below come from.

The ES|QL compute engine changed that shape, and TSDB again shows the size of the effect:

• Vectorized execution of time series aggregations was worth up to 8x on its own.

• Decoding on-disk data straight into the primitive arrays the engine aggregates over, with no intermediate copies, was worth roughly another 10x.

• Constant blocks turned repeated values into a form of in-memory run-length encoding.

• Filter pushdown moved filters down to Lucene, where skippers can discard whole blocks unopened.

Together with the rest of the block-level query work, query latency improved by up to 160x compared to earlier versions.

That work continues. Skipper-aware operators, aggregations that group on ordinals and convert to real values as late as possible, and richer per-block summaries are all in progress, and they benefit every index mode because every mode reads doc values underneath.

What Columnar Mode changes for logs and analytical data

Storing values in columns is a storage detail. A columnar database needs five things:

1. The columns are the only copy of the data.

2. Each column is encoded for the data it actually holds.

3. Metadata is columnar, too.

4. Fields add indices only when something needs them.

5. The query engine processes blocks of values rather than records or documents.

TSDB reached all five for metrics in Elasticsearch 9.4, which is why the numbers in this post come from metrics rather than from a slide. Columnar Mode applies the same treatment to logs and security telemetry, along with analytical data. It’s in technical preview in Elasticsearch 9.5, with GA targeted for 9.7.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

PromQL compiles to the same compute engine that runs ES|QL and inherits its planner and distributed execution, along with its release process. We didn’t build a second engine for this, and there’s no plugin to install.

This post is about how we built it.

Key takeaways:

• One engine: The implementation combines Elasticsearch’s mature distributed planning, storage, and testing infrastructure with its newer compute engine, which provides a columnar execution runtime. This lets PromQL reuse proven Elasticsearch capabilities while executing through a modern, native vectorized pipeline rather than introducing a separate runtime.

• One server: Elasticsearch implements the Prometheus remote write and query APIs directly, so Prometheus-compatible ingest and queries run without any additional plugins.

• Engineered for efficiency: Supporting PromQL required new engine primitives for range-aligned evaluation grids, backward-looking windows, dynamic label grouping, pipeline result reshaping, and compact wide aggregation keys. These primitives allow PromQL queries to execute efficiently end to end, with the relevant semantics implemented directly in the compute engine rather than through external post-processing.

• Compatibility measured in real use: In addition to Prometheus compliance tests, we built a differential-testing and quality-control pipeline over 2,000 PromQL queries collected from public repositories. 

Read more:

• Query Prometheus Metrics in Elasticsearch with PromQL

• Ship Prometheus Metrics to Elasticsearch with Remote Write

• Bringing Fire to Elasticsearch: Adding Native Prometheus APIs

Why run PromQL on Elasticsearch

Many teams already store logs and traces in Elasticsearch while running metrics in Prometheus or another dedicated metrics back end.

Prometheus and its ecosystem are strong and widely adopted, but large deployments can also bring operational sprawl and scaling challenges, along with limited retention. 

So we set out to combine Elastic’s highly optimized time-series database (TSDB) with a best-in-class metrics ecosystem. The result is a smaller observability stack, with fewer systems to operate and metrics storage that scales horizontally and supports long-term retention.

One engine: PromQL and ES|QL share the same compute engine

We made an early architectural decision not to run a separate PromQL engine next to Elasticsearch.

PromQL is instead another front end to the Elasticsearch compute engine.

This puts PromQL in the normal Elasticsearch development lifecycle. It uses the same planner, distributed execution engine, testing infrastructure, and release process as Elasticsearch itself.

To learn more about Elasticsearch’s query engine, check our blog.

Like ES|QL’s time series queries that use the TS source command, PromQL is translated into a highly optimized query plan and executed across the cluster. Nodes process columnar batches through vectorized operators, while partial results move through exchanges until the final result is assembled. 

This also means that PromQL and ES|QL operate over the same execution engine and time-series data. ES|QL can additionally extend a PromQL computation with post-processing that PromQL doesn’t support, such as lookup joins and inline aggregations.

For example, assume Prometheus request counters are stored in metrics-* and keyed by the service label. A lookup index, service_registry, maps each instance to its owning team and environment and to its service tier:

This architecture requires the execution engine to support PromQL semantics natively and efficiently rather than ES|QL syntax sugar. The following sections describe the changes and new execution primitives we introduced to achieve that.

One server: Prometheus remote write and HTTP API built into Elasticsearch

Query execution is only half of the story. The Prometheus ecosystem also expects familiar ingest and query APIs.

Prometheus protocols are the de facto standard for everything metrics in almost every team’s observability stack. So we built the HTTP API directly in the Elasticsearch server, which eliminated the need for a third component and tightened integration stability and performance.

On the ingest side, we added an endpoint for the Prometheus remote write protocol. It accepts Snappy-compressed Protocol Buffer messages, maps labels to TSDS dimensions, maps the metric name/value into metric fields, infers counter versus gauge mappings, and writes directly into TSDS. The built-in template is dynamic, so users don’t have to predeclare every Prometheus label or metric.

On the query side, Elasticsearch exposes Prometheus query APIs. A request enters through the Prometheus endpoint and is executed in the compute engine.

Running PromQL efficiently in a columnar engine

Sharing an execution engine doesn’t mean treating PromQL as syntax sugar over ES|QL. PromQL has different time, grouping, and response semantics, as well as workload characteristics that matter at scale. Supporting it efficiently required extending the compute engine rather than compensating in the API layer.

PromQL time grids: Aligning evaluation steps with TSTEP

Time-series query engines are optimized for grouping over time.

Elasticsearch normally groups timestamps with TBUCKET(...), which truncates each timestamp to a fixed interval boundary. Truncation is cheap and produces deterministic bucket boundaries. It also makes intermediate results easier to reuse.

Prometheus defines evaluation points differently. For a range query, timestamps are laid out as fixed steps anchored to the query range, rather than derived by truncating each sample timestamp. Two queries with the same step but different range boundaries can therefore produce different evaluation grids.

To preserve these semantics, we introduced TSTEP(...), which derives its grouping grid from the query range and step,  rather than truncating timestamps to globally aligned boundaries.

PromQL uses TSTEP(...) internally, preserving Prometheus timestamp semantics while still lowering the operation to a native Elasticsearch execution primitive.

Some might see this as a simple problem, but the nuances matter. 

Take, for example, a query that finds a 5m rolling average of a metric: 

At evaluation time T, the result represents the average over the preceding five-minute range: 

(T - 5m, T]

When the query is executed with a five-minute step, each output value is labeled with the upper end of its corresponding five-minute window.

Elasticsearch previously lacked this semantic and supported only forward-looking window aggregation functions:

We rewrote the window-evaluation path so that both ES|QL and PromQL use a common backward-looking windowing implementation:

Together, TSTEP(...) and backward-looking windows preserve the two time semantics that matter for PromQL range evaluation.

Dynamic label grouping: How PromQL without() resolves at runtime

Time grids determine when a PromQL expression is evaluated. Aggregation determines which input series are combined and which labels identify each output series.

For most analytical query engines, that identity is known when the query is planned. The planner can allocate grouping columns and choose an aggregation strategy. It also carries a fixed key through the execution pipeline.

That’s how ES|QL works:

The output series are grouped by an explicit key (cluster, namespace).

PromQL can express the same operation in the opposite direction:

Now we know which dimensions not to use. We don’t necessarily know the full grouping key until the query is executed. This is a small language difference with significant execution consequences. 

One possible implementation is to discover every label used by the metric, subtract instance and pod, and rewrite the expression into an ordinary by(...) aggregation. That adds a discovery phase before planning. It also becomes inefficient for high-dimensional metrics where only a subset of all possible dimensions may have useful value in a particular series. Most queries need only a small subset of the available dimensions, so carrying the entire dimension universe as an aggregation key wastes memory and adds bookkeeping overhead.

We instead extended the time-series execution path with dynamic grouping columns. The engine loads dimensions as the series are read and applies the exclusions per time series. This avoids making the grouping schema a prerequisite for planning and avoids carrying a large sparse set of grouping columns through aggregation.

Dimension packing: Keeping wide PromQL grouping keys cheap

The idiomatic way of writing PromQL aggregations involves heavy use of without(...) over by(...):

Excluding labels rather than explicitly listing them makes dashboards and alerts resilient to schema evolution. If a new label is added to the metric, the query continues to preserve it unless it’s explicitly excluded.

The consequence for the engine is that effective grouping keys can be wide. Many of those labels often have low cardinality, yet each still participates in every aggregation stage.

In a columnar engine, each grouping column is normally represented as a separate vector. Ten grouping labels therefore mean 10 vectors flowing through every aggregation operator:

Dimension fields are declared in the index mapping; the planner knows the schema up front, and grouping keys stay narrow and predictable.

In the columnar engine, each grouping label is carried as a separate vector or block. A key with 10 labels therefore requires 10 vectors to be read, hashed, compared, and retained by aggregation operators. As key width grows, so does the amount of data and bookkeeping that must move through the pipeline:

To avoid paying that per-column cost for every additional label, we introduced dimension packing. Before aggregation begins, the engine encodes the full grouping key into a single compact representation. Hash and comparison operations run on the packed key rather than on each block independently:

Packing lets hashing and comparison operate on a single compact key rather than an increasing number of grouping blocks, making aggregation overhead less sensitive to key width. Because the engine is shared, ES|QL time-series queries will benefit from this optimization as well.

Building the Prometheus HTTP API response inside the pipeline

Unlike Elasticsearch's ES|QL column-oriented response format, the Prometheus response is row-oriented.  The Prometheus API returns one result row per time series, with its samples represented as timestamp-value pairs. 

To support a compatible API layer, we had to regroup in the HTTP layer converting the columnar results into boxed row objects and accumulate them in map- and list-based structures until the complete Prometheus response could be produced. 

We replaced this with the TimeSeriesCollapse compute operator. It groups rows by series and aligns samples to the query’s fixed step grid. It emits the reshaped result as ordinary columnar pages containing one row per series, with aligned multi-valued timestamp and value blocks. And it preserves compact, vectorized block representation throughout the pipeline: 

The HTTP layer can now serialize those blocks directly, avoiding the maps, lists, boxed objects, and associated allocations required by the earlier implementation.

Testing PromQL compatibility against 2,000 real queries

Prometheus compliance tests were our starting point.

Even though they gave us a strong baseline, they didn’t tell us how frequently individual PromQL features appear in real workloads. To complement that baseline, we built a second test corpus from over 2,000 PromQL queries collected from public repositories. 

We then classified those queries by the language features and expression patterns they exercise:

For each compatible query shape, we run the same query against Elasticsearch and Prometheus and compare the results. 

In addition to that, we actively rely on fuzz testing, which catches issues that unit tests alone are unlikely to expose, including differences in timestamp alignment, label retention, aggregation behavior, range-vector evaluation, and response encoding.

Which PromQL functions and APIs are supported in 9.5

Since 9.4 (technical preview), PromQL support in Elasticsearch has expanded substantially. In Elasticsearch 9.5, both the PROMQL command in ES|QL and the Prometheus HTTP APIs are generally available (GA), with more than 80% of the PromQL workflows in our real-world corpus now running without modification:

The main additions since technical preview are:

Feature

Example

Status

Prometheus remote write ingest

POST /_prometheus/api/v1/write

GA in 9.5

Range queries

/api/v1/query_range

GA in 9.5

Instant queries

/api/v1/query

GA in 9.5

Metric metadata and build info

/api/v1/metadata, /api/v1/status/buildinfo

GA in 9.5

Native histogram functions

histogram_quantile, histogram_count, histogram_sum

GA in 9.5

Per-selector offset modifiers

[5m] offset 1h

GA in 9.5

Top-level or operator

rate(a[5m]) or rate(b[5m])

GA in 9.5, up to eight operands

Prometheus remote write ingest

Elasticsearch accepts Prometheus remote write (v1) messages directly:

Snappy-compressed Protocol Buffer messages are decoded, and labels are mapped to TSDS dimensions. Metric names and values are written into the time-series index. The built-in template is dynamic, so users don’t have to predeclare every Prometheus label or metric.

Range and instant queries through the Prometheus HTTP API

Both range and instant query endpoints are available:

Range queries return matrices evaluated over a time window, and instant queries return vectors evaluated at a single timestamp. These endpoints can be used by Kibana, Grafana, or Prometheus-compatible alerting tools, and custom dashboards.

Metric metadata and build info endpoints

Elasticsearch exposes metadata about available metrics and a build-info endpoint:

The metadata endpoint returns metric types and help text, and the build-info endpoint returns the Prometheus-compatible server version. Grafana and other tools use these endpoints for feature detection and UI behavior.

Native histogram functions: histogram_quantile, count, and sum

Elasticsearch supports the main PromQL operations over native histograms:

Native histograms adapt their bucket layout to the data, providing useful precision across a wide value range without requiring users to configure every bucket boundary in advance. Classic histograms continue to work alongside native histograms.

Per-selector offset modifiers in PromQL

Offset modifiers shift a selector’s time window backward:

This returns the request rate from one hour earlier. Per-selector offsets are commonly used to compare current traffic, latency, or resource usage with an earlier baseline, such as the same period one week ago.

Top-level or operator in PromQL

Elasticsearch supports the top-level PromQL or operator:

In PromQL, or isn’t a Boolean operation. It performs a union between two sets of time series. Results from the left side are retained; a series from the right side is added only when its label set doesn’t match a series already returned by the left side. This is useful during migrations where the same logical metric may exist under an old and a new name.

The implementation follows Prometheus’s left-side precedence rules and preserves the __name__ label. Top-level chains of up to eight operands are supported.

PromQL features not yet supported in Elasticsearch

GA doesn’t mean complete PromQL compatibility. Some less common and more complex parts of PromQL remain unsupported. These gaps now define the next phase of the work: 

Feature

Example

Status

Advanced vector matching

on(instance) group_left

Planned

Sorting and ranking

topk, bottomk, limitk, sort, sort_desc

Planned

Label manipulation

label_replace, label_join

Planned

Absolute time modifier

@ 1710000000

Planned

Mixed-offset compound expressions

rate(...) - rate(... offset 1h)

Planned

Alerting and target endpoints

/api/v1/alerts, /api/v1/targets

Out of scope

Advanced vector matching with on() and group_left

Some binary operations that require Prometheus vector matching aren’t yet part of GA.

For example, this query divides per-instance request rates by a per-instance capacity metric:

The on(instance) clause specifies which labels identify matching series. group_left permits many request-rate series to match a single per-instance capacity series, while retaining the labels from the higher-cardinality left-hand side.

These expressions are common when joining a detailed metric with metadata or a lower-cardinality capacity metric. Basic binary expressions are supported where applicable, while the remaining vector-matching forms are planned work.

Sorting and ranking: topk, bottomk, and sort

Prometheus sorting and ranking functions are also not yet part of GA:

This returns the 10 services with the highest request rate. Similar queries are widely used in “top offenders” dashboards for traffic, latency, errors, and resource consumption.

The remaining functions include:

Label manipulation with label_replace and label_join

PromQL can construct or rewrite labels during query evaluation. These functions are particularly useful when dashboard variables, naming conventions, or label schemas don’t match exactly:

This creates an environment label from the cluster label.

Another common example combines existing labels into a display-oriented label:

This produces a target label, such as payments/api-7f6d9. label_replace(...) and label_join(...) aren’t yet included in GA.

Advanced time modifiers: The @ modifier and mixed offsets

Several advanced time modifiers and expression forms remain outside the GA scope.

For example, an absolute @ modifier evaluates a selector at a fixed Unix timestamp rather than at the query’s normal evaluation time:

This is useful for comparisons against a fixed historical point.

PromQL also permits expressions in which the two sides use different offsets:

This compares current traffic with traffic one hour earlier. Per-selector offset is available in GA, but not every combination of offsets and compound expressions is part of GA yet.

Prometheus API endpoints not yet implemented

In addition, the Prometheus HTTP API surface isn’t yet fully complete. Notably:

Alerting metadata through:

used by tools that inspect active alert state.

Target discovery through:

used to inspect scrape targets, health, and labels.

These endpoints concern Prometheus server and scrape-target state rather than querying metrics stored in Elasticsearch.

For the full list of limitations, see the PromQL limitations page. 

Try PromQL in Elasticsearch 9.5

To query Prometheus metrics in Elasticsearch 9.5 or Serverless, see the PromQL documentation and the Prometheus HTTP API reference.

Columnar mode is Elasticsearch's analytics-optimized columnar storage mode, built for scan-heavy workloads, like log analytics. In this mode, keyword fields don't get an inverted index by default, so term and wildcard queries scan doc values. DocValuesSkippers (zone maps) already trim how much data a scan touches, but these rewrites cut the cost of what's left. 

How Lucene's query rewrite mechanism works

In Lucene, every query has the option to implement a rewrite method that returns another query. This method returns a query with the same semantics but a different implementation. The query engine repeatedly calls the rewrite method until the returned query doesn’t change. This final query is the one that’s actually evaluated. Importantly, the rewrite can see the actual query arguments and specialize the implementation based on these.

For example, in a query looking for documents where a string field contains the value "foo", the rewrite method knows that the term we’re searching for is "foo". In theory, rewrite could replace the general query class with something specific to "foo". For example, the original query class ScanningBinaryDocValuesTermQuery could be replaced with FooQuery. Now this rule probably wouldn't be helpful, but it gives a sense for the level of specialization that’s achievable with rewrite rules.

Rewrite rules and query optimization in database systems

It's worth placing rewrite rules in the larger context of database systems. Lucene and Elasticsearch aren’t the first systems to use transformation rules to optimize queries. Most (or maybe all) database systems use some kind of rule system during query optimization. The most influential rewrite rule system was in IBM's Starburst database. This system's core contribution was extensibility; for example, it was possible to add new data types and storage methods, along with (most importantly to us) optimizer rewrite rules.

Each rule consisted of two parts:

1. A condition function: A predicate determining whether the rule applies to the current query graph.

2. An action function: The transformation that rewrites the query plan into a more optimal form.

A rule engine applied matching rules until a stopping condition was met.

Though Lucene's rewrite method is superficially different from these condition and action functions, it achieves the same goal. It checks whether certain conditions match, and if they do, it applies the rewrite by returning a new query. If conditions don’t match, the rewrite returns this, replacing the query with itself; that is, choosing not to apply the rule.

Why these rules live in Lucene, not the ES|QL query optimizer

Elasticsearch actually contains a separate rewrite rule system within the Elasticsearch Query Language (ES|QL) optimizer. This operates on the high-level structure of a query; for example, doing predicate pushdown to avoid unnecessary computation on documents that will be filtered out. But it’s still useful to have the rule system within Lucene. Since Lucene acts as the storage layer for ES|QL (and classic _search) queries, it’s easier to express rewrites that take advantage of the physical data format in Lucene rather than in a higher-level optimizer.

A query rewrite rule for wildcard queries: Simpler code, no automaton

Wildcard queries support the ? and * operators to match any character once or any character multiple times. These operators can appear any number of times in a wildcard query. As with regexes, to evaluate whether a string matches a wildcard query, we build an automaton from the query string and then use the string bytes to do state transitions through the automaton. This is relatively fast, but if you have to evaluate it for every document, the latency really adds up.

But maybe we don't always have to run an automaton. Consider a query like *foo*. How would you implement this if you were writing a simple query engine to find matching strings in a list of strings? Pretty much every programming language has the tool you want built in: a method that finds a substring within a given string. This function doesn't need a complicated automaton; it probably just consists of a couple of for loops.

Now of course we couldn't use this function to implement an arbitrary wildcard query, but we don't have to. The rule rewrite system isn't for the general form. It's for implementing special cases, and it can see the specific query. It knows that we’re looking for *foo* and realizes that this specific case doesn't require the heavyweight automaton machinery. And it can do the same for any query that starts and ends with a *, with some term in the middle.

The following pseudo-code shows the pattern. At the top, we have the generic WildcardQuery. It has two notable fields: the query string (for example, *foo*) and the automaton built for that query. The matches method checks whether the field value for a given docId is a match by using it to evaluate the state transitions of the automaton. More interestingly, its rewrite method checks whether the query matches our special case. We show this with a regex that checks whether the query string starts with a *, has any non-*characters at least once, and then ends in a *. If so, we return the special case as a ContainsQuery and pass in the inner query string (since it doesn't care about the *s). The ContainsQuery then just does a simple contains check to see whether the term bytes are somewhere within the value bytes.

Benchmarking the wildcard rewrite on ClickBench Q20

The wildcard rewrite is straightforward, but does it actually work? Yes, we can use the ClickBench benchmark, which has several queries of this form. Query 20 (Q20) is FROM hits | WHERE URL LIKE "*google*" | STATS count = COUNT(*). It's exactly the query shape that this rule matches: a string match against the wildcard query *google*. And since the query is just counting, we can see exactly how well this technique works. It turns out to be quite effective. Q20 saw a 1.75x improvement on median latency of hot query times, with no filter cache. All benchmarks in this post were run on an Intel Core i9-13900H.

Adding SIMD to the substring search: 1.75x to 2.3x

But can we do better? Yes, switching to a simple contains check opens up a new possibility. Instead of using the two for loops, we can swap scalar logic for single instruction, multiple data (SIMD) logic. Elasticsearch uses the Panama vector API (see our post on SIMD in Elasticsearch), which lets us implement the contains check in SIMD. This works particularly well for longer strings that can take advantage of the wide SIMD registers; for strings under 24 characters, we still use the scalar approach. With this change, we saw another 1.32x improvement, resulting in a total speedup of 2.3x over the automaton-based approach.

A query rewrite rule for empty strings: Less data, no decompression

One benefit of Lucene-based rules is that they’re low level and can fit to the data format. That’s the case for this rule, which applies to string data. 

How columnar storage encodes string data

In Elasticsearch's standard mode, string values are stored by document; this is a row-major format. But in columnar mode, unsurprisingly, the data is stored in columnar-major format. A column of string data is stored in chunks. Each chunk contains many string values and consists of an array of integer offsets and a (Zstd-compressed) blob of the strings' bytes. For a string at index i, offsets[i] points to the offset in the decompressed byte blob where the string starts. So the length of string i can be computed from offset[i+1]-offsets[i]. (There's a dummy extra offset at the end, so we can easily compute the length of the last string). The following diagram shows how a chunk with the strings ‘Feta’, ‘Asiago’, ‘’, ‘Stilton’, and ‘Brie’ is encoded.

Why a term query has to decompress the chunk

Now that we understand the columnar format, let's get back to query optimization. First, consider a term query for the query foo. We’re looking for documents where a given string field exactly matches the string foo. So how do we implement this on a string column in the above format? The algorithm is straightforward:

The bottleneck is the Zstd decompression step. But there's not much we can do about that; if we want to check the bytes, we have to decompress the chunks. But remember, we aren't trying to optimize the general case, we’re looking for special cases. (In reality, you don't just try to think up special cases. These optimizations came about by first running a useful query, realizing that it could be faster, and then looking for ways to improve it.)

Rewriting the empty string query as a length check

One special case we found that’s worth improving is a query for the term "". Admittedly, it's a silly term, but empty strings are all over the place. Since they're rarely useful, we usually filter them out with a query like term != "". Thankfully, this is a query we can optimize.

Consider the above algorithm for the empty string term. The line if value == term is a bit weird; we’re asking Does this value equal the empty string? We can do that, but there are no bytes to compare, so the check unwinds:

1. We only need to know whether the value has length 0.

2. If we only need the length, we don't need to look up the value in the decompressed chunk.

3. If we never look up a value, we don't need any bytes from the chunk at all.

4. If we need no bytes from the chunk, we don't need to decompress it.

All we need are the lengths, and those live in the offsets array. It's compressed, too, but with cheap integer compression rather than Zstd, which is much faster.

With this realization, we can rewrite empty string term queries. The one new operation we need is docValues.loadLength(docId), which reads directly from the offset array without touching the compressed bytes. After the previous example, this should look familiar. The most interesting part is TermEqualsQuery.rewrite; it finds the empty string special case and replaces the query with the simpler version that only checks the length.

Benchmarking the empty string rewrite: 1.6x faster

Now let's see how this stacks up. There aren't any pure-scan ClickBench queries that use this rule as directly as Q20 does for the previous rule, so we'll make our own. Consider the query: FROM hits | WHERE SearchPhrase != '' | STATS count(*). On this query, we see a 1.6x speedup, which is a great improvement for a fairly uncomplicated change. Better yet, ES|QL can take advantage of loadLength directly. Any time that ES|QL accesses a string's BYTE_LENGTH, without needing the string itself, the request uses this same specialized length loading to avoid unnecessary decompression.

What makes a good query rewrite rule

The two rules covered here follow the same shape: identify that a query is a special case, and then swap it for a cheaper implementation. But they reduce cost in different ways. 

Wildcard rule

Empty string rule

Query shape detected

*term*

field == ""

Replaced with

SIMD substring search

Length check on the offsets array

Cost reduced

Algorithmic work

Data access

Speedup

2.3x

1.6x

The underlying pattern is worth noting: finding a query that leaves performance on the table, finding a special case that can be optimized, and swapping in a cheaper implementation. The hard parts are finding queries that uncover these opportunities for optimization and then identifying the special cases. The actual fix is often relatively straightforward, as both rules here show. Our work on columnar mode has provided many opportunities to run interesting queries and hunt down exactly these kinds of wins.

That's also why extensibility in a rule system is so important. These rules can't be built into a database from the start; they're found through an incremental discovery process. Lucene's rewrite system makes that practical. As columnar mode grows to handle new workloads, rules like these will keep emerging.

To try columnar mode and the optimizations described in this article, use Elastic Cloud Serverless or Elasticsearch 9.5 or later, where columnar mode is available as a technical preview.

This can be a lot of effort, time, and context switching for what should be just a glance. In this blog, we explain how Elasticsearch Query Language’s (ES|QL’s) SPARKLINE works, what it does, and how to get started.

How it works

SPARKLINE is an ES|QL aggregate function with a straightforward signature:

SPARKLINE(aggregation, key, buckets, from, to)

• aggregation: Expression that calculates the y-axis value, including any supported aggregation: COUNT(*), SUM(bytes), AVG(latency), or others.

• key: Date expression from which to derive buckets.

• buckets: Target number of buckets.

• from / to: The time range boundaries. (In Kibana, they bind to the time picker via query parameters.)

Under the hood, SPARKLINE buckets the time range, computes the aggregation per bucket, and packs the results into a single ordered array. Empty buckets are zero-filled so every group shares the same x-axis grid for fast, easy visual comparison.

The function composes naturally with STATS ... BY, so you can combine it with any grouping.

Where sparklines shine

The first place you'll see SPARKLINE in action is Discover's log pattern analysis, starting 9.5. When you run a CATEGORIZE query, Discover constructs the SPARKLINE query under the hood and renders trend lines next to each pattern. You don't write SPARKLINE yourself here; Discover handles it when you use the “identify patterns” option in the ES|QL editor. The result is immediate: You scan dozens of log patterns and instantly see which ones are spiking right now versus which have been steady all day.

Here’s the query under the hood:

Consider a platform team investigating how they can cut logging costs. They point Discover at tens of millions of documents and let CATEGORIZE cluster them into patterns. Two patterns rise to the top: verbose lifecycle messages like "fetching resource..." and "completed resource...", each with several millions of hits. The sparklines next to those rows tell the rest of the story: flat, steady streams running around the clock. Not incident-driven. Not bursty. Just constant noise, silently consuming hundreds of terabytes of storage.

The fix in cases like this is usually straightforward: Adjust log levels, drop the pattern at ingest, or route it to a cheaper tier. The hard part was always finding it. With log pattern analysis and inline sparklines in ES|QL, that discovery takes seconds instead of hours. 

Elastic’s internal site reliability engineering (SRE) teams routinely use log pattern analysis successfully, using every ES|QL enhancement. 

These are just some concrete examples. Break down request counts by region, and see which regions are trending up. Compare latency across container IDs. Monitor queue depths by consumer group. The uses are endless.

Simple by design

We deliberately kept SPARKLINE focused:

• It's an aggregate function, not a new command. It composes with existing STATS ... BY syntax, so there's nothing new to learn structurally.

• It returns data that the consumer can render in context. The function produces an array of values. How those values are rendered, as a mini-chart in Discover, a line in a notebook, or a JSON array in an API response, is up to the consumer.

• It fills empty buckets. Every group gets the same number of values aligned to the same time grid. This is a deliberate choice: Sparklines are most useful when you can compare shapes across rows at a glance, and that requires consistent alignment.

The pattern is always the same: one query, many trend lines, instant visual triage.

What's next

SPARKLINE ships as a technical preview in Elasticsearch 9.5. Future work includes rendering sparklines in more ES|QL surfaces, beyond the initial integration, with the CATEGORIZE context in Kibana. This includes dashboards but also Elastic Observability use cases like the following:

In application performance monitoring (APM) workflows, engineers routinely analyze rate, errors, and duration (RED) metrics to understand service health. The challenge is that aggregate numbers hide dimensional outliers. A service might look healthy overall, but one region, one container, or one newly deployed version could be quietly degrading.

Today, the Elastic APM UI lets you break down metrics by transaction name, but root-cause analysis requires slicing by arbitrary dimensions: availability zone, service version, container ID, cloud region. SPARKLINE can make this practical. Break down error rate by service.version, and instantly see which version's trend line diverges from the pack.

Get started

SPARKLINE is available in Elasticsearch 9.5 as a technical preview. Try it with the ES|QL _query API or in Kibana's Discover. Check the SPARKLINE function reference for the full syntax and supported types.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Managing context depends on good retrieval. Rather than have agents rediscover the same content for every question, burning tokens by retracing similar steps over and over again, Elastic’s agentic AI capabilities enable us to precompute these details and store them in a structured, searchable form, and they let agents load that context directly. We call this precomputed unit of context a Knowledge Indicator.

The default agentic retrieval augmented generation (RAG) pattern does the opposite. It retrieves whole documents and dumps them into the model's context at query time, paying for that retrieval in tokens and latency on every single question. Precomputing the answer as a KI moves that cost out of the hot path and does it once.

How it works: AI Index, Kibana Workflows, and the query-ki skill

Building context through AI indices has three main parts: the AI Index (a special Elasticsearch index where KIs live), Kibana Workflows to create your KIs, and a query-ki skill to help agents directly query KIs using ES|QL: 

This blog post is similar to Part 1 in that we’re using the same core building blocks. But in this post, we’re demonstrating a very different use case. Instead of precomputing index metadata, we’re distilling specific facts from our indexed documents that may be used to directly answer agents’ questions without subsequent searches. We've also provided a notebook, if you'd like to create the same KIs yourself, end to end, as you go through these examples. 

Prerequisites: Elasticsearch Serverless and an LLM API key

This tutorial assumes you have:

1. An Elasticsearch Serverless project. You can sign up for a trial if you don't have one.

2. An API key to access your Elasticsearch project.

3. An OpenAI-compatible large language model (LLM) API key, to access AI indices via Deep Agents scripts.

Load the BrowseComp-Plus sample corpus into Elasticsearch

First, we’ll need some sources. Sources can be data that already exists in your Elasticsearch indices or external data accessed via connectors or ES|QL data sources.  For this blog, we’ll create an index, browsecomp-plus, to hold our example data, with the following mappings:

and populate it with a small sample of BrowseComp-Plus data via the _bulk API. You can use the supporting notebook to load a sample of this data in your project. 

Create the AI Index that stores your KIs

Just like in Part 1, the first step is to create an AI Index:

This is preconfigured with the same required mappings as we listed out in Part 1. We perform hybrid search here using semantic_text out of the box.

How agents retrieve KIs using ES|QL

A KI is a document in the AI Index. What makes KIs useful is retrieval, or querying the AI Index to find the right content. This query is packaged within a small, portable skill that’s harness-agnostic and can be run in any agent harness. 

Here’s a sample query-ki skill:

Save this asskills/query-ki/SKILL.md.

Here’s what this skill is doing: 

• We’re defining corpus_entry as our KI use case.

• We’re performing a hybrid ES|QL search on our AI indices, filtering by the appropriate type, using reciprocal rank fusion (RRF) as the default method to fuse results.

• The KI results will directly ground the agent’s answer when determining what facts are relevant to the users’ query.

When we say that AI indices and KIs are harness-agnostic, it’s because the skill is just instructions plus a query. It will work in Elastic Agent Builder, a Kibana workflow agent, Claude Code, or any other harness. We’ll be using Deep Agents for examples of how to query it outside the Kibana ecosystem. Since an AI Index is, at its core, an Elasticsearch index, you can also explore your data directly. 

Precompute facts as KIs for agentic RAG

In this example, we extract actual facts so agents can retrieve an answer without consuming a full document. We generate one fact-based KI per selected document, though the actual number and structure of KIs you generate are completely customizable.

We'll use a sample of the BrowseComp-Plus corpus, indexed into a browsecomp-plus index, with docid, url, title, and text fields.

Baseline: Retrieving whole documents with RRF

As a baseline, here's a simple RRF query:

This drops several hundred words of raw body text into the model's context. It may work, but it's expensive, and the cost compounds with every miss.

Build the Kibana workflow

The workflow below reads a batch of documents with a single ES|QL query and writes one fact-level KI per document into the AI Index. Each iteration runs two steps: generate_ki distills a raw document into a structured KI, and sink_ki writes it to the AI Index keyed on docid so reruns are idempotent.

Copy and paste the following YAML into the Elastic Workflows editor:

Here’s what this workflow is doing: 

• query_corpus runs an ES|QL query against the browsecomp-plus index, applying some rules, like dropping documents with empty bodies and trimming each body to 12,000 chars so the agent prompt stays inside the context window.

• Note: In this example, we’re cherry-picking some concrete KI IDs, because generating KIs for every document in the index would take a long time, and we want this exercise to be short for those following along.

• loop_corpus_docs iterates over every returned document, running the following two steps per document: 

• generate_ki reads the document and calls an LLM to emit a strictly grounded, structured KI.

• sink_ki bulk-writes each KI into the AI Index (ai-index-idx-my-corpus) as a KI of type corpus_entry. It forces _id to be the same as the document’s docid so rerunning the workflow is idempotent.

To summarize, this workflow turns each raw corpus document into a compact, searchable metadata record that agents can find and judge without reading the full source into the context window.

This workflow is used for example purposes, and the same foreach caveat as in Part 1 applies. For scale, use workflow.executeAsync or native parallel support. The cheat sheet is useful for optimizing Workflows. There could also be cost and efficiency gains in production by using ai.prompt or by choosing different models with which to create KIs. 

Inspect the KIs in your AI Index

Once the workflow runs, you can query the AI Index to browse what was written:

Here’s an example of what one of the KI documents looks like: 

Query KIs from LangChain Deep Agents

We’ll use LangChain Deep Agents with an OpenAI-compatible key to show that AI indices and KIs will work with any agent harness, inside and outside of Kibana’s Agent Builder ecosystem. 

First, let’s create facts_baseline_agent.py to measure our baseline before applying KIs: 

When I ran this baseline query, What was the actress who played Torvi from Vikings also known for?, it output the following:

(Note: Deep Agents automatically adds the read_file tool to handle paginated tool results, which is why it shows up in the output.) 

Next, let’s create an agent that knows how to use our query-ki skill, facts_ki_agent.py: 

This agent will query the KI indices to generate the answer, rather than pulling every document into context. 

When I ran these results against the same query, here was the output: 

How much can precomputing facts reduce agent token usage?

Both agents had similar conclusions, but they took far different paths to get there: 

The same question and the same grounded answer result in 93% fewer tokens and two tool calls instead of eight, when answering from KIs.

Baseline (No AI Index)

With AI Index

Total tool calls

8

2

read_file calls

2

1

esql_query calls

6, all against the browsecomp-plus index

1, from ai-index-idx-*

Tokens consumed

386,187

27,625

Latency

44.86s

15.22s

Answer

Grounded, correct

Grounded, correct

Exact tool call counts, latency, and answers will vary between runs and using different agents. 

Both agents produced solid, grounded answers. The difference is cost. Querying KIs from the AI Index cut token use by 93% and cut latency by roughly two thirds. Here’s how both paths went, side by side:

That was in-depth, but it shows what AI indices and Workflows do together: the same answer, at a fraction of the tokens.

Build precomputed context in Elasticsearch Serverless

This walkthrough shows how to generate more sophisticated KIs based on documented facts and query them for knowledge retrieval use cases using Elasticsearch primitives. 

Managing context is critical in agentic search systems. And at its core, context is a retrieval problem. AI indices help you manage context within the Elastic Stack. Try it out in Serverless, and let us know what you think in our Discuss forums or the #stack-kibana channel in our Community Slack.

We’d also love to hear from you about what use cases you’d like to solve using AI indices.

In Elasticsearch 9.5, Elasticsearch Query Language (ES|QL) queries no longer fail when a field isn't in the mapping. The new unmapped_fields setting lets queries load values from _source or fill with nulls, so queries keep working even when a backing index changes and a field goes missing, and can use unmapped data without reindexing. Here’s how we built that: the design choices and the edge cases (including a class of fields we nicknamed PUNKs), along with the testing strategies that gave us the confidence to ship it in general availability (GA).

Why ES|QL queries fail when a field is unmapped

You built a visualization using an ES|QL query. You refined it, and the query grew. You’re at 15 chained commands and counting, but it does just the right thing. It works, and your dashboard is useful.

Your query uses an index from a remote cluster, say my-remote:logs-foo. But actually, logs-foo is an alias, and at some point, the remote cluster makes it point to a different backing index. The new index is missing a field that’s used in your query, and your query and visualization break.

Or maybe you have an already fairly large index, and while building ES|QL queries on top of it, you realize that you’d like to use a field in the indexed documents that unfortunately never made it into the index mapping. You could reindex the data, but that would take hours.

ES|QL’s unmapped_fields setting is meant to deal with these types of situations.

If your query looks like this:

and some_field is unmapped, ES|QL’s default behavior is to fail with a verification exception.

You can use the unmapped_fields setting to instead either fill some_field with nulls or read it from the document’s _source, like so:

How ES|QL resolves queries with field caps

Before we jump into the inner workings of unmapped_fields, we have to look into how ES|QL resolves queries regularly. Let’s consider the above query:

We said that if some_field isn’t in the mapping for index, ES|QL will reject the query. How does it make that decision?

How field caps tells ES|QL which fields exist

In a typical schema-on-write fashion, Elasticsearch clusters maintain mappings with their respective indices. As a first step, ES|QL makes an internal request to the field caps endpoint to determine which fields the index has. It then passes the query, together with the field caps response, to the query planner, which consists essentially of the query analyzer (unrelated to analyzers of text fields) and query optimizer. The analyzer makes sense of raw names, like some_field, and notices that they correspond to index fields (or not). If all went well, the query is then passed on to the optimizer, which rewrites the query for efficiency, before it’s handed to the compute engine for execution.

How the analyzer resolves field names in the query plan

Let’s zoom in to the analyzer. The parsed query is represented in a tree structure, and the analyzer partially rewrites it, one command at a time, until it either has resolved all references or not.

For illustration, let’s use a somewhat more complex query and see how the analyzer would resolve it:

The parsed tree is actually a chain here, and it looks something like this:

The analyzer then moves up through the query tree to try and resolve the field names used in every command.

This is a simplified version of how we represent parse trees in tests and when debugging. The bottom of the chain corresponds to the FROM command and contains a list of all mapped fields that we know about, obtained from the field caps endpoint. (The {f} suffix marks an actually mapped field for better distinction later.)

The two EVAL nodes on top of it correspond to the remaining commands, and their fields are still unresolved, expressed by the question mark ? in front of the name. At this point, the analyzer still has to check whether they correspond to existing index fields.

For the EVAL that defines uppercased_mapped, it can see that the previous command outputs mapped_field, so the unresolved ?mapped_field marker can be replaced by a real field reference:

Next, it encounters the topmost EVAL, which defines uppercased_unmapped. The previous tree nodes produce only two fields: [mapped_field, uppercased_mapped]. The reference ?unmapped_field thus has to remain unresolved. We bail here and emit the verification exception to the user.

How unmapped_fields LOAD and NULLIFY work

Adding unmapped fields to the query plan

When using unmapped_fields=”NULLIFY” or ”LOAD”, we do something else; we act as if the field was actually in the index. The analyzer adds unmapped_field to the From node and marks it as unmapped to signal to the compute engine that this has to be read from _source or filled with nulls. Let’s express this with a {u} (for unmapped):

After amending the From, the analyzer can continue trying to resolve the topmost Eval node. It sees that the upstream nodes produce the fields [mapped_field, unmapped_field, uppercased_mapped] and thus unmapped_field can be correctly resolved:

The query plan is now fully resolved and can be passed down the regular optimization-execution pipeline. Other than the actual value extraction mechanism, everything stays the same. Schematically, the workflow looks like this:

Example: enabling unmapped fields with the SET directive

To give an example, let’s fire up a cluster and create an index with non-dynamic mappings.

We can run the example query, above:

This should result in the error message:

Unknown column [unmapped_field], did you mean [mapped_field]?

To make things work, we can prepend SET unmapped_fields=”...”; with LOAD or NULLIFY:

Inspecting the analyzer's rewrite steps

If you want to see what the query analyzer is doing to the parse tree, you can log the query rewrite steps, like so:

This will log a line containing Rule rules.ResolveUnmapped applied with change… You’ll see that unmapped_field is added to the bottom of the parse tree as described above.

Why we have to infer the schema

Of course, this isn’t the only possible method to deal with unmapped fields. Here are some alternatives:

1. We could also scan or probe the documents in index to determine that their _source actually has the unmapped_field.

2. We could disable verifications in the analyzer and make the compute engine blindly pass unmapped fields through individual computation steps.

The first alternative front-loads more work to understand the actual schema of an index and thus generally increases latency. It doesn’t scale to large, highly distributed datasets. The second alternative isn’t viable since it means a large-scale change to how ES|QL’s compute engine is built, because it passes around streams of data with fixed columns from one operator to another.

In contrast, the approach we chose is neatly compatible with ES|QL’s existing optimization pipeline.

The trade-off is that the analyzer has to correctly infer a schema based on the actual index mappings (obtained from the field caps endpoint) and additional fields used inside the query. 

This isn’t always straightforward. There were two main challenges:

1. There are many different query shapes and commands that can be used. The mechanism needs to detect unmapped fields, update the proper FROM command, and pass the new field through the halfway resolved plan correctly in all cases.

2. There are many different mappings we have to deal with, and we specifically need to make our feature work correctly when mappings change over time on top of that.

In the following, we’ll focus on LOAD, although some problems (generally many fewer) also apply to NULLIFY.

Which index to load unmapped fields from for LOOKUP JOIN and FORK

To briefly illustrate the first problem, here are some choices we needed to make:

• Which index do we load from when using lookup joins? This one?

  The unmapped_field cannot be attributed to both indices. We chose index since this is where we expect mappings to change more often than in lookup indices.

• Similarly, how do we deal with subqueries and views or the FORK command? In the following:

  one fork branch triggers loading of an unmapped field. Is it also present in the other fork branch? (Yes, it should be, but it’s not obvious and is specifically not true if the two FORKs are replaced by independent subqueries.)

Two principles to keep queries working

The second problem, diversity of mappings and their evolution over time, is a far bigger driver for complexity. We strove for two basic usability principles:

• Queries that work in the default mode should generally still work when using unmapped_fields=”NULLIFY” and ”LOAD”.

• Queries that work when all fields are mapped should generally still work with NULLIFY and LOAD when a field becomes unmapped and vice versa.

The type of unmapped fields and inadvertent type conflicts

Let’s talk about data types to see where this leads to complexity. First, when using unmapped_fields=”LOAD”, we need to assume a data type for unmapped fields. We chose KEYWORD, which allows us to avoid type conflicts when reading from _source. One document can contain ”unmapped_field”: “foo”, and another can contain ”unmapped_field”: 123.4. It’s fine because we treat both as strings.

However, this is a violation of the second principle when a non-KEYWORD field happens to go unmapped. Consider this query:

If some_field becomes unmapped, we’ll have to assume that the KEYWORD type and the query will fail with a type conflict.

Type conflicts aren’t new and can be dealt with by using explicit casts in the query, like so:

It would be great if ES|QL just inferred a useful type to cast to, but this is something for the future.

Type conflicts with partially unmapped fields, or: making PUNKs well behaved

In addition to fully unmapped fields, partially unmapped fields are everywhere and should also work with LOAD. Let’s look at a query that uses multiple indices.

Let’s say that there are indices index and index_without_some_field, containing just the following documents.

Now let’s consider the query:

and assume that some_field is unmapped in index_without_some_field. This will return:

because ES|QL doesn’t load unmapped fields per default.

Of course, when setting unmapped_fields=”LOAD”, we want to load from _source for index_without_some_field:

As with fully unmapped fields, the case is simple when some_field is mapped as KEYWORD in index. When loading from _source for index_without_some_field,  we treat the field as KEYWORD as well, so there’s no conflict.

What makes a field a PUNK

The case is less clear when some_fieldis partially unmapped and the mapped leg is of a type other than KEYWORD. Such fields caused a lot of trouble until we found the best solution, which makes their acronym quite fitting: partially unmapped non-keyword fields, or PUNKs.

Unfortunately, PUNKs are far from being esoteric. For instance, it’s very natural to filter on a PUNK:

If some_field is mapped as INTEGER in index, the type conflict looks like this:

• Mapped as an INTEGER in index.

• Unmapped in index_without_some_field and thus treated as KEYWORD.

This can again be resolved manually by providing an explicit cast:

But this is far from acceptable. Even queries that work fine without NULLIFY and LOAD typically have some PUNKs; the unmapped leg is simply treated as null then. Both guiding principles are violated if LOAD requires an explicit cast here.

Casting implicitly to the mapped type

The solution is to introduce an implicit cast to the mapped type. In this case, we know that some_field is an INTEGER in index, and thus we treat it essentially as if the user wrote:

This means that queries that work without LOAD keep working. (ES|QL may even give you more data because we load the unmapped leg of PUNKs from _source.) Queries that used to work when a field is fully mapped also keep working when it goes unmapped in some (but not all) of its indices without having to alter the query in any way.

Behavior

Default

NULLIFY

LOAD

Unmapped field in query

Query fails

Query runs

Query runs

Values returned

None

null

Read from _source

Assumed type

n/a

NULL

KEYWORD

Partially unmapped field (PUNK)

Unmapped leg is null

Unmapped leg is null

Cast to the mapped type

Pushdown optimization

Full

Full

Per-node where fully mapped

Don't throw it all away: Keeping ES|QL query optimization with unmapped fields

There's one more thing to get right; that is, to make sure that optimizations still work correctly with LOAD. Consider the previous query:

ES|QL’s optimizer aggressively pushes down such WHERE filters and turns them into Lucene queries, so the compute engine doesn’t perform unnecessary work.

For this query, evaluating the filter in the compute engine would require fetching each and every document from the index; meaning, a full scan, very slow. If some_field was mapped as an INTEGER in both indices, we would instead perform a Lucene query, which looks like this:

The compute engine then doesn’t have to load each document separately and check whether it matches the filter. Documents with some_field <= 10 are never fetched from the Lucene index, which is very efficient at this kind of filtering. Nice.

Why filter pushdown is unsafe for unmapped fields

If some_field is unmapped in index_without_some_field, however, it’s wrong to narrow the documents down using the same Lucene query, as Lucene interprets an unmapped some_field as null and thus no documents from index_without_some_field will ever match. This edge case is easy to miss, and it doesn’t help that there are several flavors of similar pushdowns. For instance, in the query:

the compute engine pushes even the counting to Lucene. Again, this is only correct if some_field is fully mapped.

This means that such optimizations can't apply to unmapped fields. It would be disappointing if a query used hundreds of indices and only one of them happened to not map some_field, causing the whole query to run unoptimized.

How the local optimizer recovers the fast path

Luckily, this problem has a solution, too. ES|QL actually has multiple optimizer runs:

1. First, a preliminary optimizer run on the node handling the _query request.

2. Then, a second, local optimizer run on every node we fan out to because we need to fetch documents from its shards.

The workflow after the initial optimization looks more like this:

If the current node happens to map some_field in all shards, the local optimizer detects this situation and treats some_field like any other fully mapped field, including performing Lucene queries to greatly narrow down the dataset to be processed. In fact, data nodes process LIMIT queries like:

in batches of shards (to avoid loading too much data too eagerly), which includes a full local optimizer run per batch. This makes it even more likely to encounter batches where some_field is fully mapped, allowing ES|QL to run a fast Lucene query.

Is it working now? Testing unmapped_fields across every ES|QL query shape

As we have seen from the optimizer issues above, problems can hide in plain sight, even for very simple queries. Because unmapped_fields=”LOAD” can affect each and every kind of query, the surface area for bugs is essentially all of ES|QL.

Accordingly, getting good test coverage was tricky and challenged us to refine our testing strategies.

Reusing spec tests with unmapped_fields

Conveniently, ES|QL has an extensive corpus of test queries, together with expected result sets; we call them spec tests because they’re written using a simple text specification language, which looks roughly like this:

This lets us create new tests out of the existing ones by introducing slight variations. For instance, any existing test that runs without SET unmapped_fields=”...” should produce the exact same results when run with SET unmapped_fields=”NULLIFY”.

It also helped find major issues early in the development process, especially for NULLIFY. The LOAD setting changes the meaning of queries much more dramatically, limiting the usefulness of this approach. However, ES|QL also uses what we call generative testing; that is, we string together random commands, run the query, and then check whether the server reports a bug. This approach cannot confirm the correctness of results, but it still helped greatly with finding query types that didn’t work properly and resulted in some kind of error. (Property-based tests would be a refinement in the future by running the queries against a reference implementation. This way, correctness of results can also be checked.)

Testing type conflicts across different mappings

In the end, one of the most important testing dimensions was using different indices with various mappings in the same query. (Recall how, above, we had to deal with type conflicts to come up with a solid approach for PUNKs? It doesn’t end there; all kinds of type conflicts are more complex with LOAD.) Since we couldn’t automatically generate correct expected results, ES|QL’s test suite had to grow by adding more than 10,000 lines of CSV spec tests. Fortunately, adding such tests is a well-suited task for an AI agent, which has cut down the effort dramatically. (Of course, the test results were still reviewed by humans.)

All testing strategies together provided us with good confidence for the GA release of unmapped_fields with Elasticsearch 9.5.

What you'll discover

In this post, you'll learn how to:

• Use Elastic APM's built-in views to explore search latency and throughput and to explore errors.

• Define Service Level Objective (SLOs) for search, including latency targets and availability, along with search quality.

• Create SLOs in Kibana that track your search health over time, with burn rate alerting.

• Build operational dashboards with Elasticsearch Query Language (ES|QL) that show latency percentiles and time breakdowns and that include trends.

• Set up alerts for latency regressions and error spikes, along with zero-results rate increases.

• Establish an incident response pattern for search degradation.

What you'll need

• Search instrumentation from Blogs 2–4 (search spans with search.* attributes in Elastic).

• Kibana access with permissions to create SLOs and alert rules.

• Basic understanding of SLOs. (We'll explain the search-specific parts.)

• An Elastic cluster with an Enterprise subscription, an Elastic Cloud trial, or a local deployment with the trial activated.

Why search monitoring matters beyond cluster health

Search is the primary navigation path for a significant share of visitors, and search-initiated sessions tend to show stronger purchase intent than browse sessions. A search outage is a revenue event, rather than a minor feature degradation. A latency regression from 100ms to 500ms changes user behavior before anyone files a ticket.

Most platform teams monitor search at the infrastructure level, checking whether the Elasticsearch cluster is healthy and whether nodes are responding. They also determine whether the disk is full. These are all necessary but not sufficient. A cluster can be green while search quality silently degrades; for example, queries returning stale data after a bad index deployment or latency creeping up as the index grows. This could also include zero-results rates climbing because a synonym list wasn't updated.

The gap is between "search is up" and "search is working well."

A note on examples: As we have throughout this series, we use ecommerce search for concrete examples, but these reliability patterns apply equally to any search application, including content platforms, internal knowledge bases, job boards, and support portals.

OpenTelemetry search spans as monitoring signals

If your team followed Blogs 2–4 in this series, every search request already emits an OTel span with search.* attributes. These include the query text, result count, Elasticsearch execution time, and error status. Those spans land in traces-generic.otel-default in Elastic.

Following along with code? The reference project has all the instrumentation from Blogs 2–4. Generate traffic, and then follow along with the SLO and dashboard setup below. See queries/blog6_reliability.esql for ready-to-run queries.

The search team built that instrumentation for product analytics; that is, understanding what users search for and measuring click-through rates (CTRs) and conversion rates. They’ve prioritized relevance work, but the same spans contain everything you need for operational monitoring. Span duration gives you a latency signal, and search.result_count == 0 value reflects quality. Span errors point to availability signals.

This post shows how to put this operational value to work, beginning with what you can see right now in Kibana and then building SLOs and alerting on top of it, along with incident response.

Search monitoring out of the box with Elastic APM

Before building anything new, let's look at what Elastic APM already gives you out of the box.

If your search API is instrumented with Elastic Distribution of OpenTelemetry (EDOT) (as in Blog 2), it appears automatically as a service in the Elastic APM UI. Open Observability > APM > Services in Kibana, and select your search service (named search-analytics-demo if you're using the reference project). You'll immediately see the following:

Elastic APM service overview for search

The service overview page shows latency distribution and throughput over time, along with error rate, and doesn’t require configuration. You can see at a glance whether search is healthy, and the time-series charts make regressions obvious. If latency crept up after yesterday's deployment, you'll see it here.

Trace waterfall: breaking down search request latency

Click into any transaction, and you'll see the trace waterfall, which is a visual breakdown of every span in the request. For a search API call, this typically shows:

The waterfall makes the invisible visible. You can see that the 503ms API response breaks down into HTTP handling, a 241ms query rules lookup, and a 260ms Elasticsearch query,  plus the custom search span (36ms) carrying all of our search.* attributes. Click any span, and the metadata flyout shows exactly what was captured: search.query: "usb hub", search.result_count: 33, search.took_ms: 43, the index name, hit IDs, and more.

Blog 2 discussed the gap between search.took_ms and span duration. The waterfall shows you exactly where that gap lives, without writing any queries.

Automatic search error capture with OpenTelemetry

One of the most valuable things OTel auto-instrumentation gives you is automatic error capture. When an Elasticsearch query fails because of issues like a tripped circuit breaker or a timeout, or if an index isn’t found,  the span records the exception type and message, along with the stack trace. Blog 4 mentioned this as a side benefit of span-based conversion tracking; here it becomes an operational lifeline.

The errors tab on your service page automatically aggregates these, grouped by error type and frequency. The instrumentation captures the details for you, so you don't need custom error handling or logging. During an incident, this is often the fastest way to understand what's actually failing.

Service map: search API and Elasticsearch dependencies

The service map shows dependencies between your search API and Elasticsearch, making it easy to see whether a latency problem is in your service or in the cluster it depends on.

All of this is available the moment you deploy the instrumentation from Blog 2, without building any dashboards or writing any queries. This is the foundation everything else in this post builds on.

Defining service level objectives for search

An SLO defines good enough in measurable terms. You define what working means, measure it continuously, and alert when you're burning through your error budget too fast, instead of reacting when something breaks.

Elastic Observability has a built-in SLO framework that handles Service Level Indicator (SLI) calculation and budget tracking. It also takes care of burn rate alerting. You create SLOs directly in Kibana. No ES|QL or custom pipelines are required for the core indicators.

Three SLOs every search service needs

Navigate to Observability > SLOs in Kibana, and click Create SLO. The SLO creation workflow walks you through three steps: Define the SLI (what to measure), set the objective (the target), and describe the SLO.

1. Search latency SLO: 99% of queries under 250ms

Indicator type: Elastic APM latency target: 99% of searches complete in under 250ms.

The Elastic APM latency indicator is purpose-built for this. Select your search service (search-analytics-demo), and set the threshold to 250ms.  Elastic handles the rest, including calculating the percentage of transactions below the threshold and tracking your error budget over time.

Note: The Elastic APM latency indicator measures all HTTP transactions for the search-analytics-demo service, including health checks and click/cart/checkout endpoints, along with static asset requests, not just the POST /api/search endpoint. For a search-only latency SLO, use a custom Kibana Query Language (KQL) indicator with name: "search" AND attributes.search.query: * on the traces-generic.otel-default index. The Elastic APM indicator is still valuable for whole-service health. For full coverage, combine both.

This measures end-to-end span duration; that is, what the user actually experiences. If your Elasticsearch query takes 50ms but the user waits 300ms because of network overhead or slow application logic, this SLO catches it. Use search.took_ms in the Elastic APM waterfall to diagnose where the latency lives when the SLO starts burning.

2. Search availability SLO: 99.9% success rate

Indicator type: Elastic APM availability target: 99.9% of searches succeed.

The Elastic APM availability indicator calculates the percentage of successful transactions for your service. When the Elasticsearch client throws an exception or the search endpoint returns a 5xx, the span's status records an error and this SLO counts it.

Note: Like the latency SLO, the Elastic APM availability indicator covers all HTTP transactions on search-analytics-demo, not just POST /api/search. Click/cart/checkout errors will consume this budget. For a search-only availability SLO, use a custom KQL indicator with name: "search" AND attributes.search.query: * for good events and name: "search" as the total query.

An 0.1% error budget on 100,000 daily searches means that you can tolerate 100 errors per day. That's tight, but search errors are hard failures and the user gets nothing. Availability SLOs should be stricter than latency SLOs.

3. Search quality SLO: tracking zero-results rate

Indicator type: Custom KQL target: 85% of searches return at least one result (zero-results rate < 15%); index: traces-generic.otel-default; good query: name: "search" AND attributes.search.result_count > 0; total query:name: "search" AND attributes.search.query: *.

Note on KQL versus ES|QL: The SLO framework uses KQL for its indicator filters rather than ES|QL. KQL uses field: valuesyntax and is the same language you see in the Kibana search bar. The ES|QL queries throughout this series are for ad hoc analysis and dashboards; KQL here is the SLO indicator's document filter. Both query the same traces-generic.otel-default index.

This is the SLO that surprises most teams. A search that returns an empty result set isn't an error; HTTP status is 200 and the span status is OK. Plus, no exception was thrown. But from the user's perspective, it failed. They asked for something and got nothing.

The quality SLO uses the custom KQL indicator type because it relies on our custom search.result_count attribute, which the built-in Elastic APM indicators don't know about. But the SLO framework handles everything else, including budget tracking and burn rate calculation, along with alerting.

A sudden spike in zero-results rate, such as from 12% to 40% over an hour, is almost always an infrastructure event, like a failed index deployment or a mapping change that broke queries. It could also be a synonym list misconfiguration. That's an operational problem, not a relevance problem.

Reading your search health in the SLO overview

Once the latency, availability and quality SLOs are created, the SLO overview page shows your search health at a glance:

Each SLO shows the current value, the target, the remaining error budget, and the burn rate. Green means healthy:  Search availability is at 100%, and Search quality is just above its 85% target. Red means violated: Search latency is at 50% against a 99% objective, with the burn rate breached at 200x the sustainable rate. When a budget bar starts shrinking faster than expected, you know something changed, even before users complain.

Clicking into the SLO detail shows burn rate across multiple time windows (1h, 6h, 24h, 72h) and the historical SLI trend. It also shows remaining error budget. For the latency SLO, the Elastic APM latency indicator tracks the percentage of transactions below your 250ms threshold. For the quality SLO, the custom KQL indicator uses traces-generic.otel-default with the good query filtering for attributes.search.result_count > 0. This is where the custom search.* attributes from Blog 2 pay off, since they're the foundation of meaningful SLOs.

Burn rate alerting for search SLOs

When you create an SLO through the Kibana UI, a default burn rate alert rule is automatically created. This is where the real operational value lives.

Burn rate alerts improve on threshold alerts ("error rate > 1%"), which are noisy and miss slow degradation. : Burn rate alerts measure how fast you're consuming your error budget relative to the SLO window.

A burn rate of 1.0 means that you're spending budget at exactly the sustainable rate, but a burn rate of 10.0 means that you're burning 10x too fast and you'll exhaust the budget in 1/10th of the window.

The default burn rate rule uses a multi-window approach, with four severity levels:

Severity

Burn rate

Long window

Short window

What it means

Critical (page)

> 14.4x

1 hour

5 minutes

Exhausts budget in ~50 hours

High (ticket)

> 6.0x

6 hours

30 minutes

Exhausts budget in ~5 days

Medium (review)

> 3.0x

24 hours

120 minutes

Exhausts budget in ~10 days

Low (awareness)

> 1.0x

72 hours

360 minutes

Trending toward exhaustion

The short window prevents alerting on brief spikes that self-resolve, and the long window catches sustained degradation. Together, they balance responsiveness with alert fatigue.

Routing search alerts to PagerDuty, Slack and Jira

Alerts are only useful if they reach the right people in the right tools. Elastic's alerting framework supports a wide range of connectors out of the box, including:

• Incident management: PagerDuty, Opsgenie, xMatters for on-call routing.

• Chat: Slack, Microsoft Teams for team notifications.

• Case management: Jira, ServiceNow for automatic ticket creation when SLOs breach.

• Custom: Webhooks for integrating with any system via HTTP.

You can also use Elastic's built-in cases to track incidents directly within Kibana, linking alerts and traces in one place, along with investigation notes, with push to Jira or ServiceNow when escalation is needed.

A typical routing setup:

Alert

Severity

Channel

Latency SLO burn rate > 14.4

Page

PagerDuty

Availability SLO burn rate > 14.4

Page

PagerDuty + Slack

Quality SLO burn rate > 6

Ticket

Jira (auto-create) + Slack

CTR anomaly (machine learning [ML] job)

Notification

Slack (search team)

Anomaly detection for search quality

Some search degradations are gradual shifts that slip past threshold-based alerts, rather than sudden spikes. A relevance regression after a model update might reduce CTR by 15% over a week, and latency might creep up by 5ms per day as the index grows. These are real problems, but they don't trigger burn rate alerts until it's too late.

Elastic's anomaly detection is built for exactly this. It learns normal patterns in your search metrics and flags deviations automatically, and you don’t have to configure any thresholds. 

Detecting search quality degradation with ML anomaly detection

• Latency anomalies:

 Elastic APM anomaly detection can be enabled directly from the Elastic APM UI for your search service. It learns the typical latency distribution, including daily and weekly patterns, and alerts when behavior deviates. A gradual 5ms/day creep will eventually register as anomalous before it hits your SLO threshold.

• CTR drops:

A relevance regression is invisible to traditional monitoring; latency is fine and errors are zero, plus the result counts are normal, but the ranking changed and users aren't clicking. Anomaly detection on click volume per query is a practical proxy: When a query that normally receives 20 first-click events per hour drops to 5, something likely changed.

To set this up: In Kibana → Machine Learning → Anomaly Detection, create a new job. Use the Multi-metric wizard, and select traces-generic.otel-default as the index. Configure a count detector on attributes.search.first_click split by attributes.search.query. This creates a per-query click-volume baseline and alerts when individual query engagement drops outside the expected range.

Note: This job detects click-volume anomalies per query, not CTR (which requires dividing clicks by searches). Click volume is a useful proxy (a CTR regression usually manifests as a drop in absolute click count), but be aware that a traffic surge with flat click volume would show as a CTR drop without triggering this alert. For true CTR anomaly detection, use a scheduled ES|QL transform to materialize hourly CTR values and run anomaly detection on the computed ratio.

Route the resulting ML alert rule to your Slack search channel.

• Throughput shifts:

A sudden drop or unexpected surge in search volume can indicate upstream problems (like load balancer changes or traffic shifts) or downstream issues (such as search becoming unresponsive or users retrying).

Configure ML anomaly alert rules to route these to your notification channels. These complement your SLO burn rate alerts; burn rates catch budget consumption, and anomaly detection catches pattern changes.

Building a search monitoring dashboard with ES|QL

SLOs tell you whether search is healthy. When they indicate a problem, you need a dashboard that tells you why.

The search team and the on-call team need different views of the same data. A search engineer wants query-level detail, such as which queries have low CTR and which ones return nothing. They’re also interested in where to invest in relevance. But an on-call SRE wants the operational picture, including whether search is fast and whether it’s up. It also wants to know whether search is degrading, and if so, since when.

Search monitoring panels for the on-call dashboard

Build it in Kibana dashboards using Kibana Lens panels. Lens supports ES|QL as a data source, so the queries from Blogs 2–4 can power dashboard panels directly. The key panels include:

• Search throughput over time:  A sudden drop is often the first sign of a problem.

• Latency percentiles (p50, p95, p99) over time: When they diverge (p50 flat, p99 spikes), you have a subset of slow queries.

• Error rate over time: Spikes here mean hard failures.

• Zero-results rate over time: A step change upward, especially correlated with a deployment, means something changed in the index or query pipeline.

The ES|QL for each panel follows the patterns from earlier blogs. For example, a latency percentile panel:

This includes three lines on one chart. When they diverge, such as when p50 stays flat but p99 spikes, you likely have a subset of queries that are slow while the majority are fine. That's a different diagnosis than all queries slowing down (cluster-level pressure).

Drill-down panels: slowest queries and top zero-result queries

For investigation, add a few detail panels, such as:

Slowest queries: A table showing the queries with the highest p95 latency and their search volume. During an incident, this narrows the problem from "search is slow" to "these specific queries are slow."

• Top zero-result queries: A table showing which queries most frequently return nothing. When zero-results rate spikes, this panel immediately shows which queries are responsible.

These drill-down panels use the same ES|QL patterns as Blogs 2 and 3, just surfaced on a persistent dashboard instead of run ad hoc.

Search incident response using OpenTelemetry traces

As an example, an alert fires, noting that search latency has spiked. What happens now?

The trace data from Blog 2's instrumentation gives you a structured path from symptom to root cause.

Step 1: Assess scope

Start at the on-call dashboard, and get answers to the basics:

• When did it start? Narrow the time range to the degradation window.

• How bad is it? Is p50 affected (all queries slow) or just p99 (a subset)?

• Is it just search? Check the Elastic APM service map to determine whether the Elasticsearch dependency is also degraded.

Step 2: Find the problem queries

If the problem is a subset of queries (p99 spike but p50 is fine), use the slowest queries panel or run:

Adjust the 100 ms threshold to match your environment's normal range. It can be lower for a fast cluster or higher if your data volume makes 100ms typical. This narrows the problem from "search is slow" to "these specific queries are slow." That's the difference between restarting the cluster and investigating a specific query pattern.

Step 3: Drill into the trace waterfall

Pick a slow query, and open it in the Elastic APM trace view. The waterfall shows exactly where time was spent. (Refer back to the trace waterfall GIF above to see a real example of a POST /api/search trace broken down into its component spans.)

The overhead gap between search.took_ms (Elasticsearch time) and span duration (end-to-end time) is your diagnostic tool:

Scenario

search.took_ms

Span duration

Diagnosis

Elasticsearch slow

400ms

430ms

Elasticsearch problem: Check slow log, cluster metrics.

App slow

50ms

350ms

Application / network overhead: Check serialization, network.

Both slow

400ms

700ms

Multiple issues: Investigate both.

If the problem is in Elasticsearch, drill into the Search Profile API or cluster monitoring. If it's application overhead, look at the spans around the search span in the waterfall.

Step 4: Correlate with events

Check whether the degradation correlates with:

• Deployments: Did someone deploy a new version of the search service or push a new index?

• Cluster events: Is Elasticsearch under memory pressure, or are there long garbage collection pauses? Or maybe the disk is I/O saturated?

• Network: Is latency between the search service and Elasticsearch elevated?

Elastic Observability's unified platform makes this correlation straightforward because traces, logs, metrics, and infrastructure data all live in the same Kibana instance. You're adding filters in the same interface, rather than switching between tools.

Going further: Infrastructure metrics and cost attribution

This post focuses on what you get from trace data; that is, the spans your search API already emits. But OTel and Elastic Observability support a wider instrumentation picture that becomes valuable as your search infrastructure matures.

• Infrastructure metrics. Adding host and container metrics (like CPU, memory, disk I/O, and network) alongside your traces lets you correlate search performance with infrastructure use. When p99 latency spikes, you can immediately see whether the Elasticsearch nodes are under memory pressure and whether garbage collection  pauses are increasing. You can also check whether disk I/O is saturated, and you can do all this in the same Kibana interface. The Elastic Agent collects these automatically for your infrastructure, and the infrastructure monitoring UI surfaces them alongside your Elastic APM data.

• Total cost attribution (TCA). With infrastructure metrics flowing alongside traces, you can start attributing infrastructure costs to specific services and operations. How much compute does your search service consume? How does that correlate with query volume? If a new ranking model doubles CPU usage per query, you can see the cost impact directly. This is particularly valuable for teams running search on cloud infrastructure where costs scale with resource consumption; understanding the cost per search helps justify infrastructure investment and identify optimization opportunities.

• Logs correlation. OTel auto-instrumentation injects trace context (such as trace ID and span ID) into your application logs. This means that when you're investigating a slow search in the trace waterfall, you can click through to the exact log lines from that request, including Elasticsearch slow log entries and application debug output. It also includes error details that don't fit in span attributes. The logs correlation feature automatically ties them together.

These are natural next steps once you have traces working. Each one extends the same unified platform, without new tools or separate pipelines.

How search analytics and search monitoring share one data pipeline

Here's how it all fits together:

The data flows from the instrumentation you built in Blog 2. This one investment supports two audiences: The search team gets product analytics (Blogs 2–5), and the SRE team gets operational monitoring (this post). Neither team needs separate data pipelines.

Getting started with search monitoring in Elastic

This is the last post in the series, and it brings us full circle. Blog 1 describes the vision: Instrument search once with OTel, and send spans to Elastic. Then use ES|QL to answer any question about search behavior. Blogs 2–4 build the instrumentation and analytics, and Blog 5 shows how to feed that data back into relevance improvements. This post shows how the same data powers operational monitoring, including SLOs, alerting, anomaly detection, and incident response.

The key takeaway for search engineers is that the instrumentation you built for analytics already generates the signals. The SLOs and alerts are built-in capabilities of Elastic Observability, as are the dashboards. You're closer to production-grade search monitoring than you might think. Observability isn't a separate discipline you need to learn from scratch. 

If you've been following along and built the instrumentation from Blogs 2–4, start here:

1. Open Elastic APM: Look at your search service, and explore a trace waterfall. You can also check the errors tab.

2. Create three SLOs: Latency (Elastic APM latency), availability (Elastic APM availability), and quality (custom KQL for zero-results).

3. Enable anomaly detection: One click in the Elastic APM UI for latency anomalies.

4. Build the on-call dashboard: Four Lens panels with the queries from this post.

By the end of an afternoon of work, your search service can have the same observability coverage as any other critical production system.

Get started

Working code

• Reference project: Working code for the entire blog series; clone, configure, and run.

Elastic APM and traces

• Elastic APM Overview: Elastic APM concepts and trace analysis.

• Elastic APM UI: Service overview, transactions, dependencies, errors.

• Service Maps: Dependency visualization and health.

• OpenTelemetry Integration: OTel ingestion in Elastic.

SLOs and alerting

• SLOs in Elastic Observability: Creating and managing SLOs.

• Create an SLO: SLI types (like Elastic APM latency, Elastic APM availability, or custom KQL), time windows, budgeting.

• SLO Burn Rate Alerts: Multi-window burn rate alerting.

• Alert Connectors: Slack, PagerDuty, webhook, email integrations.

• Alerting Framework: Rule types and configuration.

• Jira Connector: Automatic ticket creation from alerts.

• ServiceNow Connector: ITSM integration.

• Elastic Cases: Built-in incident tracking with external push.

Anomaly detection

• Anomaly Detection Overview: Unsupervised time series anomaly detection.

• Elastic APM Anomaly Detection: Enable ML for latency, throughput, error rate.

• ML Anomaly Alert Rules: Alerting on detected anomalies.

Dashboards and ES|QL

• Kibana dashboards: Building operational dashboards.

• Lens: Visualization editor.

• ES|QL in Lens: ES|QL-powered dashboard panels.

• ES|QL Overview: Language reference.

Elasticsearch operations

• Slow Log Configuration: Threshold-based query slow logging.

• Search Profiling: Profile API for query execution analysis.

• Monitoring Elasticsearch: Cluster stats, search rate, latency.

From this series

• Modern search analytics with OpenTelemetry: The vision.

• Instrument your search API : Search spans and search.* attributes.

• Measuring search quality : CTR, MRR, click distribution.

• From clicks to conversions: Conversion tracking and revenue attribution.

• Personalizing search from behavior: Judgment lists, rank features, Learning To Rank (LTR).

This is the final post in a six-part series on search analytics with OpenTelemetry and Elastic. Start from the beginning: Modern search analytics with OpenTelemetry, or to start building, jump to Instrument your search API.

Why dynamic mapping breaks down with schemaless data

Elasticsearch wants a schema. Every field you index has a mapping that defines its type, its analyzer (for text), and how it is stored. That schema makes storage compression efficient, with fast search and cheap aggregation. It becomes a liability the moment your data stops looking like a database table.

Consider the shapes that show up in real systems:

• Log events where every service adds its own attributes. One emits labels.region, another labels.k8s.pod, and another labels.tenant_id.

• OpenTelemetry resource attributes, where the set of keys is defined by whatever agent happened to send the span.

• User-supplied metadata bags, feature flags, or tagging systems, where the keys are open-ended by design.

If you map each key as its own field, the mapping grows unbounded. This is the classic "mapping explosion." Thousands of dynamically created fields inflate the cluster state, slowing down every mapping update and eventually hitting the field limit. Each field also carries overhead in the index. You pay a structural cost for keys you didn’t plan for and may query only once.

The naive escape hatch is to store the whole object as a string and give up on querying its contents. That trades one problem for another: you keep the data but lose the ability to filter or group by anything inside it.

The flattened field type is the better path. You can index an entire JSON object under a single mapped field, keeping the keys queryable with almost none of the mapping-explosion cost. 

This post covers how a flattened field stores the data on a disk and how ES|QL reads it back.

How flattened fields index dynamic JSON keys under one mapping

Map one field as flattened:

Then write arbitrary nested JSON into it:

There is exactly one field in the mapping, labels, no matter how many keys appear across your documents. The cluster state doesn’t grow when a new key shows up. The subkeys remain individually searchable. You can reference labels.region or labels.k8s.pod in queries, even though neither was ever declared.

The catch and central tradeoff is that every leaf value is a keyword. The number 4 above is indexed as the string "4". There is no numeric typing, no date parsing, and no range math on dynamic keys. Flattened fields exchange per-field richness for schema flexibility. That fact explains almost every design decision that follows.

How Elasticsearch stores flattened field JSON keys on disk

There are two types of queries on flattened fields: an unkeyed query on the root flattened field, and a keyed query on a specific subfield. Following the previous example, a query of the form labels: "us-east-1" matches a value under any key, while the querylabels.region: "us-east-1" matches a value under the specific regionkey.

To support these two distinct query formats, the flattened mapper writes each leaf value into two distinct Lucene fields.

Take this document:

The mapper produces:

• A root field under labels, holding the bare values:

• A keyed field under labels._keyed, holding the flattened key concatenated with its value:

In the keyed field, nested objects are dot-flattened into a single key (k8s.pod), and the key is joined to its value with a reserved NULL byte (\0) as the separator. Keys that contain a NULL byte are rejected at parse time, so the separator is always unambiguous. To find the value, you split on the first NULL.

These two fields make both query shapes work:

• labels: "us-east-1" matches a value under any key, so it searches the root field.

• labels.region: "us-east-1" matches a value under a specific key. It rewrites the query to the term region\0us-east-1 and searches the keyed field.

Query restrictions on flattened field subkeys

Because every key's terms live in one sorted list, the keyed field cannot answer every query shape a plain keyword field can. Three restrictions follow:

1. No fuzzy, regexp, or wildcard on a specific subkey. Nothing about the layout makes them impossible,  but the pattern would have to be combined with the key prefix so it can’t walk past the key boundary. The mapper doesn’t do that at this writing.

2. Any range query on a subkey needs at least one bound. Elasticsearch already has a query for "this field has some value here, whatever it is": the exists query. On a flattened subkey it runs as a prefix query on key\0, which sweeps every term belonging to that key. A range with neither bound would sweep exactly the same terms. Rather than support two spellings of one scan, the mapper rejects the boundless range and asks for the exists query.

3. A range query on a subkey needs the field to be indexed. A flattened field can be mapped with index: false, which skips the inverted index and keeps only doc values, the columnar per-document storage covered in the next section. Exact-match queries survive that. With no terms to look up, Elasticsearch scans the doc values column instead, which is slower but gives the same answer. Range queries have no equivalent fallback, so a range on a subkey of an unindexed flattened field throws an Exception.

All three are limits on the Lucene query the mapper is willing to build, and where you notice them depends on how you query.

On the search API, where you name labels.region directly, they come back as errors.

In ES|QL you will not see them as errors at all. There, the same limits decide only whether a predicate is pushed into Lucene or runs in the compute engine on the extracted column. 

This is pushed to a term query on the keyed field:

This is not, so the filter runs per row on the extracted keyword:

Same answer, more work. That distinction is the subject of the second half of this post.

Under the hood: how range queries stay inside key boundaries

This part is internal. You don’t need it to use the field, but it explains where the bounds rule comes from.

For a handful of documents in one segment, the shared term list looks like this:

Each key owns a contiguous slice of that list. For example, all values for key "region" are clustered together in an ordered sublist. A range with both bounds set encodes each bound the same way a term is encoded, so a lower bound of "us-east" on the region key becomes region\0us-east and an upper bound of "us-west" becomes region\0us-west. Both endpoints already carry the key prefix, so the scan can’t leave the region slice. Nothing special is needed.

The half-open case is problematic. Handing Lucene a lower bound of region\0us-east with no upper bound would scan to the end of the term list, straight through tenant_id\0acme and every other key that sorts after region. So the mapper substitutes a sentinel for the missing side:

• A missing lower bound becomes key\0, inclusive. That’s the encoding of the empty value, and it’s the first term in the key's slice.

• A missing upper bound becomes key\1, exclusive. Byte 0x01 is the next byte after the 0x00 separator, so it sorts after every key\0value term and before the first term of any other key.

A one-sided range is therefore boxed into [key\0, key\1), which makes it exactly as safe as a closed one.

The upper sentinel also covers the case where one key is a prefix of another. If an index holds both region and regionx, then region\1 still sorts below regionx\0eu-west-1, because 0x01 is smaller than the x that follows the shared region prefix. A range on region cannot leak into regionx.

How flattened fields use the inverted index and doc values

Each leaf value can be written into two Lucene structures. Both are enabled by default, but can be disabled by the mapping configuration.

• The inverted index (when the field is indexed). Two untokenized keyword terms are indexed per value, one on the root path and one on the keyed path. This powers term, prefix, and range searches.

• Doc values (when doc_values is enabled). A columnar, document-ordered structure. This powers sorting, aggregations, and ES|QL reads.

The inverted index answers the question: "Which documents contain this term?" 

Doc values answer another question: "For this document, what are the values?" Doc values are laid out column by column so a scan touches only the bytes it needs. A flattened field uses both inverted indexes and doc values, so it can serve search and analytics from the same field.

The nature of the index means that the root field is only present in some cases. When the inverted index is disabled by the mapping, any value search requires a linear scan of the doc values for the searched value. In this case, when performing a search on the root field, there isn’t much additional overhead compared to just scanning the keyed field and ignoring the key markers. So the flattened mapper skips writing the root field, relying on the keyed field for both root and keyed queries.

Why flattened fields switched from dictionary to binary doc values

Historically, flattened field doc values used Lucene's SortedSetDocValues, which is a dictionary-compressed format. This means that every key\0value value indexed across all documents per segment is stored in one big sorted, deduplicated set of values. Each document tracks a list of ordinals into that value set.

This dictionary approach provides great compression for low-cardinality fields that tend to have repeated values. It is byte-efficient to store a value only once, and then refer to it by a single integer value. However, there is overhead associated with building and maintaining that dictionary of values, and that overhead is wasted effort when operating on high-cardinality fields that don’t repeat values.

Because flattened fields are a catch-all type, their cardinality tends to be very high. So while the dictionary approach works, it’s not the most compact or scan-friendly layout for this data, especially in time-series indices where flattened bags are common and storage pressure is real.

Recent versions switched the storage to use Lucene’s BinaryDocValues. This format just stores a literal binary blob for each document, which is compressed using Zstandard by our doc values codec when written to disk.

This new binary format provides an additional benefit: it allows us to maintain original array ordering without any overhead. Flattened fields support the mapping parameter preserve_leaf_arrays, which affects how multivalued fields are returned when using synthetic source. When configured to preserve_leaf_arrays: exact, returned values preserve the order, duplicates, and nulls from the original source value.

The dictionary encoding inherent to sorted-set doc values means the returned values are sorted, deduplicated, and de-nulled. To implement preserve_leaf_arrays, flattened fields have traditionally used an additional sidecar field, tracking the required metadata to reconstruct the original source value. However, the nature of binary doc values means that this sidecar field is no longer needed. The values are just stored and returned as indexed.

Limitations of schema on read with flattened fields

The limitations below follow from the keyword-only rule and the shared keyed field:

• No numeric, date, or Boolean typing on dynamic keys. 100 and "100" are the same term.

• No fuzzy, regexp, or wildcard on a specific subkey.

• No multi-fields (fields) or copy_to on the flattened field.

• A depth_limit (default 20) on how deeply nested the object can be.

If you need real typing for a known key, flattened now supports explicitly mapped subfields: you declare individual keys with real types under a properties block, and those keys are indexed by their own typed mapper instead of the keyed field. You get numeric ranges on labels.status_code, while everything else in labels stays dynamic and keyword-only. This is the escape hatch for the handful of keys you actually know about.

Schema on read: querying flattened field JSON keys in ES|QL

Search has supported flattened fields for years. ES|QL, the newer, piped query language built on a columnar compute engine, now reads flattened fields, too. Support began in the Technical Preview of Elasticsearch 9.5.0. It has two distinct pieces.

What ES|QL returns when you select a flattened field

ES|QL uses a dedicated data type for flattened fields, rather than folding it into keyword. When you select the root, you get the whole object back as a JSON string:

Keys come back sorted. You can carry this value through a query, count it, group by it, and run the multi-value and comparison functions on it. But an opaque JSON blob is not usually what you want to filter or aggregate on. For that, you need to reach inside it and process its contents.

How FIELD_EXTRACT reads JSON keys from flattened fields

There is no dotted-path syntax for dynamic keys in ES|QL. You cannot write labels.region for an unmapped key, because to the engine the flattened root is a single leaf value, not a set of columns. Instead you use a function:

The absence of a dotted-path syntax is a tentative limitation. The keyed field already addresses individual subkeys, so a more natural syntax for reaching into a flattened root is something we are planning to support.

FIELD_EXTRACT(field, path) takes a flattened field and a key, and returns a keyword. The rules are easier to see against a document. Take this one:

And this query:

The result is:

Four things to take from this:

• The dot is part of the key, not a navigation operator. The mapper already collapsed the nested object into the flat key k8s.pod, so "k8s.pod is a direct lookup, not a walk from k8s to pod.

• Matching is exact. "k8s" returns null because there is no leaf stored at k8s, only at k8s.pod and k8s.node. For the same reason, "host" will not find "host.name", and matching is case-sensitive, so "Region" will not find "region".

• Arrays come back multi-valued. "tags" yields a multi-valued keyword you can MV_EXPAND, count, or filter on. A missing key yields null.

• Everything is a keyword. "retries" comes back as the string "4", and a Boolean leaf comes back as "true" or "false".

JSONPath syntax is rejected outright, at parse time rather than per row. Both FIELD_EXTRACT(labels, "['k8s.pod']") and FIELD_EXTRACT(labels, "tags[0]") fail with field_extract path must be a literal flattened sub-field name.

Once extracted, the value is an ordinary keyword column. You can filter on it, group by it, sort by it, or use it as the join key in a LOOKUP JOIN:

How ES|QL pushes flattened field predicates into the columnar store

The obvious way to implement FIELD_EXTRACT would be to read the whole flattened root out of storage, render it as a JSON string, hand it to the compute engine, and parse it once per row to pull out a single key. But that means reading every key to use once and paying for a JSON parse on every document. So this obvious implementation is not performant.

ES|QL avoids this whenever possible. Between storage and the compute engine sits the block loader, the step that turns stored data into the columnar blocks the engine operates on. FIELD_EXTRACT hooks into that step instead of running after it.

When ES|QL loads a column, the flattened field type inspects the request. If the request is an extraction of a single constant key and the field has doc values, it routes straight to the keyed doc-values loader, which reads the key\0value entries for only that key out of the columnar structure. The column that arrives at the compute engine already contains only that key's values. The JSON string is never built and never parsed, and the other keys in the object are never read.

When extraction can’t be fused into the block loader, for example, because the key is computed per row or the root is the output of another function such as CASE, ES|QL falls back to the parse-per-row path. The results are identical either way. Only the cost changes.

The comparison can push down further. A predicate like FIELD_EXTRACT(labels, "region") == "us-east-1" can be pushed to Lucene as a term query against the synthetic keyed field, the same region\0us-east-1 term the search path uses. So a filter on an extracted subkey can be answered by the inverted index (if available), and the projection can be answered by doc values, exactly like a first-class field, even though the key was never in the mapping.

Ordering comparisons push down, too. The four, single-sided comparators (>, >=, <, <=) and closed BETWEEN-style ranges all become a range query on that same synthetic keyed field. This is where the key\0 / key\1 sentinels earn their keep: the single-sided forms are only pushable because the mapper can box an open bound inside the key. The pushed range is treated as a candidate, and the predicate is re-evaluated on the extracted keyword column afterwards, so multi-valued keys don’t slip through.

The values are keywords, so the ordering is lexicographic, not numeric. FIELD_EXTRACT(labels, "retries") > "10" compares strings, which means "9" is greater than "10". If you need numeric ranges on a key, map it explicitly under properties, or cast the value in ESQL.

Explicitly mapped subfields behave differently on purpose. Because they carry real types, comparison semantics diverge from the keyword path, so they are loaded and compared through their own typed mapper rather than fused into the keyed loader. And when you select a flattened root that has mapped subfields, ES|QL loads it from _source, so every leaf renders as a string and no keys are dropped silently.

When to use flattened fields vs. dynamic mapping in Elasticsearch

Use flattened fields when:

• The set of keys is open-ended or unknown ahead of time.

• You would otherwise cause a mapping explosion.

• Keyword-level filtering and grouping on the values is enough, and you don’t need numeric or date semantics on the dynamic keys.

• You have a few keys that do need real types. Map those explicitly under properties, and let the rest stay dynamic.

Avoid it, or map fields normally, when the schema is stable and you need full-text analysis, numeric aggregation, or date math across the board.

Remember that flattened is not a dumping ground for JSON you have given up on. It’s a real columnar-and-inverted store for schemaless data, and with ES|QL support, it’s now a first-class analytical citizen. You can keep the messy, unmapped parts of your data messy, and still filter, group, join, and aggregate across them as needed.

Does App A v1.2.3 support Feature X? Is it compatible with App B v9.8.7 when running on Kubernetes? Will I need more JVM heap space?

These are the kinds of questions that our field teams handle constantly. Answering them is harder than it looks. Documentation offers context, but it's an abstraction that can't anticipate every possible question. When we hit one it doesn't cover, our options are to interrupt an engineer who should be developing code or to hunt through that code ourselves. Often we don't have the time or expertise to navigate that much of it.

Coding agents do this well for a single repository on your laptop. Wouldn't it be great if we could scale that to our entire code estate? As a field engineer, I wanted that capability to serve my customers: agentic code intelligence across every project, dependency, platform, and version that we support. And I wanted it grounded in linked citations and always available to everyone as a service.

So I built it with Elasticsearch and Elastic Agent Builder and packaged it into a command line interface (CLI). I released it under an Apache 2.0 license and called it Sourcerer. This blog post reports multiple performance benchmarks of Sourcerer as a code research agent and walks through the design and rationale of its implementation.

Sourcerer

Sourcerer explores code like a frontier coding agent, searching across many versioned repositories as fast as it would in a single repository, and it generates answers with linked citations that establish trust.

Sourcerer consists of:

1. A set of configuration files for tools, skills, and agents in Agent Builder.

2. A set of index templates to store and search code from Git commit snapshots.

3. A CLI to install those assets and index and prune commit snapshots from remote Git repositories.

At Elastic, we're using Sourcerer to support our customers with verifiable information about our software directly from the source. Our internal deployment has indexed over a billion lines of code from our own public and private repositories. It also includes our core dependencies, such as Apache Lucene and OpenJDK, along with our common integrations, like Kubernetes and OpenTelemetry. Our solution architects, customer architects, consulting architects, and support engineers no longer have to hunt for answers in documentation or reach out to our engineers who should be building software rather than supporting it.

Code search benchmarks

Agentic code retrieval

SWE-Explore is a new benchmark, published on June 5, 2026, by Zhang et al., that evaluates "how well coding agents explore, localize, and rank repository context." It appears to be the only benchmark that specifically tests agentic code retrieval quality. I ran the benchmark with Sourcerer to see how it performs and compares to the other coding agents from the original paper, and again with Claude Code to measure and compare its token usage and task durations with Sourcerer's.

Retrieval scores

Sourcerer performs as well as frontier coding agents on relevance metrics for code retrieval (see Figure 1). The composite retrieval score is the arithmetic mean of all retrieval metrics weighed by their Pearson correlations (r) as reported in the paper; I did this to rank the agents by a measurement of overall retrieval quality. Sourcerer trailed Claude Code by 0.002 and surpassed Codex by 0.022 on a 0.0–1.0 scale, which should be interpreted as a statistical tie, given that the results vary slightly on each run due to the indeterminism of large language models (LLMs).

Generally, all the coding agents, including Sourcerer, performed well on precision metrics and suboptimally on recall metrics, although recall metrics had lower Pearson correlations and thus less importance. Table 1 shows Sourcerer's retrieval scores alongside the scores of the other agents tested in the original paper (page 8, table 6). "SignalReg" is the inverse of what the authors called "NoiseReg" (that is, 1 – NoiseReg); I did this to keep that metric consistent with the other metrics whose ranges imply that higher is better.

Token usage and task duration

Zhang et al. didn’t publish metrics for token usage or task durations, so I ran the benchmark again to capture those metrics for Claude Code. Given the high token cost of running the benchmark with any agent that uses an LLM, I opted to test only one coding agent, and Claude Code was the one that I expected most people would find useful as a comparison.

Compared to Claude Code, Sourcerer used ~13.9% more tokens to complete all 848 benchmark tasks. Sourcerer used 156,379,977 input tokens and 1,612,175 output tokens, while Claude Code used 137,629,435 input tokens and 1,137,176 output tokens. Sourcerer took ~8.9% longer to complete all 848 benchmark tasks. Sourcerer took 40,717 seconds, and Claude Code took 37,460 seconds.

I view these results on token usage and task duration as an acceptable modest tax in exchange for efficiently searching across multiple repositories and versions. That said, there’s room to explore optimizations to Sourcerer's tools, skills, and system prompt, the harness of Agent Builder, or the search engine of Elasticsearch and Lucene.

Single-repo vs. multi-repo scope

Critically, the SWE-Explore benchmarks only measure the retrieval scores, token usage, and task duration of agents searching within the boundaries of a single commit snapshot of a repository for any given task. This is the typical search space of a development coding agent. Sourcerer's intended scope is much broader, covering many commit snapshots of many repositories. The benchmark on "search speed and scalability," covered next in this report, shows Sourcerer's unique advantage when searching across many repositories.

Retrieval benchmark methodology

Appendix A explains the configuration of these benchmarks in detail.

Sourcerer searched all indexed representations of the SWE-Explore-Bench dataset (see Appendix A). Both benchmark runs used the same GPT-5.4 model that was used in the original paper. Sourcerer communicated with GPT-5.4 through Elastic Inference Service (EIS), while Claude Code communicated with GPT-5.4 through a shim proxy to be compatible with the OpenAI API.

I made a best effort to ensure that Sourcerer's benchmark task prompt was like-for-like with Claude Code's (see Appendix A). Both agents received identical instructions for their roles and tasks, along with output formats, and differed only in their brief harness-specific instructions. I instructed Sourcerer not to use its repo discovery skill and instead gave it explicit repo filtering instructions, ensuring that it was on a level playing field with Claude Code, which already receives the resolved directories. An alternative could have been to leave Sourcerer's repo discovery skill active while instructing Claude Code to find the repository in a filesystem that has all the repositories for the benchmark. I left Sourcerer's system prompts and skills, in addition to its tools, unmodified from their defaults, given that we're comparing the two harnesses overall, and much of which in Claude Code is closed source and not visible or controllable anyway.

Code search speed and scalability

Coding agents tend to use external tools to match substrings or regular expressions as a first line of retrieval. LLMs are trained to use shell commands, like ls and grep, when exploring code on a filesystem. Claude Code's own built-in Grep tool invokes ripgrep. This is the behavior I wanted to reproduce in Elasticsearch.

Elasticsearch has a wildcard field type that can scale regular expression matching to billions of documents. Sourcerer mimics the inputs and outputs of grep in Elasticsearch using sourcerer.code.grep, an Elasticsearch Query Language (ES|QL) tool that performs anRLIKE query on awildcard field of an index where each document has the contents of a single line of code. Sourcerer also providessourcerer.code.search, which performs a BM25-ranked MATCH query against an analyzed text field, for relevance-ranked discovery rather than exact substring retrieval.

I benchmarked the speed of both approaches against the speed of ripgrep and grep on a filesystem using two different corpus sizes and pattern rarities. Corpus sizes included one commit (7,215,509 lines of code) and 52 commits (200,325,684 lines of code) from the elastic/elasticsearch repository. The single commit covers the release tag for v9.4.3. The 52 commits cover the latest patch release tag for every major and minor release from v6.0.1 to v9.4.3. Sourcerer searched the corpus as indexed in Elasticsearch, while ripgrep and grep searched the corpus as stored on a filesystem, reflecting their respective use cases. The regular expression patterns included one that appears rarely among the commits (DiskBBQ) and one that appears commonly among the commits (XContentType).

sourcerer.code.grep

The first search I benchmarked was a rare pattern for DiskBBQ that appears only in some commits:

.*[dD][iI][sS][kK][-_]?[bB][bB][qQ].*

Search latency (in seconds) spanning a single commit (605 matches found from 7,215,509 lines of code):

Retrieval method

Cache

p0

p50

p100

stdev

vs. sourcerer.code.grep

sourcerer.code.grep

Cold

0.069s

0.124s

0.167s

0.020s

-

sourcerer.code.grep

Warm

0.027s

0.029s

0.046s

0.005s

-

ripgrep

Cold

0.788s

0.800s

0.809s

0.005s

~6.5x slower

ripgrep

Warm

0.081s

0.088s

0.109s

0.009s

~3.0x slower

grep

Cold

3.565s

3.580s

3.821s

0.055s

~28.9x slower

grep

Warm

0.825s

0.827s

0.833s

0.002s

~28.5x slower

Search latency (in seconds) spanning 52 commits (1,041 matches found from 200,325,684 lines of code):

Retrieval method

Cache

p0

p50

p100

stdev

vs. sourcerer.code.grep

sourcerer.code.grep

Cold

0.159s

0.164s

0.270s

0.028s

-

sourcerer.code.grep

Warm

0.027s

0.031s

0.053s

0.006s

-

ripgrep

Cold

22.356s

22.364s

22.459s

0.026s

~136.4x slower

ripgrep

Warm

16.017s

16.123s

16.297s

0.058s

~520.1x slower

grep

Cold

101.962s

102.386s

104.281s

0.752s

~624.3x slower

grep

Warm

73.082s

73.507s

74.915s

0.536s

~2,371.2x slower

Table 2. p0/p50/p100/stdev retrieval speeds of sourcerer.code.grep, ripgrep , and grep, under cold and warm caches, at two corpus scopes (20 runs per method per cache state; three warmup runs discarded before each warm-cache measurement). All percentiles computed via linear interpolation. Ratios are computed against sourcerer.code.grep's p50 at the matching cache state. See the “Methodology” section for cache definitions and query/command syntax.

The wildcard field type indexes trigrams of each value and uses them as a filter to narrow the candidate set for a regular expression before verifying full matches. For a selective pattern like this one (605 matches out of 7.2 million lines, 1041 out of 200 million) this approaches sublinear time complexity relative to corpus size. ripgrep and grep both perform scans with linear time complexity, with ripgrep using multithreading and single instruction, multiple data–accelerated (SIMD-accelerated) literal prefiltering to speed up searches, but neither has a mechanism to skip the vast majority of a corpus the way that an indexed trigram search can.

This shows up starkly in how each approach scales. Going from the single-commit corpus to the all-commits corpus is a 27.8x increase in line count. sourcerer.code.grep warm-cache time barely moves from 0.029 seconds to 0.031 seconds. ripgrep's warm-cache time goes from 0.089 seconds to 16.1 seconds, a 183x change; and grep's goes from 0.83 seconds to 1.2 minutes, an 89x change. Both filesystem tools scale worse than linearly with corpus size on this hardware, while the indexed approach is nearly flat.

The second search I benchmarked was a common pattern for XContentType that appears in all commits:

.*[xX][cC][oO][nN][tT][eE][nN][tT][tT][yY][pP][eE].*

Search latency (in seconds) spanning a single commit (7,999 matches found from 7,215,509 lines of code):

Retrieval method

Cache

p0

p50

p100

stdev

vs. sourcerer.code.grep

sourcerer.code.grep

Cold

0.158s

0.172s

0.224s

0.015s

-

sourcerer.code.grep

Warm

0.055s

0.057s

0.140s

0.023s

-

ripgrep

Cold

0.794s

0.804s

0.824s

0.007s

~4.7x slower

ripgrep

Warm

0.093s

0.095s

0.096s

0.001s

~1.7x slower

grep

Cold

3.385s

3.399s

3.421s

0.010s

~19.8x slower

grep

Warm

0.681s

0.683s

0.686s

0.001s

~12.1x slower

Search latency (in seconds) spanning 52 commits (290,662 matches found from 200,325,684 lines of code):

Retrieval method

Cache

p0

p50

p100

stdev

vs. sourcerer.code.grep

sourcerer.code.grep

Cold

1.587s

1.644s

1.756s

0.047s

-

sourcerer.code.grep

Warm

1.480s

1.541s

1.627s

0.040s

-

ripgrep

Cold

22.426s

22.440s

22.548s

0.028s

~13.6x slower

ripgrep

Warm

16.021s

16.276s

16.498s

0.119s

~10.6x slower

grep

Cold

97.699s

97.922s

99.150s

0.404s

~59.6x slower

grep

Warm

68.883s

69.184s

70.374s

0.360s

~44.9x slower

Table 3. p0/p50/p100/stdev retrieval speeds for the pattern .*[xX][cC][oO][nN][tT][eE][nN][tT][tT][yY][pP][eE].* under the same conditions as Table 2.

The relative search latencies of sourcerer.code.grep compared to grep shows why the pattern's match count matters as much as the corpus size:

Corpus

Corpus size

Cache

Speed of sourcerer.code.grep with a rare pattern (DiskBBQ)

Speed of sourcerer.code.grep with a common pattern (XContentType)

1 commit

7,215,509 lines

Cold

~28.9x faster

~19.8x faster

1 commit

7,215,509 lines

Warm

~28.5x faster

~12.1x faster

52 commits

200,325,684 lines

Cold

~624.3x faster

~59.6x faster

52 commits

200,325,684 lines

Warm

~2,371.2x faster

~44.9x faster

Table 4. This table shows how much faster sourcerer.code.grep was compared to grep when searching across two different corpus sizes and two different pattern rarities.

The pattern with far more matches shows a dramatically smaller Elasticsearch advantage, most strikingly at all-commits scope, where the advantage drops from 2,371x to 45x. The reason is visible in the absolute numbers: sourcerer.code.grep's warm-cache time at all-commits scope jumps from 0.031 seconds (DiskBBQ) to 1.541 seconds (XContentType), a 49.7x increase for a 279x increase in match count, while grep's warm-cache time barely changes (73.5 seconds to 69.2 seconds, effectively flat, since it scans the same number of bytes regardless of how many of them match). The wildcard field's trigram index has sublinear-in-corpus-size behavior that comes specifically from narrowing the candidate set before verification. Once a pattern matches hundreds of thousands of lines, the bottleneck shifts from narrowing candidates to collecting and serializing all of them, a cost that scales with match count rather than corpus size. sourcerer.code.grep still wins by a wide margin even in this less favorable case, but the margin depends heavily on how selective the search is, not just how large the corpus is.

sourcerer.code.search

Sourcerer's other retrieval tool, sourcerer.code.search, performs a BM25-ranked MATCH query rather than an exact-substring regex match. Its speed on both patterns is included below for reference.

Pattern

Corpus scope

p0

p50

p100

stdev

Matches found

DiskBBQ

One commit

0.013s

0.017s

0.018s

0.002s

288

DiskBBQ

52 commits

0.014s

0.020s

0.046s

0.007s

493

XContentType

One commit

0.062s

0.063s

0.154s

0.020s

7,177

XContentType

52 commits

2.203s

2.359s

2.611s

0.125s

249,737

Table 5. p0/p50/p100/stdev retrieval speeds of sourcerer.code.search, warm cache only, for both patterns at both corpus scopes (20 runs per row; cold-cache figures omitted; see Methodology). Match counts are sourcerer.code.search's own, not the regex-based methods' BM25 matches on tokens rather than substrings, so these aren’t directly comparable to Tables 2–4.

What drives the speed advantage

sourcerer.code.grep outperformed ripgrep and grep at every corpus scale and pattern rarity tested, along with every cache state tested. But it wasn’t by a fixed margin. The advantage ranged from ~3–30x on a common pattern to over 2,300x on a rare pattern, because indexed and brute-force search respond to different things. The trigram narrowing of the wildcard field does less work as a pattern gets more selective, while grep and ripgrep do the same amount of work regardless of how much of the corpus happens to match. sourcerer.code.search is a third option for the cases where the exact string isn't known at all.

Figure 2: Sourcerer's sourcerer.code.grep tool outperformed ripgrep and grep in every combination of corpus size and pattern rarity benchmarked, sometimes by multiple orders of magnitude. Its outperformance was strongest when searching rare patterns in large corpora and weakest when searching common patterns in small corpora.

These search speed benchmarks show what scalable means in practice for an enterprise code search agent. It's being able to search the histories of any number of repositories at interactive speeds, just like how a developer coding agent searches the working state of a single repository on a filesystem.

Speed benchmark methodology

Appendix B explains the configuration of this benchmark. Note that while the Elasticsearch deployment had two data nodes each with the same specs as the virtual machine used for ripgrep and grep, the benchmark consisted of one index with one primary shard and one replica shard. Each search ran on a single shard, which means that sourcerer.code.grep had the same amount of vCPUs and memory available per search as ripgrep and grep, despite having twice as much capacity across the overall deployment. Having two data nodes actually incurred a slight latency penalty compared to just one data node. For brevity, I've omitted results from the benchmark with one data node. We don't recommend single-node deployments in production, so it's worth including the realistic latency overhead that comes with a multi-node deployment in this benchmark.

I compared all retrieval methods under both cold and warm caches, 20 cold runs and 20 warm runs per method. The definitions of cold and warm caches weren’t like-for-like between the Elasticsearch and filesystem benchmarks. For ripgrep and grep, a cold cache meant performing a full-page cache drop (sync; echo 3 > /proc/sys/vm/drop_caches) immediately before each cold run; and a warm cache meant three discarded warmup executions immediately followed by the 20 measured runs, with no cache drops in between. For ES|QL, a cold cache meant calling POST /_cache/clear before each run. This only clears Elasticsearch's internal caches, not the OS page caches of the data nodes, which can't be cleared by hand on Elastic Cloud Hosted (ECH). A warm cache for ES|QL meant three discarded warmup queries immediately before the 20 measured runs, to help ensure that the caches on both data nodes would be warm. sourcerer.code.search's cold-cache figures are omitted from Table 5 for the same reason discussed elsewhere in this post: Its cold-cache measurements came back statistically indistinguishable from its own warm-cache measurements, evidence that the OS-level cache-clearing limitation affects it more than it affects sourcerer.code.grep, which showed a consistent, physically sensible cold/warm gap throughout.

Code search indexing throughput

I didn't conduct a formal benchmark of indexing throughout. I'll share my general observations instead.

Typically, I see a sustained indexing throughput of 20K–25K lines per second on data nodes that each have ~16GiB RAM and ~8 vCPU on c4a-highcpu instances on Google Cloud Platform (GCP). That includes writing to a primary shard and its replica shard. I've seen throughput as high as ~60K lines per second on Elastic Cloud Serverless with Search Power set to "Performant."

An engineering team at Elastic compared Sourcerer's indexing throughput to semantic code search implementations that used either sparse vector generation (Elastic Learned Sparse EncodeR [ELSER]) or dense vector embedding generation (Jina). Sourcerer indexed ~25x faster than .elser-2-elastic and ~15x faster than .jina-embeddings-v5-text-small, while retrieval quality was similar among all of them. More concretely, what took ~6 hours to index with ELSER took ~14 minutes to index with Sourcerer.

Code search solution design and rationale

The remainder of this blog post explains the rationale for my design decisions of Sourcerer, giving expert insights for practitioners of Elasticsearch and generative AI (GenAI).

Goals

Ultimately, we want an agent that answers questions about deployed software and its supporting infrastructure by searching the primary sources of truth (the code itself) and generating verifiable responses that cite those sources so they can be trusted. Inspired by the success of coding agents with grep, my main functional goal for Sourcerer was to reproduce the search behavior of a coding agent and generate responses with citations, all using Agent Builder. My nonfunctional goals were to keep it fast and scalable, as well as accurate, when searching across many versioned repositories, while maintaining acceptable costs and ease of use. Of these goals, reproducing the search behaviors of coding agents would be the most consequential, as it would dictate the access pattern, index design, and query design, plus their effects on nonfunctional goals.

Access pattern

From a human perspective, the intended access pattern is simple: We expect to ask natural language questions about software and receive plain language answers grounded in the source of truth. From the perspective of the agent handling those questions, the intended access pattern is to reproduce the search behavior of coding agents to find what it needs. The LLMs used by coding agents are heavily trained to explore code with shell commands, likelsor find, grep or ripgrep, cat, head, tail, and so on. Claude Code's built-in tools, such as Glob and Grep, provide similar functions.

I chose to go with the grain of how models are trained. So my intended access pattern for Sourcerer was to reproduce the names, inputs, and outputs of shell commands as ES|QL tools in Agent Builder. That way, the Agent Builder harness would allow an LLM to use its trained intuition to achieve similar results as a frontier harness, like Claude Code or Codex, without having to fill the LLM's limited context window with instructions for using a different search interface. This was the main context engineering problem to solve with Sourcerer. Solving it would enable faster searches that scale across many repositories at once, allowing an agent to answer questions about deployments in which many different versioned software projects work together.

Index design

I decided on three index templates to fulfill this access pattern:

• sourcerer-refs: Each document indexes the high-level metadata for a single Git reference or ref identified by its unique commit hash, which can have a tag name or branch name associated with it. A ref represents an entire snapshot of a repository at a point in time. This is a small index. The agent mainly uses this to discover the repositories and snapshots that are available to search.

• sourcerer-files: Each document indexes the metadata for a single file of a given ref. This is a larger index. The agent mainly uses this to navigate files and directories using ls semantics.

• sourcerer-lines: Each document indexes the contents of a single, numbered line of code for a given file of a given ref. Yes, every line of code becomes a document. This is the largest index (but perhaps not as large as you might expect). The agent mainly uses this to search and view code using grep and cat semantics.

The design is almost entirely denormalized. Each index has the same namespacing fields for fast, joinless filtering. Each indexed ref stores all files and lines from its commit snapshot, rather than storing diffs and reconstructing them at search time or storing unique files and lines with a mutable array of ref names associated with each. These choices trade duplicative storage (the cheapest compute resource) for faster searches and less segment merging pressure.

Namespacing

All three indices use four fields to namespace the ref, file, or line of code:

• git.host: A Git hosting provider (for example, github, gitlab).

• git.org: An account name (for example, elastic).

• git.repo: A Git repository (for example, elasticsearch, kibana).

• git.commit: A commit hash, stored as the full 40-character SHA-1 digest for integrity.

The document _id hashes for files and lines are also namespaced by {git.host}, {git.org}, {git.repo}, and {git.commit} to allow for idempotent indexing. That means you can safely rerun an indexing job without duplicating any documents.

Likewise, the index names are namespaced with the same semantics, using tildes (~) as a reliable separator since it's a disallowed character in Git repository names and organization names:

• sourcerer-v*-files~{git.host}~{git.org}~{git.repo}

• sourcerer-v*-lines~{git.host}~{git.org}~{git.repo}

This namespace convention has many benefits:

• Agents can quickly narrow the search space for refs and files, along with lines of code, by these common scoping fields, keeping searches fast and focused.

• The semantics reflect common permission boundaries. You can reproduce the access policies of your Git hosting provider by implementing your choice of index-level security and/or document-level security based on host or organization or based on repository.

• You can instantly delete indices for a whole repository or organization, or for a host, without an expensive _delete_by_query.

• The index names are future proofed for different levels of granularity. Sourcerer might eventually allow indexing code by {git.host}, {git.host}~{git.org}, or {git.host}~{git.org}~{git.repo}~{git.commit} for selective shard sizing optimizations. The query syntax would be unaffected because they target index aliases (sourcerer-files and sourcerer-lines), not individual indices.

Settings

Three index settings help to optimize storage costs and search speed:

• Index sorting gives faster searches and better compression at the cost of reduced indexing throughput. Each index sorts documents on disk by git.host, git.org, git.repo, git.commit. File and line documents are further sorted by file.path, and line documents are further sorted by line.number.

• Synthetic _source discards _source and instead reconstructs it as needed when reindexing. None of the queries access _source, which makes it dead weight. Enabling this setting reclaims ~50% storage space in the files and lines indices. While it requires an Enterprise license, enabling it on a non-licensed deployment won’t prevent the index from being created; instead the setting will be ignored.

• best_compression is a fallback for Elastic deployments that lack an Enterprise license to use synthetic _source. It provides decent compression for _source (~11% storage savings by my observations) in exchange for a modest tax on indexing throughput (~15% slower), while search speeds are essentially unaffected because the queries don't fetch _source.

I use index aliases to support zero-downtime upgrades when reindexing to a new schema.

Mappings

The indices mainly use keyword fields. They facilitate efficient filtering with basic wildcard support and aggregations, along with optimal storage usage and indexing throughput.

The line.content field is indexed both as a wildcard field and as a text field with tokenization and similarity settings tuned for code search. This gives agents the option to search code using familiar and effective grep-like regular expressions on the wildcard field or using the inverted index of the text field to return only the highest ranking matched lines to reduce token usage and increase search speed. Both options are remarkably fast and scalable, taking milliseconds to finish in most cases.

Shards

While shards are becoming less relevant with the rise of stateless platforms like Elastic Cloud Serverless, I want the solution to accommodate all deployment modes of Elasticsearch. So I've given attention to the effect of the index design on the number and size of primary shards. Based on my observations, I expect that most users won’t need to give much attention to shards.

By default, Elasticsearch enforces a soft limit of 1,000 shards per data node (including replicas). That means this solution will hit a soft limit of just under 250 repositories indexed per data node, because each repository is written to a files index and a lines index, each with one primary shard and one replica. Additionally, there’s a conventional best practice of limiting shard sizes to ~50GB, which affects how many refs you can index per repository. Both of these limits can be pushed a bit. But they reveal that this solution design really is optimized for its intended use case of searching the commit snapshots of supported, deployed software. You wouldn't use Sourcerer to index every repository on the Internet, and you shouldn't use it to index every ephemeral development branch. Plus, you should decide how many refs are worth retaining for each repository.

Repository-level granularity of indices appears to strike the right balance of shard counts and shard sizes. For reference, Kibana is one of the largest repositories on GitHub (source). I observed its shard size to be a manageable 60GB–75GB when retaining only the latest patch release for every major and minor version release from v6.0.0 to v9.5.0. That's great coverage for the Elastic deployments we see in the wild. If that's one of the largest repositories out there, you can expect just about any other repository to fit in a single shard as long as you have a reasonable retention policy, which I discuss in the next section (“Pruning”).

Pruning

By default, Sourcerer retains everything you index. Pruning lets you delete old refs to prevent unbounded growth. You can define ref retention policies based on the age of refs and the number of refs indexed in the repo. You can also define these policies based on the number of semantic versions indexed in the repo at any level of granularity for major, minor, patch, build, and prerelease versions. Some common configurations are to retain only the latest commit of the default branch or the most recent patch release tag for every major and minor version release tag.

Elastic Agent Builder tools

With the index design in place, we can review the tools that query those indices.

ES|QL tools in Agent Builder are parameterized ES|QL queries with descriptions to guide the agent's use of them. Sourcerer has tools for several purposes: repo discovery, file discovery, code search, and code display. These tools reproduce the names, inputs, and outputs of shell commands that coding agents prefer to use when exploring code, making them intuitive enough for the LLM to use with minimal instructions passed into its context window.

Repo discovery

These are typically the first tools that the agent calls. Unlike most coding agents, which search within a single repository on a filesystem, Sourcerer is aware that its search space likely has multiple repositories and versions, and so its first step is to decide which repos and refs to scope its searches to.

• sourcerer.repos.list: Lists the repos that are available to search.

• sourcerer.repos.search: Lists the repos whose file contents best match a given query.

• sourcerer.refs.list: Lists the repos and refs that are available to search.

File discovery

All file discovery tools support glob matching (* and **) on file paths.

• sourcerer.files.ls: Lists files and directories that match a given pattern.

• sourcerer.files.tree: Lists files and directories that match a given pattern in a tree-like format.

• sourcerer.files.wc: Counts lines, words, characters, bytes, and longest lines for each matching file.

Code search

All file code search tools support glob matching (* and **) on file paths.

• sourcerer.code.grep: Searches lines of code using RLIKE on a wildcard field for rapid execution of regular expressions.

• sourcerer.code.search: Searches lines of code using MATCH on a text field that has been tuned for code search.

Code retrieval

All file code retrieval tools concatenate the desired lines of any matching file and return them as a single, contiguous block of code in grep -n format, which is a format preferred by coding agents, including Claude Code's built-in Grep tool. This lets the agent see a faithful representation of file contents with line-level attribution for precise citations, without requiring the agent to reconstruct the contents or infer line numbers through reasoning.

To illustrate, here's how the sourcerer.files.head tool formats the first five lines of Kibana's README.md file, reconstructed from five documents from the lines index:

All file code retrieval tools support glob matching (* and **) on file paths. Agents typically use these tools to display the contents of a single file.

• sourcerer.files.cat: Concatenates and displays all lines for each matching file.

• sourcerer.files.head: Concatenates and displays the first n lines for each matching file.

• sourcerer.files.tail: Concatenates and displays the last n lines for each matching file.

• sourcerer.files.read_lines: Concatenates and displays the range of lines between two given line numbers for each matching file.

Agent Builder skills

With the tools implemented, we can review the skills that guide the agent's proper use of them. Agents typically invoke the follow skills in this order:

• sourcerer-repo-discovery: Guides the agent in discovering and selecting the repositories that are available to search for a given prompt. While this is typically the first skill an agent invokes, the agent might return to it when tracing dependencies from other repositories or when answering questions that span multiple repositories or versions.

• sourcerer-ref-resolution: Guides the agent in resolving the names of tags or branches to their unique, immutable commit hashes. This lets the agent reliably filter its searches to a single commit snapshot.

• sourcerer-code-search: Guides the agent in exploring code, with basic best practices on when and how to use the available tools for maximum efficiency.

• sourcerer-code-citations: Guides the agent in citing files, directories, lines of code, and ranges of lines of code. Sourcerer auto-generates more specific citation skills for each major Git hosting provider, so that its citation links conform to the URL formats of each respective host.

Agent system prompt

The final packaging of the agent comes with a system prompt that succinctly describes the agent's role and its high-level instructions. The configure file that defines the system prompt also defines the tools and skills that are made available to the agent, so that it can only execute what we permit it to execute.

Sourcerer CLI

The Sourcerer CLI assists with setup and indexing, along with pruning to keep operations simple. Configuration is managed through a sourcerer.yml configuration file.

• sourcerer setup: Idempotently loads the index templates and Agent Builder configurations, in addition to Kibana dashboards. This is typically a one-time operation and takes a few seconds.

• sourcerer index: Checks for new refs that match patterns defined in sourcerer.yml and then idempotently indexes them, skipping any refs that have already been indexed or that qualify for pruning based on retention policies. It calls git to clone repos and to list remote refs, as well as to  check out refs.

• sourcerer prune: Checks the retention policies for any refs that qualify for pruning and then deletes them from all three indices using _delete_by_query.

You can easily schedule indexing and pruning using external schedulers, like cron. For our internal use at Elastic, we maintain sourcerer.yml files in a private Git repository and schedule indexing and pruning with GitHub Actions. I prefer to index code frequently, while pruning outside of normal working hours to prevent agents from suddenly losing context in the middle of a conversation.

Feature license summary

For transparency, here’s a summary of the license levels for all non–open source software (non-OSS) features referenced in this solution design.

Subscription features:

• Agent Builder (optional; you can query the indices from a different harness using the Elasticsearch API).

• Synthetic _source (optional).

• Document-level security (optional).

Free features, proprietary to Elastic (not OSS as defined by the Open Source Initiative [OSI]):

• ES|QL.

• wildcard field.

A subscription gives you the magic of "everything just works" with Agent Builder, along with resource optimizations, finer security permissions, and platform support. Without a subscription, you can still index and prune code with the Sourcerer CLI and search the code using the Elasticsearch API.

Conclusion

Sourcerer demonstrates that Elasticsearch + Agent Builder is an exceptional solution for agentic enterprise code intelligence, as evidenced in many ways:

• Accurate: Sourcerer's agentic code retrieval quality is at parity with frontier coding agents as demonstrated by its performance on the academic benchmark SWE-Explore.

• Fast: Sourcerer's query speed ranges from milliseconds in a single repository to multiple seconds across a billion lines of code. Indexing is also much faster than could be achieved with vector embedding generations.

• Scalable: Elasticsearch sharding enables horizontal scaling, making it possible to search the current and historical states of an entire enterprise software estate. Alternatively, the stateless architecture of Serverless naturally scales without having to plan shards.

• Resilient: Elasticsearch replication enables high availability to keep the agent operational 24/7. Likewise, the stateless architecture of Serverless naturally provides high availability.

• Polyglot: Sourcerer's indexing and retrieval methods are completely language-agnostic and tolerant of malformed code.

• Secure: Elasticsearch document-level security enforces access policies for humans and agents at the organization, repository, and commit levels.

• Efficient: Sourcerer demonstrates an efficient use of storage, memory, compute, and token consumption for its intended use case.

• Manageable: The Sourcerer CLI, and its use of native git commands and Elastic REST APIs, makes it easy to get started with and operate, as well as schedule.

• Universal: Organizations from all industries build software, and Git is the source of truth for ~85% of them (source). Sourcerer has value to all of these organizations. 

Sourcerer is currently less than two months old, and the benchmark results suggest that there’s room to improve recall and token efficiency, along with task duration. I'll continue to work on this project in the near future.

Try it yourself

Sourcerer depends on Elasticsearch and Kibana. You can get started with those in a couple ways:

• Elastic Cloud (includes an Enterprise trial).

• Local setup with Docker (includes an Enterprise trial).

Then you can get started with Sourcerer.

I built this with Agent Builder. What will you build?

References

Sen, S. (2026). Is Grep All You Need? How Agent Harnesses Reshape Agentic Search. arXiv. https://arxiv.org/abs/2605.15184

Zhang, S. (2026). SWE-Explore: Benchmarking How Coding Agents Explore Repositories. arXiv. https://arxiv.org/abs/2606.07297

Appendices

Appendix A. SWE-Explore benchmark configuration

Benchmark environment:

• Sourcerer version: v1.0.0 (commit hash: 26d2e84e3f9e4c1e1598a48532eba9a739465c3e)

• Elastic Cloud Hosted (ECH):

• Region: GCP - Los Angeles (us-west2)

• CPU Optimized hardware (c4a-highcpu)

• 3x Elasticsearch data nodes (each with 16GiB RAM, 8 vCPUs)

• 2x Kibana instances (each with 2GiB RAM)

• Elastic stack version: v9.5.0 (commit hash: dbedef007f580447413782705acb9afec41f945b)

• Claude Code version: 2.1.202

• LLM: GPT-5.4

Indices as reported by GET /_cat/indices:

Benchmark task prompt:

Table 6. This table compares Claude Code's task prompt used in the original paper and Sourcerer's task prompt used in this benchmark run. The highlights indicate where Sourcerer's task prompt differed from Claude Code's: red indicates an instruction from Claude Code's task prompt that wasn't used by Sourcerer's, and green indicates an instruction that was unique to Sourcerer's task prompt.

Appendix B. Code search speed and scalability benchmark configuration

Benchmark environment:

• Sourcerer version: v2.0.0 (commit hash: dcc373b3bf7b24168e00ed7b172aea7227f705f9)

• Elastic Cloud Hosted (ECH) for ES|QL:

• Region: GCP - Los Angeles (us-west2)

• CPU Optimized hardware (c4a-highcpu)

• 2x Elasticsearch data nodes (each with 16GiB RAM, 8 vCPUs)

• 1x Kibana instances (2GiB RAM)

• Elastic stack version: v9.5.0 (commit hash: 8d4246a64bc255212407b1b313fe402391299c88)

• Virtual machine for ripgrep and grep:

• Region: GCP - Los Angeles (us-west2-a)

• CPU Optimized hardware (c4a-highcpu-8-lssd)

• 16GiB RAM

• Image: projects/ubuntu-os-cloud/global/images/ubuntu-2404-noble-arm64-v20260717

• Provisioned IOPS: 3300

• Provisioned throughput: 215

Indices as reported by GET /_cat/indices:

Cluster settings adjusted to avoid truncating the returned match count on frequent patterns:

Regular expression used by ripgrep and grep:

• DiskBBQ: .*[dD][iI][sS][kK][-_]?[bB][bB][qQ].*

• XContentType: .*[xX][cC][oO][nN][tT][eE][nN][tT][tT][yY][pP][eE].*

ES|QL query syntax used for sourcerer.code.grep:

ES|QL query syntax used for sourcerer.code.search:

ES|QL query parameters used in all tests:

• git_host="github"

• git_org="elastic"

• git_repo="elasticsearch"

• n=1000000

ES|QL query parameters used for specific tests:

• Corpus with single commit: git_commit="45f6a06b1b441b41fe711059b8720013173e7c89"

• Corpus with all commits: git_commit="*"

• sourcerer.code.grep pattern for rare pattern (DiskBBQ): regex=".*[dD][iI][sS][kK][-_]?[bB][bB][qQ].*"

• sourcerer.code.grep pattern for common pattern (XContentType): regex=".*[xX][cC][oO][nN][tT][eE][nN][tT][tT][yY][pP][eE].*"

• sourcerer.code.search pattern for common pattern (DiskBBQ): q="DiskBBQ"

• sourcerer.code.search pattern for common pattern (XContentType): q="XContentType"

n was set high enough to recover the true, uncapped match count for each pattern (605 and 1,041 matches for DiskBBQ;  7,999 and 290,662 matches for XContentType), confirmed in each case by comparing hits_returned against ripgrep's and grep's own match counts on the same underlying data.

Command syntax used for ripgrep :

rg -n --no-ignore -j 6

Command syntax used for grep:

grep -E -r -n --binary-files=without-match --exclude-dir=.git

Directories of each cloned repository snapshot and the approximate total sizes of their nonbinary files tracked by Git, which is the search space of ripgrep and grep:

What you'll discover

In this post, you'll learn how to:

• Build judgment lists from click data to evaluate and improve relevance.

• Apply basic search tuning (field weights, boosts, query rules) informed by analytics.

• Create rank features from behavioral signals, like popularity and conversion rate.

• Understand LTR and how click data becomes training data.

• Close the feedback loop between analytics and relevance improvement.

What you'll need

• Search analytics data from the previous blogs in this series (search, click, and conversion spans in Elastic).

• An Elasticsearch index with product data (the reference project includes sample data with rank features).

• Familiarity with Elasticsearch queries (BM25 [Elasticsearch's default text scoring algorithm], rank_feature, function scores).

Search relevance techniques at a glance

Technique

Effort

What it improves

When to use

Field weight tuning

Low

Relevance for attribute-rich queries

First step; quick wins

Query rules

Low–medium

Specific high-value queries

Known bad results for specific terms

Rank features

Medium

Blending behavioral signals with text score

Popularity, conversion, freshness boosts

LTR

High

Systematic ranking from click data

When you have ≥5k labeled query-doc pairs

From search analytics to search relevance improvements

Over the past three posts, you built a full instrumentation pipeline, including search spans with search.* attributes and click tracking with position data and click-through rate (CTR) / Mean Reciprocal Rank (MRR) metrics. This pipeline also includes conversion spans tying searches to revenue. And it all sits in traces-generic.otel-default, queryable with Elasticsearch Query Language (ES|QL).

Now you have dashboards, and you know that your CTR is 28%. You also know which queries generate revenue and which ones users abandon. Plus, you can tell your product manager exactly where the funnel leaks.

Now what?

The real value of search analytics is using behavioral data to improve relevance and close feedback loops. It’s also important to make search learn from its users. Your experience tells you that measurement is necessary but not sufficient, and a dashboard that shows poor ranking doesn't improve that ranking. 

This post covers four practical areas:

1. Judgment lists: The foundation for evaluating and improving relevance.

2. Basic search tuning: Field weights, boosts, and query rules informed by analytics.

3. Rank features for personalization: Feeding behavioral signals back into ranking.

4. LTR: Training machine learning (ML) models on click data to optimize ranking automatically.

Each one draws directly from the search.* attributes you're already collecting, and they don’t require any new instrumentation.

What are judgment lists?

Before diving into specific techniques, it's worth understanding the concept that ties them all together: judgment lists.

A judgment list is a set of query-document pairs with relevance grades: For a given query, how relevant is each document? They look like this:

Query

Document

Grade

Label

"wireless headphones"

SKU-001 (Sony WH-1000XM5)

3

Highly relevant

"wireless headphones"

SKU-042 (AirPods Max)

2

Relevant

"wireless headphones"

SKU-099 (Wired earbuds)

0

Not relevant

Judgment lists serve three purposes:

• Evaluating current relevance. Given a set of queries and known-good results, how well does your search rank them? Metrics like Normalized Discounted Cumulative Gain (NDCG) and the Rank Eval API use judgment lists to score your ranking quality. This gives you a baseline before making changes.

• Measuring the impact of changes. When you adjust field weights or add synonyms, judgment lists tell you whether the change helped or hurt. The same is true when you change boosting rules. Run the same evaluation before and after: If NDCG went up, the change improved relevance for those queries.

• Training LTR models. LTR algorithms need labeled training data, such as, For this query, these documents are relevant and those aren't. Judgment lists are that training data.

Manual vs. automated judgment lists

Traditionally, judgment lists are created by human assessors who manually rate documents for a set of test queries. This works well for a small number of high-value queries (your top 50 searches, for example), but it doesn't scale. A large catalog with thousands of distinct queries and frequent inventory changes makes manual assessment impractical.

This is where your click data becomes valuable. Every click is an implicit relevance judgment; that is, a signal that for a given query, a given document was relevant enough to engage with. Conversions are even stronger signals. By aggregating this data, you can build judgment lists automatically from real user behavior, at a scale that manual assessment can't match.

The trade-off is noise. Clicks are influenced by position bias (users click higher-ranked results more often regardless of relevance) and presentation effects. They’re also influenced by accidental clicks. We'll cover techniques to handle this noise later in this post. But the key insight is that approaches like personalization and LTR lean toward automated judgment lists because it's not scalable to manually create lists for every query or user segment, nor for every inventory change.

The judgment lists guide on Search Labs covers the concept in depth, including how to structure lists for evaluation with the Rank Eval API.

How search analytics identify search relevance problems

The most immediate use of your analytics data is to identify and fix specific relevance problems. This only requires using data to direct manual improvements, no personalization or ML.

Finding problem queries with search analytics

The ES|QL queries from the click in your application give you per-query CTR and MRR. Sort by search volume descending and CTR ascending to find your highest-impact relevance failures:

This is the CTR-by-query query re-sorted to surface problems first. The WHERE searches > 5 filter removes one-off queries that would dominate the low-CTR list with small sample noise.

• Zero-CTR queries with results.  Your ranking is returning content, but none of it’s compelling. These queries often benefit from synonym expansion. You can also improve them with boosting rules or pinned results.

• Low-CTR, high-volume queries. These are the biggest relevance investment opportunities, and they affect the most users.

• Low-MRR, high-CTR queries. Users find what they need but have to scroll for it. In these instances, the relevant documents exist, but they're ranked wrong.

Try it: Close the loop in the reference project

If you have click data in the reference project(python generate_traffic.py --blog 3 --sessions 100), you can run the problem-query query above against traces-generic.otel-default and observe your lowest-CTR queries.

The reference project's app.py already uses rank_feature boosting: rank_features.popularity, rank_features.conversion_rate, rank_features.margin_score, and rank_features.freshness are indexed on every product and blended into the BM25 score at query time. To see the effect of adjusting a boost:

1. Open reference/app.py, and find the "should" clause in _build_search_query().

2. Change the "boost" value on rank_features.popularity from 2 to 5.

3. Restart the server (python app.py) and rerun generate_traffic.py --blog 3 --sessions 50.

4. Compare the top-queries and click position distribution in ES|QL before and after.

This is informed manual tuning using the behavioral data you collected, not ML. That's the pattern for the rest of this post: First, ES|QL tells you what's wrong. Then, configuration changes and, eventually, LTR models fix it.

Field weights, boosts, and scoring for search relevance

The simplest tuning lever in Elasticsearch is adjusting how different fields contribute to the relevance score. A multi_match query across title, description, and brand fields can weight the title higher because a match there is usually more relevant. Your analytics data tells you where these weights are wrong: If users consistently click products with titles that don't match the query but with descriptions that do, your title boost may be too aggressive.

Beyond field weights, you can incorporate business metrics directly into scoring. A common example is boosting by profit margin; that is, products with higher margins rank slightly higher when relevance scores are similar. The rank_feature field type is designed for exactly this: Index a numeric signal (margin, popularity, recency) alongside each document, and the rank_feature query blends it with text relevance at query time. For more complex scoring combinations, like decay functions, weighted field values, and scripts, the function_score query gives you full control. The boosting by profit and popularity guide on Search Labs walks through this in detail, including the trade-offs between additive and multiplicative boosting approaches.

Query rules for targeted search relevance fixes

For targeted interventions, query rules let you pin, boost, or exclude specific results for specific queries, and no model training is required. They're the search equivalent of a manual override.

Your analytics data tells you exactly where to apply them. When you see a zero-CTR query, like "returns policy", that consistently returns product results instead of the returns page, you can pin the returns page at position 1 for that query. If you notice a high-revenue query where the best-selling product appears at position 4, you can boost it.

Query rules are valuable precisely because they're simple. They solve known problems immediately while you build toward more sophisticated approaches. The query rules tutorial on Search Labs walks through the setup.

Building judgment lists from click data

The basic tuning above handles individual problems. To improve relevance systematically, you need judgment lists, and your click data can build them automatically.

Converting click data into relevance grades

The simplest approach is to count clicks per query-document pair and assign graded relevance:

This gives you a table of (query, document, click_count, avg_position) tuples. The grading step maps click counts to relevance levels:

Click count

Suggested grade

Label

0

0

Not relevant (never clicked for this query)

1

1

Marginally relevant

2–3

2

Relevant

4+

3

Highly relevant

The thresholds depend on your traffic volume. For a high-traffic site, you might need 10+ clicks before calling something "highly relevant." For lower traffic, even two or three clicks is a meaningful signal. The point is that click frequency across users is a stronger signal than any single click.

You can strengthen judgments further by incorporating conversion data, a document that gets clicked and added to cart is a stronger relevance signal than one that gets clicked and abandoned:

A document with five clicks and three cart additions is a stronger candidate for grade 3 than one with five clicks and zero cart additions.

Handling position bias in click data

There's a problem with raw click counts: position bias. Users click position 1 more often because they see it first, not necessarily because it's the most relevant result. Blog 3 introduced this concept when we discussed click position distribution and referenced the foundational work by Joachims et al. (2005).

Position bias matters for judgment lists because it means raw click data over-weights whatever the current ranking happens to surface first. If you train an LTR model on biased judgments, it learns to replicate the existing ranking, which defeats the purpose.

Two practical approaches to handle this:

• Position-normalized click rates. Instead of raw click counts, calculate click-through rate per position. A document clicked 3 out of 10 times when shown at position 5 is arguably more relevant than one clicked 5 out of 10 times at position 1. Position 5 gets less visibility, so a higher click rate there is a stronger relevance signal.

• Skip-above heuristics. When a user clicks position 3 but skips positions 1 and 2, those skipped documents are implicitly judged "less relevant" for that query. This is the core insight from Joachims et al.: Skipped-above results provide negative training signals. You can extract these pairs from your click data:

Queries where the minimum click position is greater than 1 are sessions where the user skipped the top result(s). The documents at those skipped positions, for those queries, are candidates for grade 0 in your judgment list, because the user saw them and chose something lower.

For production judgment list generation, the LTR tutorial on Search Labs walks through the complete pipeline. The training LTR models with user behavior data guide covers the specific workflow of deriving training data from click-through behavior, including the Clicks Over Expected Clicks (COEC) algorithm for debiasing. And the LTR Jupyter notebooks provide working Python code for feature extraction and model training.

Rank features for search relevance personalization

Basic tuning applies the same ranking to every user. Personalization means adapting results based on who’s searching (their history, preferences, or segment). Rank features are the most accessible way to do this in Elasticsearch.

Building behavioral signals from click streams

You can aggregate your click and conversion data at the document level to produce signals that reflect overall or segment-specific popularity:

These document-level signals (click popularity, cart rate, and conversion rate) are indexed as rank_feature fields on each product document. The workflow:

1. Run the ES|QL query above periodically (daily or weekly).

2. Write the results back to a click_popularity or conversion_rate rank_feature field on each product document.

3. Use a rank_feature query to blend text relevance with behavioral signals.

The rank_feature query applies a saturation function by default: Initial popularity gains matter most, diminishing as values get large. This prevents a single viral product from dominating all queries. You can tune the function's pivot point to control how much influence the feature has relative to text relevance.

Personalizing rank features by user segment

The query above produces global popularity; that is, it’s the same for every user. Personalization comes from segmenting these signals. If you're tracking enduser.pseudo.id or user.id, you can compute features per user cohort:

• Category affinity: How often does this user click products in "electronics" versus "clothing"?

• Price sensitivity: Does this user tend to click and convert on higher- or lower-priced items?

• Brand preference: Which brands does this user engage with most?

These become additional rank features, applied at query time based on who's searching. The personalized search with LTR guide on Search Labs walks through training per-user ranking models. For a lighter approach, the cohort-aware ranking guide shows how to use multiplicative boosting to personalize at the segment level without ML, just analytics-derived weights applied to rank features at query time.

Combining multiple signals with the linear retriever

When you're blending text relevance with semantic search and behavioral rank features, you need a way to combine them. Elasticsearch's linear retriever gives you precise control over how different query types contribute to the final ranking. It computes a weighted sum of normalized scores, so you can say text relevance matters 60%, semantic similarity 30%, popularity 10% and adjust those weights based on your analytics. This is unlike Reciprocal Rank Fusion (RRF), which only considers relative rank positions.

This is particularly useful for personalization because you can vary the weights per user segment. A returning customer might get more weight on purchase history, while a first-time visitor gets more weight on global popularity.

If you're also using semantic search with embedding models, the Elastic Inference Service (EIS) now offers GPU-accelerated embedding generation, including multilingual models, directly within Elastic Cloud, making it straightforward to add a semantic retriever alongside your text and behavioral signals.

How Learn To Rank uses click streams to optimize ranking

Rank features handle individual signals, but LTR handles all of them at once. It trains an ML model to combine features (like text relevance, popularity, CTR, recency, margin, or user affinity) into a single ranking function optimized for your users.

What you need for Learn To Rank in Elasticsearch

Elasticsearch has supported native Learning to Rank since version 8.12 as an Enterprise subscription feature. The pipeline:

1. Judgment list. Query-document pairs with relevance grades (built from click data, as above).

2. Feature extraction. Numeric signals for each query-document pair (BM25 score, popularity, CTR, recency, price, margin, user segment features).

3. Model training. Typically XGBoost or LambdaMART, trained on your judgment list with features.

4. Deployment. Upload the trained model to Elasticsearch via Eland, and use it as a rescorer.

The click and conversion data from this series feeds steps 1 and 2, and the judgment list is your training labels. The document-level features from the rank features section above (click popularity, conversion rate, margin) are additional features alongside text relevance scores.

Why automated judgment lists from click data scale better

This is where the scalability argument for automated judgment lists becomes concrete. A manually curated judgment list might cover your top 100 queries well, but personalized ranking needs judgment data across thousands of queries and multiple user segments. You can't hire assessors to rate results for "wireless headphones" separately for electronics enthusiasts, budget shoppers, and professional audio engineers.

Automated judgment lists from click data scale to every query your users actually run and update as inventory and user behavior change. They can also be segmented by cohort. The training LTR models with user behavior data guide demonstrates this end-to-end workflow, showing how to go from raw click events to trained ranking models.

For teams looking to close this loop even further, the agentic autotuning approach demonstrates using an AI agent to continuously monitor search quality and generate judgment lists from user interactions. It automatically retrains LTR models, turning the feedback loop into an autonomous system.

What Learn To Rank requires: Traffic, pipeline, and evaluation

LTR requires enough traffic to generate meaningful judgment lists and engineering time to build and maintain the pipeline. It also requires ongoing evaluation to ensure that the model improves over time. But for search applications with sufficient volume, it's the most effective way to make ranking learn from user behavior. The LTR introduction on Search Labs covers the full scope of what's involved.

Elasticsearch Relevance Studio: Visual search relevance tuning

Between query rules (manual, targeted) and LTR (ML, systemic), there's a middle ground: visual relevance tuning. Elasticsearch Relevance Studio is a tool for comparing and tuning search configurations side by side. It lets you adjust boost values, field weights, and query structures while seeing the results update in real time.

Analytics data, especially CTR and MRR, tells you which queries to focus on. Start from a ranked list of problem queries (the low-CTR, high-volume queries from the ES|QL analysis above), and work through them systematically in Relevance Studio, instead of guessing which searches need tuning.

The workflow:

1. Identify problem queries. Run the CTR-by-query and MRR-by-query analyses.

2. Open those queries in Relevance Studio. See current results alongside the tuned version.

3. Adjust field weights and boosting. Experiment with configuration changes.

4. Evaluate with judgment lists. Use the Rank Eval API to confirm that the change improves NDCG for your test queries.

5. Measure the impact in production. Rerun the analytics after deploying changes, and compare CTR / MRR.

This before-and-after loop is where analytics and tuning connect. With the data analytics you can prioritize the queries that affect the most users and verify that changes actually helped. Without this data, tuning is guesswork, and you're adjusting weights without knowing which queries matter or how to measure improvement. 

If you are looking at tuning and evaluating relevance you should also investigate the Elasticsearch Relevance Studio project which lets you directly compare search strategies. 

Closing the loop between search analytics and search relevance

Search relevance improvement follows a consistent loop: measure quality with ES|QL analytics, apply changes (query rules, field weights, rank features or LTR models), then measure again.

The key insight is that the same instrumentation that measures search quality also generates the data to improve it. Your search spans produce the queries to analyze, and your click spans produce the judgment data for evaluation and LTR training. Plus, your conversion spans tell you which improvements matter most to the business.

Measuring search relevance improvements after changes

After deploying a change, like a new query rule or updated field weights, or deploying an LTR model, you need to know whether it helped:

Metric

What it measures

How to compare

CTR trend

Whether more users are clicking results after the change

Run the CTR-by-query ES|QL query for the week before and after; compare per-query percentages

MRR trend

Whether clicks are shifting toward higher-ranked positions

Compare mean reciprocal rank per query before and after using the MRR-by-query analysis

Conversion rate trend

Whether more clicks are turning into purchases or cart additions

Run the click-to-conversion ES|QL query for both periods; compare cart rate percentages

Revenue per query

Whether the business impact of the change is positive

Compare revenue attributed to affected queries via conversion spans before and after deployment

Run the same ES|QL queries before and after. If your change was a query rule for "laptop bag", compare that query's CTR and MRR from the week before to the week after.

A/B testing with experiment attributes

The feature_flag.key attribute  is designed for exactly this. Route a percentage of traffic to a new ranking configuration, and set feature_flag.key to the experiment name and optionally feature_flag.result.variant to the variant (for example, "control" or "variant-boost-v2"). Then propagate it to click spans the same way you propagate `query_id`, and compare metrics between groups:

This gives you an A/B comparison of CTR by experiment variant. You don’t need a separate experimentation platform for basic comparisons, although you’ll want a proper framework for statistical rigor on sample sizes and significance.

Monitoring for search relevance regressions

Once you've identified your high-value queries, that is, the ones that drive revenue and have been tuned for good engagement, you need to protect them. Relevance regressions on your top 20 revenue-generating queries are business-critical incidents.

This is where monitoring comes in. In the next blog in the series, we'll cover setting up alerts on these metrics: CTR drops on high-revenue queries and MRR regressions after deployments. We’ll also cover conversion rate anomalies. The same ES|QL queries that power your analytics dashboards can drive alerting rules; the feedback loop covers measurement and improvement. It also provides operational protection.

Search relevance improvement roadmap: Week 1 to Quarter 1

Here's a practical starting point. You don't need to implement LTR on day one. The approaches build on each other:

• Week 1: Evaluate and fix known problems. Run the CTR-by-query analysis. Find your zero-CTR queries with high search volume, and fix the worst ones with synonyms or pinned results via query rules. You could also use adjusted field weights. Use judgment lists (even a small manual one for your top queries) to confirm that the changes improve NDCG before deploying. You should get immediate, targeted impact.

• Month 1: Add behavioral rank features. Extract document-level click popularity and conversion rates from your analytics. Index them as rank_feature fields, and blend behavioral signals with text relevance using the linear retriever or `rank_feature` queries. Then consider simple business boosts like margin. This lifts baseline quality across all queries without manual intervention per query.

• Quarter 1: Automate with LTR. Once you have enough click data (typically several weeks of production traffic), build judgment lists automatically from click and conversion data. Train an LTR model that combines text features, behavioral features, and business features, and then deploy it as a rescorer. The ranking now learns from user behavior and improves as you collect more data.

At each stage, measure the impact with the same metrics. CTR and MRR are your scorecards, as is conversion rate. If a change doesn't move them, it didn't help, regardless of how sophisticated the approach.

What's next: Search reliability engineering and SLOs

We've covered the full sequence: instrument search, measure quality, track conversions, and improve relevance. The missing piece is making sure it all keeps working.

Next, we close the series with search reliability engineering, including Service Level Objectives (SLOs) for search quality and alerting on metric regressions, along with operational dashboards that catch problems before users notice them. The same metrics you've been building become the basis for search health monitoring, turning your analytics pipeline into a reliability system.

Get started

Working code

• Reference project: Working code for the entire blog series; clone, configure, and run.

What is metric temporality in OpenTelemetry?

Metrics stores usually receive client-side, pre-aggregated metrics. For example, if an application records request response times, it won’t send each individual response time as a single data point to your metrics back end. Instead, the application (or rather the OTel SDK) pre-aggregates those raw response times into counters or histograms. These pre-aggregated values are then exported at a periodic interval, dramatically reducing the number of data points. Temporality is about how this pre-aggregation works. There are two temporality models: cumulative and delta.

Cumulative temporality in OTel metrics

With cumulative temporality, each data point represents the totalamount of change in the metric value since the process started. Values monotonically increase, with occasional reset to 0 (for example, when the process restarts).

Take a counter tracking the total CPU time consumed by a Java Virtual Machine (JVM):

Timestamp

Value

Meaning

10:01

12.4s

12.4s total CPU time since start

10:02

13.1s

13.1s total CPU time since start

10:03

13.9s

13.9s total CPU time since start

To compute the rate of change between 10:01 and 10:02, we subtract: 13.1 - 12.4 = 0.7s of CPU time was consumed in that interval. Dividing by the time range of the interval gives us the rate. This is the default temporality for counters in both Prometheus and OTel.

Delta temporality in OTel metrics

With delta temporality, each data point represents the change since the last measurement. Values are independent of each other. In other words, after each export, the OTel SDK resets all values for all series.

The same raw observations from the cumulative example above would look as follows with delta temporality.

Timestamp

Value

Meaning

10:01

0.5s

0.5s of CPU time in this interval

10:02

0.7s

0.7s of CPU time in this interval

10:03

0.8s

0.8s of CPU time in this interval

To compute the rate or increase, we can use the value directly, without any subtraction.

Trade-offs between cumulative and delta OpenTelemetry metrics

Both temporalities have practical trade-offs:

• Resilience to data loss: Cumulative counters are self-describing: If you miss an export, the next data point still gives you the correct total. Delta values are incremental, so a lost data point means that the corresponding increase is lost.

• Metric producer memory footprint: For cumulative temporality, the OTel SDKs need to keep a state for every series in memory. For delta temporality, the footprint is much lower. There, the SDKs only need to keep track of counters or histograms which changed since the last export. If there are a lot of counters or histograms and many of them don’t increase each period, this difference can be quite substantial.

• Aggregation across restarts: Cumulative counters require reset detection logic, which in edge cases can fail: If the metric value decreases, it’s detected as a reset. We assume that the application was restarted and the counter started from 0 again. This can be missed if the first reported counter value after the restart is higher than before the restart. A concrete example:

• The service consumes 1 second CPU time and restarts.

• After the restart, the service performs a CPU-intensive task and consumes 2 seconds of CPU time before the metric is exported again.

• The metric back end just sees 1 followed by 2 as the metric value. It never observes a decrease and therefore misses the reset.

Delta values don't have this problem since each value is independent.

If you’re using histograms, the trade-offs have an even bigger effect:

Trade-off

Cumulative

Delta

Histogram size

Buckets accumulate across exports, consuming more storage

Buckets reset each export, producing smaller histograms

Min/max accuracy

Approximated from buckets for custom time ranges (tracked values represent extremes since process start)

Exact per-export minimum and maximum values

Query performance

Faster: only the first and last value in a time range plus resets are needed

Slower: all histograms in the queried range must be combined

OpenTelemetry supports both models and lets you choose per SDK via the OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE environment variable.

Why native temporality support eliminates OTel Collector workarounds

Prometheus and most other metrics back ends pick a side: All metrics have to be either cumulative or delta. Elasticsearch previously followed that pattern, too, with native storage of cumulative counters and delta histograms, and workarounds for everything else. Delta counters were stored as gauges, functional but without native counter semantics for rate queries. And cumulative histograms were unsupported.

One workaround for unsupported temporalities is to configure your metric producers (for example, OTel SDKs) to produce data with the temporality that your back end supports. In large-scale deployments, this can be a very challenging task. And sometimes this isn’t even possible (for example, if you consume OTLP metrics from third-party services).

Another workaround is to convert the temporality prior to ingestion. In the OTel Collector, you would typically use the cumulative-to-delta processor, which comes with a big warning sign about statefulness. The conversion is inherently stateful, requiring ordered delivery of metric series to the same collector and persisted state across restarts. In practice, it works, but at scale, it comes with a lot of deployment headaches.

With Elasticsearch 9.5, you can skip the conversion pipeline entirely. Elasticsearch natively stores and queries metric data with both temporalities. It doesn’t require any stateful conversion required or explicit configuration of your OTel SDKs.

Demo: ingesting cumulative and delta OTel metrics side by side

To demonstrate the temporality support, we'll reuse a demo setup from our OTel histogram metrics ES|QL blog post: a Java Renaissance benchmark instrumented with the OTel Java agent. The twist this time: We run two instances of the benchmark, each configured with a different temporality:

• renaissance-delta: Exports metrics with delta temporality.

• renaissance-cumulative: Exports metrics with cumulative temporality.

Both instances report the same metrics under the same service name renaissance, but with different service.instance.id values. Here’s the relevant section of the docker-compose.yml that can be found in the companion code:

To run the demo yourself, you'll also have to fill out the managed OTLP endpoint URL and the corresponding API key:

After starting the demo with docker compose up --build, both instances will start reporting metrics to Elasticsearch.

Querying OTel counter metrics with ES|QL and PromQL

Let's query the first few raw data points of jvm.cpu.time for both instances to see the different temporalities in action:

This gives us the first five data points for each service instance:

The benchmark consumes CPU at a nearly constant rate. This is directly visible based on the delta temporality data: The values are nearly constant between exports. In contrast, the cumulative temporality values grow over time, as they represent the total CPU usage of the benchmark instance.

Now let's have a look at how to properly query this metric using PromQL:

The screenshot shows that both benchmark instances consume a nearly constant of 1 to 1.2 number of CPU cores with some variance. This query works because we made our rate implementation respect the temporality: Every time series (so every service instance in our case) stores the temporality as a metric dimension. The rate implementation looks at this dimension and interprets the data accordingly: For delta temporality, values are summed up; for cumulative temporality, a difference computation is done. This all happens automatically in the background, without requiring any changes to your queries.

We’ve adapted rate, increase, and irate to work this way. The same applies when using those functions in ES|QL TS queries:

Because Elasticsearch tracks the temporality as a dimension, you can have multiple series with different temporalities for the same metric, just like in the demo use case. Aggregating across series also works as expected, because at that point rate, increase, or irate already took care of normalizing the data:

Querying OTel histogram metrics across temporalities

Metric temporality applies to histograms in the same way it applies to counters: histogram buckets are effectively a set of counters, each tracking values in a specific range.As in our histogram demo, we use exponential histograms, where bucket boundaries adapt automatically to minimize relative error.

Due to this similarity, histograms can also be cumulative or delta. Either the counter per bucket is reset after each metric export or the cumulative count carries over between exports.

Let's query the median major garbage collection (GC) duration for our benchmark instances, which is a histogram metric:

Or the equivalent ES|QL query:

Again, both queries will automatically load the temporality per series and interpret the histograms accordingly. In PromQL, this is handled by the increase function. Note that in ES|QL, you don't explicitly call increase on histograms. The TS command automatically handles the temporality-aware merging of histograms when you use aggregation functions, like PERCENTILE, MEDIAN, or AVG.

How Elasticsearch stores metric temporality in TSDB

Elasticsearch's time series database (TSDB) stores metric temporality in a dedicated dimension field on each document. The index.time_series.temporality_field index setting lets you specify which field carries the temporality information. The field must be a keyword field with time_series_dimension: true and the permissible values "delta" or "cumulative".

As soon as this setting is present on a time series index, ES|QL and PromQL will load the corresponding field when performing temporality-dependent aggregations. If the field isn’t present or has no value on a document, we fall back to defaults based on the type of the corresponding metric: counters default to cumulative, and histograms default to delta. This matches the historical behavior and ensures existing queries and existing data continue to work without changes.

When you ingest metrics via the OTLP endpoint, Elasticsearch automatically adds a temporality dimension field to each document, populated from the OTLP AggregationTemporality metadata. For custom (neither OTLP nor Prometheus remote write) ingestion, you’ll have to manually set up the index.time_series.temporality_field setting and populate your temporality dimension.

The temporality is also respected during downsampling: As it’s a dimension, it’s preserved automatically and used to compute the aggregate values.

Getting started with mixed-temporality OTel metrics in Elasticsearch

With Elasticsearch 9.5, cumulative versus delta is no longer a decision you have to get correct at the start. Ingest both temporalities side by side, even for the same metric name, and let ES|QL and PromQL handle the rest. You can switch between both without having to touch your queries. For more details, see the metric temporality documentation.

While AI indices will be included in future Stack releases, today we recommend using Serverless.

How it works: AI Index, Kibana Workflows, and the query-ki skill

Building context in this walkthrough has three moving parts:

1. An AI Index, where KIs live. It's a regular Elasticsearch index or data stream with a specific naming convention triggering component templates to configure the right mappings automatically.

2. Kibana Workflows, which read from your data sources, run an LLM to structure content into KIs, and write those KIs into the AI Index.

3. A query-ki skill, a skill that queries KIs directly from the AI Index using ES|QL, and that a chat agent can call as a tool.

Prerequisites

This tutorial assumes you have:

1. An Elasticsearch Serverless project. You can sign up for a trial if you don't have one.

2. An API key to access your Elasticsearch project. 

Create sample indices for agent routing

First, we’ll need some sources. Sources can be data that already exists in your Elasticsearch indices, or external data accessed via connectors or ES|QL data sources. 

For this blog, we’ll create some indices with example data. We’ll start with an example using three datasets: BEIR/fiqa (financial), beir-nfcorpus (biomedical/nutrition), and beir-scifact (scientific fact-checking). Each index is populated with its own _meta.description. 

Here are the mappings we define for these indices:

Then, using the above convenience scripts, load a handful of documents into each index with the _bulk API.

Now imagine an agent with a question and the indices we’ve just created. The agent has no idea which one is relevant at the start. Without pre-computed context, it either performs exploratory lookups (mappings, test searches) to figure out which source to use, or searches all three and hopes the merged results contain something useful. Either approach costs tokens, and if you multiply that inefficiency across every query an agent makes, it adds up.

Create your AI Index

Before generating any KIs, you need an index to store them. We call this an AI Index.

The naming convention is what triggers automatic configuration. Any index whose name starts with ai-index-idx- is a regular index; ai-index-ds- is a data stream. You’ll want to choose data streams for observability use cases, time series data, and when recency is important. Conversely, standard indices are a good choice for static data that will exist for a long while, where recency is not as much of a concern, and may need to occasionally be updated on demand. This naming convention is required for AI indices. 

When Elasticsearch sees the ai-index- prefixes, it automatically applies component templates that configure the right mappings and settings.

Creating an AI Index is a single call:

To see exactly what the component templates applied, inspect the mappings:

The response shows the fields every AI Index gets out of the box:

title, description, and content are each a text field with a .semantic sub-field of type semantic_text, supporting hybrid retrieval.

Data stream indices (ai-index-ds-*) additionally carry a default 90-day data retention policy. This blog uses a standard index (ai-index-idx-*).

Index Metadata as a Knowledge Indicator

The target use case for this example is how the query-index-metadata-ki skill can route an agent to the correct Elasticsearch index, even when index or field names are vague. This reduces mistakes from choosing the wrong index or formulating queries based on incomplete schema exploration.

Since we're creating KIs for our own indices, we can give the LLM a head start: annotate index mappings with human-written _meta.description content. The workflow generates better KIs with more context to work from.

To address this, we'll manually create a Kibana Workflow that profiles each index and writes routing KIs into the AI Index. The workflow chains four steps:

Step

Type

What it does

get_mapping

elasticsearch.request

Read the mapping, including _meta.description and per-field descriptions.

sample_docs

elasticsearch.search

Pull a few real documents so the profile reflects actual value shapes.

profile_index

ai.agent

Generate a structured index profile as structured output.

sink_index_ki

elasticsearch.bulk

Write the profile into the AI Index as a KI.

Paste the following YAML into the Workflows editor:

Let's walk through what this workflow does. We loop over three specified indices with a foreach loop. For each:

1. get_mapping fetches the Elasticsearch index mappings, including any _meta.description annotations we added earlier.

2. sample_docs pulls 3 real documents. Concrete examples give the LLM much better signal than schema alone.

3. profile_index calls ai.agent with the index name, mappings, and sample documents. The LLM returns structured output describing the index's purpose, key fields, and an example ES|QL query showing how to use it.

4. sink_index_ki writes the result into the AI Index as a KI of type index_metadata_entry, keyed on the index name so re-runs are idempotent.

A few things to point out: 

• This workflow hard-codes a specific set of indices. In practice, you could derive the list from an index pattern or a dynamic source. 

• The foreach loop also runs iterations sequentially, which is fine for this guide but slow in production because each iteration involves an LLM call. For scale, use workflow.executeAsync or native parallel support. The cheat sheet has tips on both.

• In the profile_index step, the agent prompt is the special sauce. This is what shapes the accuracy and usefulness of the KIs. 

• Using ai.prompt can improve workflow efficiency (and cost) if you don’t need to load other tools. 

• Cost can be controlled in multiple ways. Richer prompts and structured output often result in higher token utilization, and of course the model you choose significantly impacts total costs. The Elastic Inference Service (EIS) can be a great playground to test different models against the profile_index’s ai.agent step to compare how different models stack up against each other when generating KIs. 

Query your AI Index to verify Knowledge Indicators

Once the beir-index-profile-ki workflow runs, query the AI Index directly in the Discover tab using the following ES|QL query to confirm what got written:

This will result in the following output: 

Build a portable skill to retrieve AI agent context

Retrieval is a critical component in an AI index. A KI is a document in the AI Index, and finding one is a single ES|QL query. We package that query as a small, portable skill so any agent can call it, regardless of the harness it runs in.

We write the skill as a SKILL.md: a YAML header with a name and description, followed by markdown instructions. This is the same Agent Skills format that many harnesses, including Claude Code, LangChain's Deep Agents, and others, load directly. 

The harness reads the header content up front, and only pulls in the full instructions when a question matches the description. The one thing the skill asks of the harness is a way to run ES|QL against Elasticsearch.

Here is a sample query-index-metadata-ki skill:

Let’s break down what the skill is doing: 

• We’re defining index-metadata-entry as a KI type/use case.

• We’re performing a hybrid ES|QL search on our AI indices, filtering by the appropriate type using RRF as the default method to fuse results.

• The KI results will directly ground the agent’s answer when determining what indices are relevant to the query.

Because the skill is just instructions plus a query, it travels wherever your agent does. You can point the same file at a Kibana Workflow agent, Claude Code, LangChain Deep Agents, or any other harness without changing a line of it.

Connect your AI Index to an agent harness 

We want to demonstrate how you can use AI indices to query your data with any harness. For these examples, we’ll use LangChain Deep Agents and an OpenAI-compatible key, but any other agent harness can be easily substituted in, including Elastic Agent Builder.

First, let’s create a baseline to see how an agent will perform without using KIs:

Here’s a modified example that could run the same agent, but now with the ability to search AI indices to return KIs: 

This agent will always query the KI indices to get the answer.

How much do Knowledge Indicators reduce agent token usage?

Since we’re using agents, the results of these scripts are non-deterministic. However, when I ran these results against the query Is there scientific evidence that vitamin D supplementation prevents cancer?, both agents led to the same conclusion, but they took different paths to get there: 

Baseline (No AI Index)

With AI Index

Total tool calls

12

8

read_file calls

0

2

get_mapping calls

3

0

esql_query calls

9

6 

Total indices queried

2 (bounced between beir-scifact and beir-nfcorpus)

1 (beir-nfcorpus)

Tokens consumed

167,763

92,711

Latency

39.58s

36.15s

Answer

Grounded, correct

Grounded, correct

The KI answers were both grounded and correct, but an interesting datapoint is the fact that the overall tool usage and token utilization was smaller when using KIs (latency was roughly equivalent). Here’s how both paths went, side by side: 

Run the full AI Index pipeline in Serverless

This walkthrough offered a deep dive into the build-it-yourself version of AI indices and KIs. In production, you wouldn't hand-write these workflows; a setup agent would generate them, and a feedback loop would refine KIs from the agent's own traces. But the primitives are exactly what you just used: extract KIs with a workflow, store them in an AI Index, and retrieve them with a skill.

Managing context is key to a relevant and efficient agentic search system, and AI indices are a way to manage this context with the full power of the Elastic stack. Try it out in Serverless and let us know what you think in our Discuss forums or the #stack-kibana channel in our Community Slack! 

We wrote about the architecture behind consolidating those shapes in Why Elasticsearch is becoming a columnar database. This post is about the other side of the ledger. What does the split actually cost, and what does "search and analytics on one platform" mean in terms clear enough to hold up in a proof of concept?

What running five data engines actually costs

Anyone who has kept two sets of books for the same business knows where the hours go. Writing the second ledger is quick, but making the two agree is what takes the week.

Each engine needs its own ingest path, so the same events get parsed and shipped twice. That gives you two sets of failure modes and two backlogs to drain when a broker slows down. It also gives you drift: A field rename lands in one copy before the other, and for a while, the two systems disagree about the same hour of data. Reconciling that disagreement is real engineering work that rarely appears in the business case.

Each engine also brings a query language, and the syntax is the small part. The cost is everything written in that language, including dashboards, alert rules, saved queries, runbooks, and the operational knowledge of the person on call this week. Two languages means two of all of it, maintained in parallel.

Then there’s correlation. You find the failing request in the log aggregator. You move to the warehouse to chart how often it happened this week and then to the metrics store to check whether the host was saturated at the time. Each of those moves is a join performed by hand, by a person under time pressure, and every one of them adds minutes to the incident.

Retention compounds all of this. Each system gets its own lifecycle policy, so the cheap system ends up keeping data that the expensive one dropped weeks ago. When you finally need the history, it lives in the engine that cannot answer your question quickly.

Why search and analytics were split across two systems

The split was a reasonable response to a real constraint. Document engines and columnar engines were built to answer different questions. A document engine is good at finding the records that match a query and ranking them by relevance, and a columnar engine is good at reading three columns out of 50 and aggregating them across billions of rows. For years, running both was the reasonable answer, because no single system was credible at both jobs. The introduction of a full columnar engine in Elasticsearch 9.5 makes the split optional rather than necessary. For the full history and what changed, check out the columnar post.

What one platform means in practice

Three things have to be true before "one platform" means anything.

1. The first is one query language. Elasticsearch Query Language (ES|QL) runs across logs, metrics, traces, security events, and documents, which means a single skill set and one set of dashboards. Because it’s one language rather than a federation of several, queries compose: A time series aggregation can sit in the same query as LOOKUP JOIN or INLINE STATS, which systems built around PromQL alone cannot do.

2. The second is one storage substrate. Doc values, the column store that has been inside Elasticsearch since 2013, is what every index mode reads and writes underneath. Various modes tune that substrate for different shapes of data. Elasticsearch's time series engine (TSDB) went fully columnar in 9.4, which is the clearest evidence so far that the approach works. Columnar Mode and Columnar Logs, both in technical preview in Elasticsearch 9.5, extend the same treatment to analytical and log-shaped data.

3. The third is one operational story. It’s the same cluster, with the same APIs, integrations, agents, access control, backup, and upgrade path that you already run.

Several index modes share one column store, and one language queries all of them inside a single cluster. That’s a narrower claim than "one engine for everything," and it’s the one that holds up when someone tests it.

When is a specialized database still the right choice?

A specialized system will always win a specialized benchmark. If your workload is one shape with a single query pattern, there’s a purpose-built engine that beats us on it, and we would rather say so than pretend otherwise. The argument for consolidation applies to data that spans more than one shape, which describes most production estates.

That said, being the general-purpose platform doesn’t mean settling for second place on every shape. We’ve climbed this curve once already. TSDB has stored metrics since Elasticsearch 8.7, and the early work concentrated on storage efficiency rather than on competing with dedicated metrics stores.

A run of releases from Elasticsearch 9.1 through 9.4 then turned it into a columnar metrics engine: OpenTelemetry (OTel) metrics now land at 3.75 bytes per data point, down from 25 a year earlier, which is 2.5x less storage than Prometheus and 2x less than ClickHouse. Gauge average and counter rate queries run up to 30x faster than Prometheus and Mimir, and on the high cardinality benchmark, Elasticsearch scans four hours of data across half a million time series in under two seconds, where the other systems needed more than 30 seconds. The full methodology and per-query results are in how we rebuilt Elasticsearch as a leading columnar metrics datastore.

Columnar Mode is at the start of that same curve. Technical preview lands in Elasticsearch 9.5 and general availability (GA) in 9.6, with phased improvements to storage, ingest, and query performance in the releases that follow. Metrics took a year of that work to get where they are, and we expect logs and analytical data to follow the same path rather than a shorter one.

Which workloads should stay on the document modes

Some workloads should stay exactly where they are. 

1. Search-first applications, like product catalogs and knowledge bases, where the answer is the 10 most relevant documents, are what the existing document modes do well, and none of those modes are deprecated. 

2. Frequent individual document updates - these workloads suit the document modes, too. 

3. Nested data models - The same is true of data models that genuinely depend on nested structure, because Columnar Mode flattens fields into key/value pairs and doesn’t support the nested field type. In Elasticsearch 9.5, semantic_text and dense_vector fields aren’t available in Columnar Mode; a columnar profile for vector retrieval comes later.

Adoption is per index and opt-in. Existing indices continue to behave as they do today, and your APIs don’t change. Plus, your dashboards don’t break.

Where Columnar Mode is today: Elasticsearch 9.5 preview, 9.6 GA

Storage and performance numbers are coming in a separate technical deep dive, and the public roadmap issues for Columnar Mode and Columnar Logs are the right place to tell us what your workload needs.

If you want a real number for your own five-tool tax, start by counting two things: ingest pipelines carrying the same events to more than one destination, and dashboards that answer the same question in two different query languages. In most estates, the second number is the one that surprises people.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Static ID lists vs. dynamic filtering with ES|QL's WHERE clause

Static ID list filtering

Dynamic filtering with WHERE IN subquery

Run query A to find the IDs you care about

Outer query asks the main question

Copy the values by hand

Subquery builds the filter list from live data

Paste them into a static WHERE IN list

WHERE field IN (subquery) applies the filter live

Run query B with the hardcoded list

Results stay current with the data

Repeat the whole process when the data changes

No copied list to maintain

Figure 1. The old copy-paste loop collapses into a single dynamic filter.

How the WHERE IN subquery replaces static filter lists

Traditional IN filtering is still useful when the list is small and static:

But many real investigations don’t start with a tidy list. They start with a question: Which users are suspicious, which hosts are noisy, which services are failing, or which accounts crossed a threshold?

That’s where the WHERE IN subquery becomes useful. The list is produced by ES|QL rather than typed by hand.

WHERE IN subquery example: filtering logs by suspicious users

Read it like this: Show me log events for users who appear in the list of users with at least 10 failed login attempts.

The outer query asks the main question, and the subquery builds the dynamic filter list, eliminating the need to copy and paste.

The subquery can target a different index or index pattern from the outer query.

Filtering without subqueries: the manual ID copy workflow

Imagine that you want to inspect traffic for the top failing services. First you run:

Then you copy the five service names and paste them into another query:

That’s fine once, but less fine when the top five change every hour.

Dynamic filtering with a WHERE IN subquery

The subquery finds the top failing services from the last two days, and the outer query returns the log events for those services. One query builds the full picture.

Excluding values with NOT IN subqueries in ES|QL

Sometimes the interesting question is about what doesn’t belong:

That pattern is useful for exclusion checks, gap analysis, and workflows that ask for the things outside an approved or expected set.

Nested subquery chains in ES|QL's WHERE clause

An IN subquery replaces the literal value list with a query in parentheses. The inner query runs first and returns a single column, and the outer WHERE filters against it. Because that inner query is a full pipeline, it can contain its own IN subquery, which lets you express a chain of lookups that would otherwise require three separate queries and two rounds of copy-paste.

Read it inside out. The innermost query finds priority regions, and the middle query finds customers in those regions, while the outer query sums revenue for those customers. Each layer is a normal ES|QL pipeline, so each one can filter, aggregate, or sort on its own before handing a clean column up to the layer above.

Combining WHERE IN subqueries with AND and OR conditions

Because an IN subquery is a Boolean condition, it composes with AND and OR like any other predicate. You can require membership in two independent sets or accept membership in either:

The AND combination finds orders placed by VIP customers for products that are being discontinued. Swap AND for OR, and you get orders that match either condition. Each subquery runs its own pipeline, so the two sets are computed independently and then combined by the Boolean operator.

Merging multiple indices into one WHERE IN subquery

The query inside an IN subquery is a full pipeline, so its FROM command can reference more than one subquery. Each branch runs its own pipeline, and the FROM command merges the rows from all branches into one result set. KEEP host_id selects the single column that the outer filter needs. This is useful when the values you want to filter against live in several indices with different schemas. For more details on how subqueries in the FROM command handle indices with different schemas, see Three indices walk into a FROM clause: ES|QL subqueries in Elasticsearch.

The IN subquery combines matching host IDs from three indices, prod, staging, and edge, into one value list. The outer query then counts alerts for any host in that combined set. Adding a fourth source means adding one more branch, with no change to the outer query.

Using an Elasticsearch subquery inside each FROM branch

A FROM subquery gives each index its own branch with its own WHERE, and that WHERE can use an IN subquery. This is how you apply the same dynamic filter across several indices that each have their own schema.

Each branch filters its index down to VIP customers before the two branches combine, so the final aggregation runs over a single normalized set of rows.

When to use ES|QL WHERE IN subqueries

• Investigations that start by finding risky users, hosts, accounts, or services.

• Operational dashboards where the interesting entities change over time.

• Top-N follow-up queries, such as events for the five noisiest services.

• Set comparison workflows, especially with NOT IN.

• Queries that would otherwise need glue code just to pass values from one step to the next.

Requirements and constraints for WHERE IN subqueries

• Return exactly one column from the IN subquery.

• Use KEEP at the end of the subquery so the comparison field is obvious.

• Make sure the outer field and the subquery field have compatible types.

• If the subquery uses SORT, add an explicit LIMIT, as unbounded SORT isn’t supported in ES|QL yet.

• Use this for membership filtering. If you need columns from both sides, a JOIN` may be the better tool.

Why ES|QL dynamic filtering replaces manual query workflows

The WHERE IN subquery turns a manual workflow into a declarative one. You can let one query build the filter for another query directly inside the WHERE command, instead of asking ES|QL for a list, copying it somewhere else, and hoping it stays fresh. 

Your WHERE clause now has a better way to handle Filter this by whatever that query finds.

If you've been exploring ES|QL, you might have noticed that a subquery used to feel like it had exactly one front door: FROM. And it made sense. Most of the time, queries start with a simple directive to go fetch documents from an index. In an earlier post, we taught the WHERE command a new trick: IN subqueries. And before that, subqueries showed up in the FROM command to combine data sources. In both instances, every subquery started the same way, with FROM.

But not every useful query starts with a bulk document fetch. Sometimes you need to evaluate time-series semantics. Other times, you just need to whip up a tiny inline row for testing. Sometimes, the absolute best input to a filter is a dynamic query that builds the list for you on the fly, rather than a static, hard-coded list.

Source command

Reads from

Best for

Requirements

FROM

Indexes, data streams, aliases, views

Stored document lookups, live filter lists

None (works with any index)

TS

Time-series data streams

Metric aggregations with counter-reset handling

TSDS with index.mode: time_series

ROW

Inline literal values

Test cases, seed values, synthetic placeholders

None (no index needed)

The FROM, TS and ROW source commands are generally available (GA) in Elasticsearch 9.5, while the WHERE IN subquery remains in technical preview in 9.5.

How ES|QL subquery source commands work

Think of subqueries as having two specific placements and three different engines.

Where subqueries go: FROM and WHERE placements

• Inside the FROM command: The subquery is an independent result source, contributing its rows to the outer query. Fields that exist in one source but not the other are gracefully filled with null values.

• Inside the WHERE command: The IN subquery contributes dynamic values to be used as a predicate or filter.

Three source commands for starting a subquery

Door #1: FROM (the classic door)

FROM is the familiar workhorse. You use it when the subquery should read stored data from indices, data streams, aliases, or views. One of its best use cases is the "stop-copy-pasting-IDs" pattern. Instead of running one query, manually copying the output values, and pasting them into the filter of another query, the subquery becomes your live filter list.

Door #2: ROW (the tiny door)

ROW is the lightweight option that requires absolutely no index setup. It allows you to build rows completely out of literal, inline values. This makes ROW useful for small seed values, test cases, allow/deny lists, or one-off "what if?" scenarios. In the query below, ROW is the perfect way to staple a synthetic sentinel row or placeholder directly onto real data.

Door #3: TS (the time-series door)

The TS command targets time-series data streams and enables time-series aggregation functions. Why not just use FROM for metrics? TS is uniquely optimized for time-series data, and it natively handles tricky scenarios, like counter resets on process restarts and uneven metric publish intervals. Using TS as a subquery lets you filter your application logs by what your metrics are saying. For example, imagine asking ES|QL: Show me log events only from clusters whose peak metric throughput crossed 800:

In this scenario, the filter is reacting to live metric behavior rather than being hard-coded.

Combining FROM, TS and ROW in one query

Because subquery placements and source commands are independent, you can freely mix and match them. You can throw all three doors into a single FROM union to generate a cohesive table containing real logs, a live metrics summary, and a synthetic row.

ES|QL subquery constraints

A few constraints to keep in mind before using subqueries:

• IN subqueries demand one column: If a subquery feeds an IN operator, it must project exactly one column. Use the KEEP command to make that explicitly clear.

• TS requires a time series data stream (TSDS): The TS command only works on data stored in a TSDS, which uses index.mode: time_series.

The takeaway: when to use each source command

Subqueries in ES|QL provide a structured way to compose queries by using the results of one query as the input to another. Choosing the appropriate source command (FROM, ROW, or TS) lets you combine data and generate inline values. It also lets you filter dynamically without duplicating query logic. For more details and additional examples, see the ES|QL subqueries documentation.

• Subquery in FROM command

• Subquery in WHERE command

What you'll discover

In this post, you'll learn how to:

• Add conversion tracking (add-to-cart and purchase spans) with search.* attributes that tie back to the originating search.

• Build a full search-to-revenue funnel: search → click → add-to-cart → purchase.

• Write Elasticsearch Query Language (ES|QL) queries to calculate conversion rates, revenue per query, and average order value.

• Identify where users drop off and which team should own each drop-off point.

• Attribute revenue to specific search queries for prioritizing relevance work.

What you'll need

• Click tracking from Blog 3 (search + click spans flowing to Elastic via OTel-native ingestion).

• Backend endpoints for add-to-cart and checkout events (example code provided).

• Basic understanding of ecommerce conversion funnels.

Why search revenue attribution matters

Your product manager walks into a meeting and asks, "Which searches are driving the most revenue?"

You can tell them what users search for (Blog 2) and what they click (Blog 3). But you can't tell them what they buy. The gap between "clicked a result" and "purchased a product" is where the business case for search investment lives, and right now it's invisible.

This post closes the loop. By adding two more span types (add-to-cart and purchase), you get a full funnel from search to revenue, built on the same search.* attributes and ES|QL queries you've been using since Blog 2.

What search conversion tracking answers for your team

Here's what conversion tracking lets you answer and why each question matters to different people on your team.

• Which searches drive revenue? This is the product manager's question. When you can attribute dollars to specific queries, you can prioritize relevance work by business impact. A query with mediocre click-through rate (CTR) but high conversion value is more important than a high-CTR query that never leads to a purchase.

• Where do users drop off? The funnel from search to purchase has four stages: search, click, add-to-cart, and purchase. Each drop-off points to a different problem. High click-to-cart drop-off suggests that product pages aren't convincing. High cart-to-purchase drop-off is checkout friction rather than a search problem. Knowing where users abandon tells you which team should fix it.

• Which queries to protect? Once you know that "laptop bag" generates $12,000/month in attributed revenue, you treat it differently. Any relevance change that touches high-revenue queries gets extra scrutiny. You can set up monitoring (Blog 6, coming soon) to alert when conversion rates drop for your top-earning searches.

Two new instrumentation points total about 20 lines of code, and they follow the same pattern as you’ve used before. You add attributes to spans, and query them with ES|QL.

Following along with code? The reference project has conversion tracking ready to enable. Uncomment the Blog 4 sections in app.py and app.js, restart, and then generate traffic with python generate_traffic.py --blog 4.

The four-stage search conversion funnel

Before we write any code, here's the shape of what we're building. Each stage is an instrumentation point, and each creates spans in traces-generic.otel-default:

The thread running through the entire chain is search.query_id. The same identifier you derived from the trace ID in Blog 2 and used to link clicks to searches in Blog 3 now carries through to add-to-cart and purchase events. This is what makes revenue attribution possible: You can trace a purchase back to the search that started the journey.

Add conversion tracking spans with OpenTelemetry

Add-to-cart span instrumentation

When a user adds a product to their cart from a search results page (or from a product detail page they reached via search), you create a cart.add span. This captures the moment that intent turns into action.

This follows the same pattern as click tracking in Blog 3; that is, an independent span linked to the originating search via query_id. The new attributes are cart.quantity and cart.price which let you aggregate revenue at query time.

Purchase span instrumentation

When the user completes checkout, you create a checkout.complete span. This is the revenue event, the one that answers the product manager's question.

• Spans capture errors automatically. This is a side benefit of using OTel spans for conversion events. If an add-to-cart or checkout call throws an unhandled exception, the span's status is automatically set to ERROR and the exception details are recorded. These errors are business-critical (a broken checkout flow means lost revenue), and they show up immediately in Elastic APM's error tracking, service maps, and alerting. You get conversion analytics and operational monitoring from the same instrumentation, with no extra code.

• checkout.total_amount is the revenue number. This is what you'll aggregate in ES|QL to get revenue-by-query. It represents the order total, not the price of a single item.

• search.query_id is conditional. Not every purchase originates from search. Users browse categories and follow promotional links. Then they return to their cart days later. The if event.query_id: guard ensures that you only attribute purchases to search when there's a genuine connection. Purchases without a query_id still get recorded; they just don't appear in search attribution queries.

• search.query is set on both cart and purchase spans. This is a deliberate denormalization. You could join back to searches via query_id to get the query text, but while ES|QL does support LOOKUP JOIN, it’s likely not the right choice here due to the need to optimize the lookup index. By putting the query text directly on conversion spans, your revenue-by-query and cart-by-query queries are single, straightforward aggregations.

Span attributes for cart and purchase events

Add-to-cart attributes:

Attribute

Type

Required

Purpose

search.action

string

yes

"add_to_cart"

search.result_click_id

string

yes

Product document ID

search.result_click_position

int

yes

Position in results when added

search.query_id

string

yes

Links to originating search

enduser.pseudo.id

string

yes

Client/device identifier (OTel Semantic Conventions  [SemConv])

cart.quantity

int

yes

Quantity added

cart.price

float

optional

Product price

search.query

string

recommended

The search query text (for cart-by-query analysis)

Purchase attributes:

Attribute

Type

Required

Purpose

search.action

string

yes

"purchase"

checkout.order_id

string

yes

Unique order identifier

checkout.total_amount

float

yes

Order total

checkout.item_count

int

yes

Number of items purchased

enduser.pseudo.id

string

yes

Client/device identifier (OTel SemConv)

search.query_id

string

recommended

Links to originating search (last search in journey)

search.query

string

recommended

The search query text (for revenue-by-query analysis)

These follow the same search.* namespace from Blogs 2 and 3, with new cart.* and checkout.* prefixes for conversion-specific data. With OTel-native ingestion, all attributes are stored under attributes.* with dot notation preserved. That means no more mapping strings to labels.* and numbers to numeric_labels.*. A search.query attribute is queryable as attributes.search.query. Likewise, checkout.total_amount is queryable as attributes.checkout.total_amount.

User identity: connecting search to purchase across sessions

You'll notice that enduser.pseudo.id appears on every span type in this series. It's the minimum identity level; that is, a persistent identifier stored in the browser's localStorage that ties events to a device across sessions.

For conversion tracking, identity becomes more important. You need to connect a search on Monday to a purchase on Tuesday, or correlate cart additions across tabs. Our schema supports three identity levels, aligned with OTel semantic conventions:

Attribute

Persistence

Purpose

enduser.pseudo.id

Permanent (localStorage)

Device/browser identifier (OTel SemConv)

session.id

Per-visit (sessionStorage)

Groups events within a single visit (OTel SemConv)

user.id

Account (auth system)

Authenticated user (OTel SemConv)

For the funnel queries in this post, enduser.pseudo.id is sufficient. It links the journey from search to purchase within a browser. If your users authenticate, adding user.id enables cross-device attribution (searched on mobile, purchased on desktop) and richer personalization. session.id helps disambiguate when the same client has multiple active sessions.

All three are optional on interaction spans. Start with enduser.pseudo.id, and add the others when your use case requires them. The important thing is consistency: Use the same identifiers across search, click, and conversion spans so the joins work.

Frontend integration

The front end needs to propagate query_id through the user journey. When the user clicks a search result, you already have query_id from the search response (Blog 3). The key is carrying it forward.

The front end sends client_id as an HTTP field name; and the back end maps it to the OTel semantic convention enduser.pseudo.id when setting span attributes. The query_id propagation is the critical piece. Store it alongside the product in the cart data structure so it survives navigation between pages. We'll discuss the design challenges of this in the attribution section below.

Using the reference project? The reference app's frontend/app.js wires up add-to-cart buttons, but the checkout flow isn’t implemented in the browser UI. It goes through the traffic generator. To simulate conversion events, run: python generate_traffic.py --blog 4 --sessions 100. This sends a realistic mix of searches, clicks, cart additions, and purchases to all three backend endpoints.

Verify that conversion events are arriving

Before building funnel queries, confirm that both span types are flowing to Elastic:

Add-to-cart events:

Purchase events:

If these return rows, you have the full funnel instrumented. If not, check the same things as always: OpenTelemetry Protocol (OTLP) endpoint, auth token, and span export.

Funnel analysis with ES|QL

Count search, click, cart and purchase events in a single query

You can count all four funnel stages in a single ES|QL query using the same COUNT(CASE(...))pattern from the CTR query in Blog 3:

Example output:

searches

clicked

carts

purchases

click_rate

cart_rate

purchase_rate

146

41

28

12

28.1%

19.2%

8.2%

You get all four counts and three conversion rates from a single query. The WHERE clause pulls all four span types into one result set, and COUNT(CASE(...)) counts each type separately. This is the same technique that made the CTR query in Blog 3 so clean.

Notice that we use search.first_clickfor the click stage rather than counting all click events. This gives you the number of searches that received at least one click (the same definition used for CTR in Blog 3). Without this, a search with three clicks would inflate the click count to three while only counting as one search, making the funnel numbers misleading. Each stage now represents a unique progression: how many searches happened, how many of those got clicked, how many led to a cart addition, and how many resulted in a purchase.

You can turn this into a funnel visualization using Kibana Lens. Run each stage count as a separate ES|QL query, save them as dashboard panels, and arrange them as a horizontal bar chart with the four stages on the y-axis and counts on the x-axis. The drop-off at each step becomes immediately visible.

The dashboard above (from the first blog in the series) shows this in practice: The Conversion Funnel panel in the lower left uses a horizontal bar chart to visualize the four stages, and the Top Queries by Revenue table alongside it shows revenue attribution. You can build these directly from the ES|QL queries in this post.

What drop-off rates tell you and who owns each bottleneck

Each transition in the funnel tells you something specific:

• Search to click (CTR): You measured this in Blog 3. Low CTR means that results aren't compelling. This is a relevance problem.

• Click to cart: The user engaged with a result but didn't add it to their cart. This could mean that the product page isn't persuasive or the price isn't competitive. It could also mean that the item was out of stock. It's often not a search problem. It’s possible that the search worked (the user clicked), but something downstream lost them.

• Cart to purchase: The user committed to buying but didn't complete checkout. Complicated forms, unexpected shipping costs, and payment issues cause this checkout friction. This is almost never a search problem, but it's useful to know where the funnel leaks so you don't waste time optimizing relevance when checkout is the bottleneck.

The diagnostic pattern is straightforward:

Drop-off point

Likely cause

Who owns it

Search to click

Relevance / ranking

Search team

Click to cart

Product page / pricing / availability

Product / merchandising

Cart to purchase

Checkout UX / payment / shipping

Checkout / growth

This is one of the most valuable things that full-funnel data gives you: the ability to point at the right problem. When the VP asks, "Why aren't searches converting?", you can show whether the bottleneck is relevance, product pages, or checkout.

Revenue attribution by search query

Here's the query your product manager actually wants:

This gives you a ranked list of queries by the revenue they generated. The search.query attribute on purchase spans (the denormalization we discussed earlier) makes this a single aggregation query, without joins or subqueries.

How to act on search revenue  data

• Protect high-revenue queries. If "laptop bag" generates the most revenue, any relevance change that affects that query gets extra scrutiny. You might add it to a regression test suite or pin specific results with query rules. You might even set up an alert when its conversion rate drops (Blog 6).

• Prioritize relevance investment. The queries at the top of this list are where relevance improvements have the most business impact. A 10% CTR improvement on a query that generates $500/month in revenue is worth more than a 50% improvement on one that generates $20.

• Identify missed opportunities. Cross-reference with the top-queries data from Blog 2. A query with high search volume but no purchase attribution is either a browsing query (informational intent) or a conversion gap worth investigating.

Top revenue queries: where to focus relevance investment

To focus your relevance team's efforts, pull the top revenue-generating queries:

Cross-reference this with the per-query CTR from Blog 3. A query with high revenue but low CTR is underperforming; even small relevance improvements have outsized business impact. A query with high CTR but no purchase attribution might be informational (such as users browsing but not buying). The queries at the top of both lists deserve the most attention from your relevance team.

Average order value by search query

Average order value (AOV) tells you how much purchases are worth, in addition to which queries convert to those purchases. Queries with high AOV are your premium-intent searches; ranking improvements there have the biggest per-purchase impact:

A query with high AOV but low volume is a different opportunity than high volume + low AOV. The first means premium intent from a small audience (consider featured results or dedicated landing pages); the second means broad reach with budget buyers (price sensitivity may be limiting conversion more than relevance).

Search revenue attribution: limitations and workarounds

Revenue attribution from search is valuable, but it's imperfect. Understanding the limitations helps you set appropriate expectations and design around them.

Last-touch attribution in multi-search journeys

Users rarely search once and buy. A typical journey might look like:

1. Search "laptop bag": Browse results and click a few.

2. Search "laptop bag leather": Refine the search.

3. Search "laptop sleeve 15 inch": Try a different angle.

4. Add to cart from the third search's results.

5. Purchase.

With the instrumentation above, this purchase attributes to the third search, the one whose query_id was on the cart item. The first two searches contributed to the journey but get no credit.

This is last-touch attribution, and it's the simplest model that works within a single query_id linkage. It's not perfect, but it's concrete and unambiguous. The alternative, that is,tracking every query_id in a user's session and distributing credit, adds significant complexity to both instrumentation and analysis.

For most teams, last-touch is a good starting point. If you need multi-touch attribution later, the raw data is there. You can query all searches and clicks for a given client_id within a time window and reconstruct the full journey:

This reconstructs a user's full search journey in chronological order. It's useful for debugging individual sessions, even if you don't build automated multi-touch attribution.

Cross-session attribution limits with query_id

A user searches for "wireless headphones" on Monday, clicks a few results, leaves, and comes back on Wednesday to buy. The query_id from Monday's search is long gone, since it was a property of that specific search request.

This is a fundamental limitation of query_id-based attribution. It works within a session (or more precisely, within the scope where the front end retains the query_id). It doesn't work across sessions.

For cross-session attribution, you'd need a different approach, typically a user-level event store where you associate product views, cart additions, and purchases with a persistent user ID and then look back in time to find the originating search. That's a more complex analytics pipeline and is outside the scope of what we're building here.

The practical impact is that your search-attributed revenue will be an undercount. Some purchases that were genuinely influenced by search won't carry a query_id. This is fine for relative comparisons (such as, Which queries generate more revenue than others?), even if the absolute numbers are conservative.

How to persist query_id from search to checkout

A few practical decisions affect how far your query_id propagation reaches:

• Store query_id in the cart. When a user adds a product to their cart, persist the query_id alongside the item. This way, even if the user navigates away and comes back to checkout later (within the same session), the attribution survives.

• Don't overwrite query_id on re-search. If a user adds a product from search A and then searches again and adds another product from search B, each cart item should keep its own query_id. The purchase event carries the last search's query_id as a summary, but per-item attribution gives you richer data.

• Accept the limitations. Not every purchase will have search attribution. Direct navigation, category browsing, promotional links, and returning customers who go straight to their cart will all produce purchases without a query_id. That's correct behavior, not missing data.

Spans vs. log events for conversion tracking

Conversion events can emit both an OTel span and a UBI-compatible log event, the same dual-signal pattern used for click tracking in Blog 3. The log event is actually richer for conversions. It can contain the full items list with per-item query_id attribution, which doesn't map cleanly to flat span attributes.

The span gives you the simple, aggregatable view (total revenue by query), and the log gives you the detailed, per-item view (which specific products from which specific searches). For the funnel queries in this post, spans are sufficient. If you need item-level attribution analysis, the log events in logs-generic.otel-default have the detail you need, and ES|QL queries them the same way, just against a different index pattern.

What's next: turning conversion data into relevance improvements

We now have the complete instrumentation picture, with four span types: search (Blog 2), search.result.click (Blog 3), cart.add, and checkout.complete (this post). These capture the full user journey from query to purchase. Every span lives in traces-generic.otel-default, and every metric is queryable with ES|QL. The search.query_id thread ties the entire funnel together.

But measuring the funnel is only half the story. The real payoff is using this data to make search better.

In a later blog in this series, we take everything we've built and turn it into relevance improvements. Click positions and conversion data become judgment lists for Learning To Rank, and per-query CTR and revenue become rank features for boosting. Plus, high-revenue queries get protective monitoring. And tools like Elasticsearch Relevance Studio give you a visual interface for tuning the searches that matter most, using exactly the data you're now collecting.

The instrumentation you've built in Blogs 2–4 is a feedback loop, beyond analytics: Measure, improve, and measure again.

Get started with search conversion tracking

• Reference project: Working code for the entire blog series; clone, configure, and run.

• Elastic Distribution of OpenTelemetry for Python: EDOT for Python.

• OpenTelemetry with Elastic: How to send OTel data to Elastic APM.

• ES|QL documentation: Query language reference.

• UBI Standard: Reference schema for search event structure.

• Query rules: Pin, boost, or exclude results for specific queries.

How MATCH and TO_TEXT enable full-text search on any ES|QL expression

Let's start with a query that was impossible in Elasticsearch 9.4, which uses the EVAL command:

In this example, summary has no mapping or analyzer configuration. It’s also not associated with any inverted index. It exists only for the lifetime of this query, but now you can search it anyway. Two additions make this work.

First, MATCH now accepts any expression as its first argument, not just a mapped field. That includes columns produced by EVAL and function results used inline. It also includes unmapped fields loaded directly from the original document. Furthermore, all data types normally accepted by MATCH are supported in this new use case.

The second part of this is the new TO_TEXT function, which is the first ES|QL conversion function that produces output of type text. Until now, text columns could only come from indexed mapped fields, and all strings produced by ES|QL expressions were keyword values rather than text. The distinction matters because MATCH treats the two differently: text values are analyzed, while keyword values are compared exactly, mirroring how a MATCH query on an indexed keyword field rewrites to a term query. TO_TEXT(x) is how you tell ES|QL: treat this string as full text.

This ships as a technical preview in Elasticsearch 9.5, and as such, it has some limitations:

• It’s currently filtering only. A MATCH on an expression doesn't contribute to the relevance score yet; only matches on indexed fields affect the score.

• Querying options like fuzziness and others aren't yet supported when matching an expression.

• Runtime text is analyzed with the standard analyzer. This isn’t configurable yet.

Work is underway to address these limitations.

Why use full-text search instead of LIKE or RLIKE in ES|QL?

ES|QL already had two ways to search strings without an index: LIKE (wildcard patterns) and RLIKE (regular expressions). Both work on any string expression, so it's fair to ask what MATCH adds. The answer is analysis, a more advanced form of search which uses techniques such as stemming and synonyms. It also uses stopword handling.

LIKE is simple substring matching, without any understanding of the words that comprise a string. Say, for example, you're looking for log messages about a fox:

This misses "Fox spotted near the henhouse" due to the capitalization, while matching "Outfoxed by the competition", which isn't about a fox at all. It fails in both directions, with false negatives on capitalization and false positives on substrings buried inside other words.

Regular expressions can patch the case problem, but the word-boundary problem gets ugly fast. Something like:

And even that's not right yet. It misses a fox at the end of a sentence followed by ! or ?, and it says nothing about tabs, quotes, or parentheses. Each fix makes the pattern longer, and the next person to read the query has to reverse-engineer what it’s actually doing.

MATCH makes the problem go away, because it runs both the query and the value through an analyzer, which tokenizes text into lowercase terms and then matches term against term:

This query will match values like "The quick brown fox" and "FOX spotted near the henhouse" but not "Outfoxed by the competition" or “FOXTROT protocol enabled", regardless of any punctuation surrounding the words. Of course, this all works for multi-term queries, like MATCH(TO_TEXT(message), "brown fox"), too, just the way you’d expect it to.

Work is underway to enable the use of the 36 dedicated language analyzers, with support for natural languages on data that was never indexed or mapped.

Full-text search use cases for unindexed and unmapped data

The examples above searched values computed from mapped fields. . The more interesting use cases for ES|QL MATCH on expressions involve data that was never searchable at all. Let's walk through a few.

How to search unmapped fields in ES|QL without adding a mapping

Sometimes you deliberately leave a field out of your mappings, such as a verbose stack trace or a raw request payload. You might even leave out a debug blob. Indexing one of these would cost disk and heap space on every document and wouldn’t be worth it for a field you might query once a quarter.

That decision has always been final, because unmapped fields were invisible to queries entirely. In Elasticsearch 9.5, you can use SET unmapped_fields="load" to make ES|QL load unmapped fields directly from the source document as keywords. Follow that up by wrapping it in TO_TEXT, and now you can run full-text search on it:

Here, stack_trace was never mapped. Every value is fetched from the original documents and analyzed on the fly. They’re matched row by row. That’s real work, and it will never be as fast as an inverted index lookup. But now, that field you didn't index is no longer unsearchable. You get to keep the mapping small for the everyday case and still answer the once-a-quarter question when it matters.

Full-text search on a keyword field without reindexing

Keyword fields can do a lot. They give you exact matching, fast aggregations, and sorting, which is why so many fields end up mapped that way. But mappings are decided when data arrives, and it’s easy to end up in a situation where you want to do something different with your data than you had originally intended. Maybe product_name was mapped as a keyword because the dashboards aggregate on it, and then, after receiving a year’s worth of product data, someone wants to be able to search within product_name values.

The old answer was to change the mapping to text (or add a multi-field) and reindex everything. This can be both time-consuming and costly, and in many cases, users simply won’t want to bother with it. The new answer is one function call:

TO_TEXT converts the keyword values to text on the fly, so MATCH analyzes them instead of comparing them exactly. This allows you to query a keyword field without creating a mapping or reindexing the source document. If the search becomes an everyday query, indexing the field as text is still the right long-term move, but TO_TEXT gets you an answer today, without any extra work.

Searching the same field across indices with different mappings

ES|QL can span many indices, and the same field doesn't need to always look the same in all of them. When the same field has different types in different indices, ES|QL treats it as a union type, and a conversion function resolves the conflict. Let’s consider an example in which the message field has type text in this year's index template but was a keyword in last year's:

Every value is analyzed at query time, whether it came from the text index or the keyword one. The keyword values from the older indices are tokenized and lowercased like everything else, so "connection reset" finds “Connection RESET by peer”, no matter which index it lives in.

Another interesting case is when a field is mapped in only one index but also present (and unmapped) in the other:

There's a nuance worth calling out here. If error_details is mapped in logs-2026 but not in logs-2025, Elasticsearch cannot push this query down to Lucene, because the indices where the field is unmapped would silently return no matches. Instead, the planner notices that the field is potentially unmapped and evaluates the whole MATCH row by row, wherever the rows came from. You don't have to know which of your indices have the field mapped; the query just answers the question.

How ES|QL analyzes text at query time without an inverted index

When ES|QL plans a MATCH against an expression, it analyzes the query string once, up front, into a set of terms. How each row is then evaluated depends on the expression's type:

Expression type

Processing

Matching behavior

text (via TO_TEXT)

Analyzer tokenizes value into lowercase terms

Token-against-token comparison; a row matches if any token equals any query term (OR semantics)

keyword,ip, date, numeric

No analysis; query constant converted once to the native type

Exact comparison per row

Both paths bypass Lucene entirely and evaluate values row by row. The non-text path mirrors exactly what a match query does when pushed down to Lucene against those field types, so the semantics stay consistent regardless of whether your query hits an index.

An inverted-index lookup does its work at ingest time and never touches non-matching documents at query time. A runtime MATCH does that analysis at query time, for every row that reaches it. One is fast because the work already happened; the other is flexible because the data doesn't need to have been indexed at all.

What's next for ES|QL full-text search

Everything in this post is the first installment of a larger effort to make search in ES|QL work on anything, not just on what you indexed ahead of time. The limitations called out earlier are actively being worked on, and the roadmap goes further:

• Scoring. Runtime matches will contribute to _score, so you can sort by relevance even when the data was never indexed.

• MATCH_PHRASE on expressions. Already available in Elastic Cloud Serverless, and coming to the Elastic Stack in 9.6.

• Configurable analyzers. Analyzer support for MATCH and MATCH_PHRASE on expressions, enabling language analyzers, stemming, and synonyms at query time.

• Match options. Options like fuzziness and operator for runtime matches.

• Vector search. Generating embeddings per row and running k-nearest neighbors (kNN) on runtime dense_vector expressions, bringing semantic search to unindexed data, too.

Try ES|QL full-text search on expressions today

You can try runtime search today. It's available now in Elastic Cloud Serverless, where new ES|QL capabilities land first, and it ships as a technical preview in Elasticsearch 9.5. Start with the search functions reference, and check the ES|QL limitations page for the current boundaries. It's a technical preview because we want your feedback: If you search something that was never indexed and it surprises you, either positively or negatively, we'd love to hear about it.

ES|QL charts performance and interactivity in Kibana dashboards

How ES|QL Fast mode runs dashboard queries up to 100x faster

For common analytical tasks, like trend tracking, top-host identification, and capacity overviews, trading a small margin of accuracy for dramatically faster results is the right call, especially since not every question needs an exact answer.

Elastic Search 9.4 introduced approximate ES|QL queries as a syntax-only command in technical preview. Now, 9.5 makes approximation GA and adds Fast mode, a UI toggle in Dashboards and Discover that enables approximate ES|QL STATS queries without writing any query syntax. This makes Kibana one of the first tools to offer smart sampling with automatic extrapolation as a simple switch.

Fast mode is an Enterprise-only feature and is off by default. Dashboard authors can save their preferred state with the dashboard, and individual queries can override the toggle with SET approximation=true or false inline.

When switched on, ES|QL STATS queries target a fixed sample size (defaulting to 1,000,000 rows for grouped aggregations and 100,000 rows otherwise) rather than scanning the full dataset. Benchmarks show heavy aggregations running up to 100x faster on large datasets, with results that are typically highlyaccurate, defaulting to a 90% confidence interval. Approximation only applies to STATS commands where results can remain accurate. When accuracy cannot be ensured (such as with small datasets or aggregations like MAX, MIN, or COUNT_DISTINCT), Kibana automatically falls back to exact execution, even with Fast mode enabled.

Further improvements to how charts communicate that results are approximate are coming in future releases.

Click-to-filter and Discover drilldowns for ES|QL charts

Two of the most popular interactions for data view charts are also landing now for ES|QL-based visualizations.

• Discover drilldowns now work on ES|QL panels. When a user clicks a data point or uses Explore in Discover, filters are translated to ES|QL WHERE clauses and Kibana Query Language (KQL) queries are carried over automatically. 

• Click-to-filter also works for renamed fields: if your query renames a column (STATS BY node = k8s.node.name), Kibana now resolves the alias back to the indexed field, so the filter applies correctly.

• Tooltips: When filtering genuinely can't work (for example, because the field was computed entirely within the query and doesn't exist in the index), Kibana now shows a tooltip explaining why, so users know that it's a query limitation.Beyond interactivity, ES|QL layers now have the same Use global filters toggle (gear icon on the layer header) as data-view-backed visualizations. When you turn it off, the layer's query runs independently of dashboard-level filters, just like form-based layers already do. This is useful for reference lines, thresholds, or baselines that shouldn't change when you filter the dashboard. And ES|QL metric charts now support a background chart, matching the styling option already available for data view metrics.

Beyond interactivity, ES|QL layers now have the same Use global filters toggle (gear icon on the layer header) as data-view-backed visualizations. When you turn it off, the layer's query runs independently of dashboard-level filters, just like form-based layers already do. This is useful for reference lines, thresholds, or baselines that shouldn't change when you filter the dashboard. And ES|QL metric charts now support a background chart, matching the styling option already available for data view metrics.

Upcoming releases aim to keep adding the remaining functionality to ES|QL visualizations, such as multilayer support and saving visualizations to the library.

Identify which Kibana panels use an ES|QL variable

Variable controls are among the most popular ES|QL-only features, since they let you parameterize chart queries to switch between fields, time intervals, or groupings without duplicating panels. On a dashboard with many panels and controls, though, it can be hard to tell which visualizations a variable actually affects. In edit mode, you can now click an ES|QL variable control's label to identify all related panels that consume the variable. Variables with no related panels display a warning to make it easier to audit wiring before saving.

Kibana metric and bar chart layout defaults

Metric chart preset layouts and density options

The metric chart appearance panel now offers preset layouts: Top, Middle, Bottom, and Custom. When you pick a template, the layout snaps into place. If you need fine-grained control, switch to Custom.

Metrics used to pack values tightly, which is great for data-dense dashboards but hard to scan when a metric stands alone. Elastic Cloud 9.5 adds a Density style option under Style > Details > Other, with two presets: Compact (the previous layout) and Default (more padding, larger typography). Newly created metrics use Default, and existing saved charts keep Compact until you change them.

Attribute

Compact

Default

Padding

Tight, minimal spacing

More generous whitespace

Typography

Smaller text

Larger text

Best for

Data-dense dashboards with many metrics side by side

Standalone metrics or dashboards with fewer panels

New charts

Must be selected manually

Applied automatically

Existing charts

Preserved until changed

Must be selected manually

Responsive bar chart labels in Kibana

Labels in horizontal bar charts used to grow unchecked, so on smaller screens, a chart with long category names could become unreadable. Bar labels now get a max width and middle-truncate automatically, so the beginning and end of a label stay visible even when the full text doesn't fit. This works by default, with no configuration needed. In 9.6, we’re planning many more improvements to bar charts and labels.

Kibana dashboard controls populated by ES|QL queries

Controls are the most user-friendly way to filter a dashboard, and most dashboards use them. One of the longest-standing requests from users has been the ability to prefilter the values that a control shows. ES|QL queries make that possible and open a much wider set of possibilities, like chaining controls in new ways using variables. Regardless of how the values are populated, controls filter every panel on the dashboard, including ES|QL and data view visualizations.

Controls can now be populated from an ES|QL query instead of selecting a data view field directly. The Create control flyout adds a Select a field / Write a query toggle. You can write an ES|QL query that returns a single column and run it, and then the control derives its options from the result. Queries can reference dashboard variables through the ?variable syntax, enabling flexible chaining between controls.

Coming soon: Progress bar visualization for Kibana tables

A new progress bar visualization type is available in Elastic Cloud Serverless and is planned for general availability in 9.6. Progress bars show a value relative to a goal or maximum, which is useful for many O11y metrics, like CPUs, memory, or Service Level Agreement (SLA) tracking.

What's next for Kibana dashboards and ES|QL visualizations

Upcoming releases will keep pushing on better defaults, improving the ES|QL visualization experience, and adding new chart types. If you have a pain point or a feature request, select Submit feedback in the top menu; we're listening.

How to try ES|QL Fast mode and the new Kibana dashboard features

If you use Elastic Cloud Serverless, you may already be using these changes. Otherwise, upgrade to 9.5, and then create a dashboard or open an existing one. Many updates apply automatically to new visualizations, while layout and style options appear in edit mode. If you aren't on Elastic Cloud yet, start a trial and explore the latest Kibana dashboards there.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

What's new in Kibana's AI dashboard creation

Capability

Technical preview (9.4)

GA (9.5)

Error handling

No retry on failed ES|QL queries

Automatic retry up to three times with query inspection and adjustment

ES|QL generation cost

All queries routed through the primary model

Tiered model routing, up to 5x cheaper

Time range

Fixed default window

Automatic selection based on data time distribution

Filter controls

Not supported

Automatically added for the most relevant fields

Vega-Lite charts

Not supported

Natural-language creation, including scatter plots, box plots, conditional formatting, custom tooltips

Chart editing

Not supported

Edit existing Vega-Lite panels through natural language

Automatic error recovery for AI dashboard generation

In the technical preview, the agent didn’t retry failed ES|QL queries. In 9.5, it detects query errors and retries up to three times, inspecting each error and adjusting the query before giving up. In practice, this eliminates the majority of empty-panel issues and produces dashboards that render correctly on the first try.

Why is AI dashboard creation cheaper in Elastic 9.5?

Not every step in dashboard generation needs the same level of reasoning. In 9.5, ES|QL generation routes through a lighter model by default and falls back to the primary model only when needed. If your connector uses Anthropic's Claude Opus 4.8, that means 5x cheaper for ES|QL generation across all panels.

Automatic time range selection based on your data

Dashboards are only useful when they show the right window of data. The agent now applies improved logic to pick a time range that makes sense for the data it's querying, unless the user asks for a specific time range. It considers the data's time distribution and adjusts accordingly, whether that means the last hour for a live incident or the last 90 days for a trend analysis, rather than defaulting to a fixed window.

Automatic filter controls on AI-generated dashboards

Dashboard creation now supports controls; that is, interactive filters that let viewers narrow a dashboard by field values without editing the underlying queries. When generating a dashboard, the agent automatically adds controls at the top for the fields most relevant to filter by.

Vega-Lite charts from natural language: Chart types and formatting beyond the defaults

Vega and Vega-Lite support a wide range of chart types and customizations in Kibana. With 9.5, you can build them from plain language instead of writing the code yourself. 

Scatter plots, box plots, and more Vega-Lite chart types

Scatter plots, box plot charts, faceted small multiples, bubble charts, and composition charts (like combining histograms with heatmaps), among many others, are supported by Vega-Lite. A prompt like Show me a scatter plot of response time vs. request size, colored by service name produces a Vega-Lite panel with the right data mappings. They use Kibana's default color palettes to blend with the rest of the dashboard.

Conditional formatting, custom tooltips, and labels on standard charts

Even for chart types that are already native to dashboards, like bar, line, or area, sometimes you need more control than the default capabilities offer. Vega-Lite through the chat fills that gap. Some examples include:

• Conditional color formatting: Color data points above a threshold differently; for example, turning data points in a line or bars red when some metric spikes beyond your Service Level Objective (SLO). Ask the agent something like Turn any points above 500ms red for my line chart.

  

• Custom marks and labels: Add emojis, symbols, or inline text labels to data points for at-a-glance status indicators.

  

• Custom tooltips: Enrich hover states with additional metrics, context, or computed values that aren't part of the chart's axes. Ask something like Add a tooltip that shows total record counts and the % per bar.

  

This also works for editing existing Vega charts. If you have a Vega-Lite panel that needs a tweak like changing a color scale, adjusting an axis, or switching the mark type, describe the change in chat instead of digging into the JSON code.

How we built natural-language Vega-Lite generation in Kibana

Generating a Vega-Lite chart from a sentence is not a one-shot ask the model for JSON prompt. We built a small agentic pipeline that turns natural-language intent into a validated, data-backed chart.

When a request comes in, the agent first determines whether Vega-Lite is the right fit. For Vega-Lite requests, it grounds the visualization in a real ES|QL query against Elasticsearch and then uses a model to generate the Vega-Lite code. Before rendering, the result goes through a normalization layer that corrects the schema and binds the canonical query. It also applies render-safety transformations.

A few design choices make this workflow reliable:

• Typed tool calling: Chart creation is a structured tool invocation rather than free-form Vega-Lite pasted into the conversation.

• Constrained generation: The model generates Vega-Lite code within a defined schema, making the output more predictable and easier to validate.

• Curated examples: Structural patterns, such as faceting, layered marks, and heatmaps, provide guidance without copying the underlying data.

• Execute-and-verify loops: Queries are executed before chart authoring, and validation failures trigger corrective retries for ES|QL generation.

Try AI dashboard creation and Vega-Lite charts in Kibana

To try natural-language dashboard creation and Vega-Lite charts, upgrade to Elastic 9.5 (or start a free trial), and open the chat in Kibana. Then ask it to build a dashboard from your data. For Vega-Lite, try asking for a chart type you've wanted but never built, like a scatter plot or a bubble chart. If the result isn't quite right, tell the agent what to change. It iterates with you.

This requires an Enterprise license. Get started.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

What you'll discover

In this post, you'll learn how to:

• Add client-side click tracking that links clicks back to their originating search via search.query_id.

• Calculate CTR; that is, the percentage of searches that produce at least one click.

• Calculate MRR; that is, how far down the results users click on average.

• Analyze click position distribution to see the full shape of user engagement.

• Write ES|QL queries for all three metrics against traces-generic.otel-default.

• Identify which specific queries need relevance tuning.

What you'll need

• A working OTel instrumentation setup from Blog 2 (search spans with search.* attributes flowing to Elastic via OTel-native ingestion).

• A front end that can send click events (JavaScript example provided).

• Familiarity with the attributes.* field mapping from Blog 2.

Why search logs alone can't measure search quality

In the second blog of the series, we instrumented search requests and ran six ES|QL queries against the data. We can see what users search for, which queries return nothing, and how fast search is.

But there's a blind spot. A search that returns 15 results looks healthy from the server side. Every metric we have says it worked. But if nobody clicks any of those results, your ranking has a problem, and none of the queries from Blog 2 will tell you.

This is the gap between results returned and results that are useful. Search volume, zero-results rate, and latency measure the mechanics of search, but they don't measure whether search is actually helping users find what they need.

To answer that question, you need a second instrumentation point: click tracking.

If you’re following along with code, the reference project has click tracking ready to enable. Uncomment the Blog 3 sections in app.py and frontend/app.js, restart, and then generate traffic with python generate_traffic.py --blog 3.

How click data measures search relevance

Before we write any code, here's what click data lets you measure and why each metric matters:

• CTR answers the most basic engagement question: What percentage of searches result in at least one click? If your CTR is low, users are seeing results but not finding them compelling enough to engage. Establish your own baseline once you have data; CTR varies considerably across product category, query type, and industry vertical.

• MRR goes deeper: When users do click, where in the results are they clicking? An MRR of 1.0 means every user clicks the top result (a perfect ranking). An MRR of 0.5 means the average click is at position 2. Low MRR with high CTR is particularly telling. It means that users are finding what they need, but your ranking is making them work for it.

• Click position distribution shows the full shape of where users click. A healthy search engine shows most clicks at position 1 with a sharp drop-off. A flat distribution across positions 1–5 means that your ranking isn't differentiating well. Per-query distributions reveal exactly which searches need relevance tuning.

Together, these metrics move you from Did search work? (Blog 2) to Did search work well?, and they pinpoint exactly where to invest in relevance improvements. Later in the series, we'll show how to turn these metrics into concrete actions: building judgment lists for Learning To Rank (LTR), tuning relevance with tools like Elasticsearch Relevance Studio, and evaluating changes with the Rank Eval API. But first, you need the data.

All three metrics require just one new instrumentation point: about 15 lines of code.

Add click tracking

Click tracking captures what happens after the results appear. When a user clicks a search result, we create a new span with attributes describing the interaction, including which document they clicked, where it appeared in the results, and which search produced it.

Here's the code:

Let's unpack what matters.

Click spans are separate traces

This is the key architectural difference from Blog 2. Search spans are created synchronously during the API request: The user searches, the span opens, Elasticsearch responds, and the span closes. Click spans are asynchronous. The user searches, gets results, browses the page, and might click 30 seconds later (or they might never click).

That means click spans aren't children of the search span's trace. They're independent traces, linked to the originating search through search.query_id. This is the same query_id we derived from the trace ID in Blog 2, and it now serves as the join key between searches and clicks across traces-generic.otel-default.

Choosing the right OTel signal for clicks

OTel gives you three signal types, and clicks could be modeled as any of them. Each has strengths:

Signal

Index

Weight

Best for…

Spans

traces-generic.otel-default

Full trace context

Same-index queries with search spans

Logs

logs-*

Lighter weight

Log-centric pipelines, high volume

Span events

logs-generic.otel-default

Lightest instrumentation

Attaching to existing spans

• Spans land in traces-generic.otel-default alongside search spans, are fully queryable in ES|QL, appear in Kibana APM views, and carry timing information. Since our search spans are already in traces-generic.otel-default, using spans for clicks means you can query searches and clicks together in a single ES|QL statement, and no cross-index joins are needed.

• Log records are also independently queryable in ES|QL, living in logs-*. If you use the same search.* attribute names, the analytics queries are almost identical; just change the index pattern. Logs are lighter weight (no trace context overhead) and are a natural fit if your team already has a log-centric observability pipeline. One of Elastic's strengths here is that traces, logs, and metrics all land in the same platform and are all queryable with ES|QL, so choosing logs over spans doesn't mean giving up any query capability.

• Span events are lightweight at instrumentation time (attached to an existing span in the OTel API). In Elastic's OpenTelemetry Protocol (OTLP) ingestion pipeline, span events are written as separate documents to logs-* data streams (for example logs-generic.otel-default), and they’re independently queryable in the same way as logs. They’re a good, lightweight option but might require more code changes than logs, which can even pull in logs from legacy code.

In this series, we use spans because they keep searches and clicks in the same index with the simplest query path. But if you're at high volume and want to optimize for cost, or if your organization already routes OTel logs to Elasticsearch, the log-based approach works well; the search.* attribute schema is the same either way, and ES|QL queries against logs-* follow the same patterns you'll see below.

The search.first_click attribute

search.first_click is a Boolean set only on the first click for a given query_id. It exists for one reason: accurate CTR calculation without post-processing.

CTR is defined as the percentage of searches with at least one click. Without search.first_click, you'd need to deduplicate clicks by query_id at query time: grouping, counting distinct values, and subquerying. By marking the first click at instrumentation time, the ES|QL query becomes a simple count.

The set above is demo-only. It grows unbounded and breaks with multiple API replicas. The reference implementation uses a thread-safe time-to-live (TTL) dict with a 30-minute expiry window (_is_first_click() in app.py). For production with multiple replicas, use a shared external cache (Redis, Memcached) keyed by query_id with a TTL matching your session window.

Where to track first click: Front end vs. back end

The search.first_click deduplication could live in either the front end or the back end. Both are valid, and here are the trade-offs:

• Frontend tracking is simpler to implement. The browser already knows the current query and whether the user has clicked before. No server-side state is required, you don’t have to worry about multiple replicas, and it works without any backend changes. The downside is that browser state is ephemeral; a page refresh, multiple tabs, or an ad blocker can interfere with accurate tracking.

• Backend tracking (our approach) gives you a single source of truth. All click events flow through one place, so the deduplication is consistent regardless of what the client does. It also means that the analytics logic is colocated with the instrumentation code, which simplifies reasoning about data quality. The trade-off is that the back end needs to maintain state: the _clicked_queries set. For a single-instance API, this is trivial; for multiple replicas behind a load balancer, you'd use a shared TTL cache (Redis or similar).

We chose backend tracking here because we want the analytics data to be authoritative. This click data will later feed into relevance tuning and judgment lists, where accuracy matters. But if you're starting simple or running a client-side–only setup, frontend tracking is a perfectly reasonable first step. The search.first_click attribute works the same way regardless of where you set it.

Sending click events from the browser

The browser sends click events to the back end when a user clicks a result. It needs three things from the search response: the document ID, the position, and the query_id.

CLIENT_ID is generated once and stored in localStorage. It survives page reloads and gives you a stable enduser.pseudo.id without requiring a login. The back end maps client_id → enduser.pseudo.id on the span.

Positions are 1-indexed; that is, the first result is position 1, not 0.

Click tracking OTel attributes and ES|QL field mapping

Attribute

Type

Required

Purpose

search.action

string

yes

Event type: "click"

search.result_click_id

string

yes

Document ID clicked

search.result_click_position

int

yes

Position in results (1-indexed)

search.query_id

string

yes

Links to originating search

enduser.pseudo.id

string

yes

Client/device identifier

search.first_click

boolean

recommended

true if first click for this query_id

search.result_click_type

string

recommended

Object type: "product", "article"

search.query

string

recommended

The search query text (for queryability)

These follow the same search.* namespace we established in Blog 2. With OTel-native ingestion, attributes map directly to attributes.* fields in ES|QL:

OTel attribute

ES|QL field

search.action

attributes.search.action

search.result_click_id

attributes.search.result_click_id

search.result_click_position

attributes.search.result_click_position

search.query_id

attributes.search.query_id

search.first_click

attributes.search.first_click

With OTel-native ingestion, search.first_click is stored as a native Boolean; you query it with == true, not == "true", and no string coercion is needed.

Verify that clicks are arriving

Before calculating metrics, confirm that click spans are flowing to APM:

If this returns rows, you're ready for analytics. If not, check the same things as we looked at in Blog 2: OTLP endpoint, auth token, and span export.

Note: Results in this post are illustrative, generated by running python generate_traffic.py --blog 3 --sessions 50 on the reference project. Running Blog 3 traffic adds click events and additional search sessions on top of the 62 from Blog 2, so cumulative search counts will exceed 62. Your exact numbers will vary based on session count and the random nature of the traffic simulator. The metric calculations and ES|QL patterns are what to focus on.

CTR

CTR is the primary signal for search relevance, answering the question that Blog 2 couldn't: Are users engaging with the results?

CTR = searches with at least one click / total searches * 100

CTR is a binary per-search metric: Either a search got clicked or it didn't. The maximum is 100%.

Overall search CTR with ES|QL

Result: 41 clicked searches out of 146 total. CTR: 28.1%

This is a single query that pulls both search spans and first-click spans from traces-generic.otel-default. The OR in the WHERE clause brings both into one result set. COUNT(CASE(...)) counts each type separately, and EVAL does the division.

This works because of search.first_click. Without it, you'd be counting raw clicks (a user who clicks three results on one search would inflate the count). The deduplication happened at instrumentation time; the query stays simple.

CTR by search query: Finding your worst relevance failures

The overall number is useful for dashboards. The per-query breakdown is where you find problems.

This shows CTR broken down by query text. 

What CTR tells you (and what it doesn't)

• High searches + zero clicks: These are the worst relevance failures. Fix these first.

• High searches + low CTR: Results appear, but they aren't compelling. Check ranking.

• Low CTR + high zero-results rate: This is a double problem; either no results or bad results.

• CTR trend over time: This measures the impact of relevance changes.

CTR doesn't measure satisfaction. A user who clicks position 1, bounces back, and then clicks position 3 still counts as one clicked search. For a fuller picture, you need to know where they're clicking. That's what MRR measures.

CTR vs. clicks per search

CTR (what we just calculated) is capped at 100%. This is the industry-standard definition.

Clicks per search is total clicks divided by total searches. It can exceed 1.0. For example, a search where the user clicks three results scores 3.0. It measures engagement depth, which is useful but different. If you need it, count all click spans (not just first_click) divided by search spans.

MRR

MRR tells you where users click. It measures how far down the results list users go before finding something worth clicking.

MRR = average of (1 / click_position) across all clicks

The reciprocal rank transforms click positions into a 0–to–1 scale, where higher is better:

Click position

Reciprocal rank

1

1.000

2

0.500

3

0.333

5

0.200

10

0.100

Overall search MRR with ES|QL

MRR can be calculated two ways, depending on what you want to measure:

• All-click MRR: Averages the reciprocal rank of every click and reflects overall click quality, including repeated interactions.

• First-click MRR: Averages only the first click per search (using search.first_click == true). It’s more comparable to traditional information retrieval (IR) evaluation and aligns with how you computed CTR.

For consistency with CTR and alignment with judgment-list workflows in Blog 5, we prefer first-click MRR:

Result: MRR = 0.495

This is decent but shows room for improvement. An MRR of 0.495 means the average first click lands around position 2. It isn’t a crisis, but there are queries where ranking can be improved.

MRR by search query: Finding your worst-ranked results

Like CTR, the per-query breakdown is where the actionable data lives. To surface the worst-ranked queries first, sort ascending.

This reveals which queries have the worst ranking. A query with multiple clicks and low MRR means that the ranking is consistently poor for that search; that is, users find results, but they have to dig for them.

What is a good MRR score for search?

• MRR > 0.8: This ranking is solid; users usually click position 1–2.

• MRR 0.5–0.8:  This is decent, but there’s room for improvement.

• MRR < 0.5: This is a ranking problem, and users are scrolling past top results.

• MRR drop after a change: This is ranking regression that should be investigated immediately.

• Low MRR + high CTR: Users are finding things, but they have to work for it.

That last pattern is particularly interesting. High CTR with low MRR means your results are relevant (users are clicking), but your ranking isn't surfacing the best results first. It's an optimization opportunity, not a crisis.

MRR limitations: Position bias and multi-click sessions

MRR is heavily influenced by the gap between position 1 and position 2 (1.0 versus 0.5). Positions 5 and beyond barely move the average. This means that MRR is most sensitive to whether your top result is good, which is often exactly what you want to optimize.

MRR also only measures clicks, not satisfaction. With all-click MRR, a user who clicks position 1, bounces, and then clicks position 3 contributes two data points, but only the second was useful. The first-click MRR queries above avoid this by counting only the first click per search via search.first_click == true.

Click position distribution

Click position distribution shows you the full picture of where in the results users are engaging.

Results:

Position

Clicks

1

21

2

10

3

7

4

4

5

3

This is a reasonable distribution: 21 of 45 clicks (47%) land on position 1, with a tapering tail. If you paste this query into Discover's ES|QL editor, Kibana auto-generates a bar chart that makes the shape immediately visible.

How to read click position distribution for search relevance

• Sharp dropoff after position 1: The ranking is effective, and the top result is usually right.

• Flat across positions 1–5: The ranking isn't differentiating well, and all positions are equally likely to be clicked.

• Spike at position 3+ for specific queries: Those queries have ranking problems.

• No clicks beyond position 5: Users don't scroll far. Top 5 ranking matters most.

Click position distribution by search query

To see the shape for specific queries:

A query where all clicks land on position 1 has perfect ranking. A query where clicks spread across positions 1–5 needs relevance tuning.

Position bias and click models

One caveat: Click position distribution is influenced by position bias; that is, users see position 1 first, so it gets clicked more regardless of relevance. A click at position 1 isn't necessarily more relevant, just more visible.

This is a well-studied problem in information retrieval. Click models are statistical models that attempt to separate genuine relevance from position bias in click data. The foundational work by Joachims et al. (2005) showed that users are significantly biased toward higher-ranked results, and, in proposed methods like skip-above analysis (if a user clicks position 3 but skips positions 1 and 2), those skipped results are likely less relevant for that query.

For the metrics in this post, you don't need to implement a full click model. The key insight is practical: Compare distributions between queries rather than treating absolute position counts as ground truth. If query A has 80% of clicks at position 1 and query B has clicks spread across positions 1–5, query B's ranking is worse, even accounting for position bias. Later in the series, when we look at building judgment lists for LTR, position bias correction becomes more important, and the click data you're collecting here is exactly what those models need as input.

CTR, MRR, and click distribution: Reading search quality metrics together

These three metrics offer three different angles on search result quality:

Metric

What it measures

Our value

Interpretation

CTR

Do users click at all?

28.1%

Moderate: Roughly a third of searches get engagement, but there’s room to improve.

MRR

Where do they click?

0.495

Decent: The average click is around position 2, but ranking can be improved.

Distribution

What's the shape?

47% at position 1

Reasonable drop-off: The top result wins most but isn’t dominant.

Together, they tell a coherent story. For our demo data, search is performing adequately: Users are engaging with results and can find what they need, but the ranking has room to improve. The CTR of 28% and MRR of 0.495 are realistic starting points for a new search implementation without tuning.

Where they're most valuable is in combination at the query level. The queries to fix first are those with high volume + low CTR + low MRR; that is, lots of users are searching, few are clicking, and those who do click are scrolling deep. That's where relevance investment has the highest return.

Using click data for LTR and relevance tuning

These metrics don't just tell you how search is performing; they're the foundation for making it better. The click data you're now collecting feeds directly into relevance improvement workflows:

• Judgment lists for LTR: Click positions and frequencies become graded relevance labels for training machine learning (ML) ranking models. A document clicked at position 1 across many queries is a strong positive signal.

• Relevance tuning tools: Per-query CTR and MRR tell you exactly which queries to focus on in tools like Relevance Studio or the Rank Eval API, which scores your ranking against expected results.

• Query rules and boosting: Zero-CTR queries with results are candidates for pinning, boosting, or synonym rules.

We'll cover these applications in detail in Blog 5. For now, the important thing is that the instrumentation you've built here is doing double duty: It measures search quality and provides the training data to improve it.

Next in the series: Conversion tracking from search to purchase

We can now measure whether users find results (Blog 2) and whether they engage with them (this post). But a click isn't a conversion. A user who clicks a product and then abandons the page didn't get what they needed.

In the next post, we add conversion tracking, the third instrumentation point that closes the loop from search to purchase. It’s the same pattern: Add search.* attributes to add-to-cart and checkout spans, query with ES|QL, and answer the question your product manager actually cares about: Which searches drive revenue?

Get started

• Reference project: Working code for the entire blog series (clone, configure, and run).

• Elastic Distribution of OpenTelemetry Python: EDOT Python.

• OpenTelemetry with Elastic: How to send OTel data to Elastic APM.

• ES|QL documentation: Query language reference.

• UBI standard: Reference schema for search event structure.

This is the third post in a series on search analytics with OpenTelemetry and Elastic. Next up: From clicks to conversions: Conversion tracking, funnel analysis, and revenue attribution.

This is where Elasticsearch earns its place in the stack. It is already where a great deal of operational telemetry lives (logs, metrics, traces, events), and it exposes them through an expressive query and aggregation layer. That makes it a natural tool for an agent debugging a live system: it can slice by attribute, aggregate over time, correlate across signals, and drill from a symptom down to the documents that produced it.

We've been building an agentic layer on top of Elasticsearch that continuously monitors a system and root-causes issues as they arise. A recurring primitive in that workflow is time series event analysis. For example, given an error rate, a p99 latency, a queue depth and a throughput counter, tell me whether something happened, what it was and when. A transient spike in errors, a regime change in latency, and a step up in CPU usage are the signatures of the underlying fault, and they're typically what an agent examines first as it forms and tests hypotheses.

Elasticsearch has shipped a single-change-point aggregation for some time. It answers "did this series change?" with one verdict and the most significant change it found. That's a good fit for a dashboard, but less so for an agent, which often wants to interrogate a long window and enumerate everything of interest in it: the error spike at 02:14, the latency regime shift at 02:30, the throughput dip while the pod was being rescheduled. So we upgraded the capability to detect and report multiple events of multiple kinds in a single series. At the same time, we took the opportunity to further harden it to work reliably against whatever telemetry the agent points it at. This post describes how it works.

Why single change point detection isn't enough for agents

Concretely, we want a single entry point that takes a numeric time series and returns a small list of interesting events, each with a type, a location, a significance, and some key characteristics. We care about three classes of event, because they map onto three different kinds of underlying fault:

Event type

What changes

Detection channel

Example fault

Structural change

Level (step) or slope (trend) shifts to a new sustained regime

Value channel

Config push doubles baseline latency; memory leak turns a flat curve into a ramp

Distribution change

Noise level (variance) shifts while the mean holds steady

Dispersion channel

Service responds erratically at the same average latency

Point anomaly

Isolated spike or dip against a stable background

Value channel (pulse detector)

Single burst of errors; one-minute throughput drop during GC pause

The hard part is not detecting any one of these on clean, well-behaved data. The hard part is doing it on arbitrary telemetry without per-series tuning. The agent does not know in advance whether the series it is examining is near-constant, smoothly drifting, heteroscedastic (quiet in places and noisy in others), sparsely populated, or has a magnitude of . It is not scalable to hand-pick parameters for every series it needs to analyze. Whatever we build has to be robust to all of that while maintaining excellent recall and precision. If it fails to detect important events it runs the risk of missing key corroborating evidence for a working hypothesis. Conversely, an analysis tool that reports an event for every minor fluctuation will pollute the context the agent reasons over.

The design goals, in priority order, are: correct on diverse data out of the box, parsimonious (report only what matters), and cheap enough to run interactively across many series.

How PELT and BIC power change point detection

Change-point detection is a well studied field. The classical offline formulation searches for the segmentation of a series that minimizes a penalized cost: a per-segment goodness-of-fit term plus a penalty for each added break to stop the optimizer from putting a boundary between every pair of points. Solved naively, this is combinatorial, but PELT (Pruned Exact Linear Time, Killick et al.) finds the optimal partition in roughly linear time by using a dynamic program to prune candidate boundaries that can never be optimal. On the labeling side, comparing nested models by an information criterion such as the Bayesian Information Criterion (BIC) gives a principled, scale-aware way to decide whether a candidate break is really important and what sort of change it constitutes.

These are good building blocks, and we use them. However, the textbook recipe assumes more than telemetry gives you. It typically assumes a single change type (a mean shift), a known and stationary noise level, and reasonably benign numerics. Real telemetry violates all three: variance changes matter as much as mean changes, the noise level is unknown, often heavy-tailed and changing, and the data spans extreme magnitudes and degenerate cases, such as perfectly constant segments, that wreck an ill-conditioned polynomial fit or a fixed-variance cost. Most of the engineering I describe below is about closing that gap.

Splitting one time series into three detection channels

Rather than trying to find one detector that does everything, we run three focused detectors and then merge their findings. Two of the three are the same structural detector applied to two different views of the data, or channels.

Two detectors run on the value channel: a structural detector that flags step and trend changes, and a pulse detector that identifies point spikes and dips. The dispersion channel, a windowed measure of spread, is fed to a second copy of the structural detector, where a variance change shows up as a level shift and is relabeled a distribution change. The two channels are complementary by construction: a step is a level shift the value channel flags but only has a single large first difference the dispersion channel ignores, while a variance change is invisible to the value channel yet shows up clearly in the dispersion channel. A thin orchestration layer then merges and de-duplicates the streams.

Keeping the three concerns separate makes each one tractable. A mean-shift detector and a variance-shift detector pull in opposite directions if you try to fuse them; a point-anomaly detector and a regime detector need opposite robustness settings. Separated, each can be tuned to its job.

PELT with a scale-free cost

For the structural channel, we fit each candidate segment with a low-order polynomial (constant or linear) and score it with the profiled-variance Gaussian cost. If a segment of length has residual sum of squares , the cost is

This is the negative log-likelihood of a Gaussian segment after profiling out the variance, i.e., after substituting the maximum-likelihood estimate back into the log-likelihood. The total objective PELT minimizes is the sum of segment costs plus a per-break penalty,

Here, is the BIC complexity term (number of parameters times where is the number of values in the time series) and the scale factor lets us trade sensitivity against parsimony in one place.

Using the profiled cost rather than a cost against a fixed global variance is a deliberate and important choice. The global noise level of telemetry is unreliable: on a smoothly varying series the natural estimate (the spread of first differences) can collapse toward zero, and a fixed-variance cost then treats every wiggle as enormously significant and over-segments. The profiled cost depends only on the ratio , so it is invariant to the absolute scale and immune to that failure. A small floor on keeps the logarithm finite, so a zero-residual segment is not rewarded without bound.

Keeping the fit stable using robust local weighting

Before PELT runs, we robustly down-weight points so that an excursion does not create spurious breaks or drag a segment boundary onto itself. Crucially, these weights enter into the weighted residual moments, so they shape not just the segment fit but its residual variance, and so the segment costs themselves. Each point gets a Cauchy weight, , of its residual from a rolling-median baseline against a robust scale : points near the local median keep full weight and points far from it are progressively discounted. A nice side effect of measuring excursions from the median on a window centered on each point is that clean structural breaks do not get down-weighted at all, because the majority of values land on the same side of the break as the point whose residual is being computed. So a sustained regime keeps full weight, but a lone spike does not.

There is a nice justification for scoring a weighted Gaussian cost when what we really want is robustness to a heavy tail. The Cauchy weight is exactly the iteratively reweighted least squares (IRLS) weight of its loss: , so the weighted normal equations are identical to the Cauchy M-estimator's estimating equations . A weighted-mean (or weighted-line) fit at those weights is therefore a Cauchy M-estimate, not a Gaussian one. The cost we actually evaluate inherits the same properties. Because is concave in , its tangent at the current residual lies above it. This gives a pointwise bound with the weight at the tangent point; summing, the weighted residual sum of squares is a tangent upper bound on the total Cauchy loss, touching it in both value and gradient at the weights' anchor point. Minimizing the weighted RSS is thus one step of a majorize–minimize scheme that provably decreases the true Cauchy objective, and the profiled-variance cost we feed the BIC is that majorizer standing in for the Cauchy deviance. The only approximation is that we anchor the weights once, at the rolling-median baseline, rather than iterating IRLS to its fixed point; this is exact for the inliers that sit near the baseline, and correct for gross outliers, whose vanishing weight removes them from the cost wherever the bound is loosest.

Finally, the trick that makes this work on heteroscedastic data is that the residual is judged primarily against a local robust scale, not a global one: the MAD of residuals in the same sliding window. On a series that is quiet in one stretch and noisy in another, this means a spike in the quiet stretch that is multiple local sigmas is correctly suppressed. The local MAD can collapse on quiet stretches, so we use a backstop that is a fraction of a global composite of robust scales and a floor related to the quantization error for discrete series and numerical precision otherwise.

From candidates to labeled events using BIC verification

PELT gives a globally optimal penalized segmentation, so we take its boundaries as candidates and verify each one. For a candidate at index we look at the window of length spanning to its nearest neighboring candidates and compare a no-change null against step and trend alternatives by BIC,

where counts the fitted parameters (the same parameter count as in the PELT penalty above). We map the BIC gain of an alternative over the null to a significance via , to turn a threshold into a decision boundary). We treat this as a significance score for ranking and thresholding, not as a calibrated tail probability.

At this stage we allow higher-degree models to avoid splitting smoothly varying trends. These are problematic in PELT itself because it considers short segments, which they overfit. We keep the most parsimonious alternative that clears the significance threshold and survives a persistence check. The persistence check re-scores with the weights immediately around the candidate muted, and if the evidence collapses, the "change" was driven by a few extreme points – an excursion, not a regime change – and we reject it. The polynomial order is applied symmetrically to the null and to each side of the split, so the alternative is always the same model class merely split at the candidate, and therefore strictly more flexible. The whole process can be thought of as Bayesian model selection with a preference for the null.

When no candidate survives, we still say something useful: we report the series as "stationary" (best no-change model is a constant) or "non-stationary" (best model has a slope), with the trend direction. For an agent, "this series is cleanly trending up over the window" is itself a finding.

Detecting distribution changes with a dispersion channel

A variance change is indirectly visible to the mean channel – worse, the robust weighting there actively mutes the excursions that signal it. So we detect it on a separate dispersion channel and reuse the exact same structural detector, because on this channel a variance change is just an ordinary level change.

The channel is built from one sample per non-overlapping window. Within a window, we take the inter-quartile range of the first differences, rescaled to a standard-deviation equivalent (), and pass it through :

Then . Three choices matter here. First-differencing cancels level and slope, so a mean step contributes a single large difference rather than inflating the whole window, and a steady ramp produces a flat channel. Non-overlapping windows keep the samples independent; overlapping windows autocorrelates the channel and makes the segmenter over-detect. And the IQR is used rather than the median (which is too robust and will miss a window that is 40% noisy then flatlines) or the raw standard deviation (which is not robust enough since one spike's two large differences inflate the window). Because the dispersion channel is a fraction of the original length, the verifier there is restricted to a lower-order null so a genuine low-high-low variance bump is not absorbed.

The functional form is worth dwelling on, because each part earns its place. The log makes the channel respond to ratios of noise level rather than absolute differences. Variance changes in telemetry are typically multiplicative: a regime is "twice as noisy". On a raw-scale channel, a doubling shows up as an enormous absolute jump at a high baseline and a negligible one at a low baseline, so an additive step-cost detector would find variance changes trivially in loud series and miss them in quiet ones. Under a log, a factor- change in scale is the same offset wherever it occurs, which is exactly the additive-step behavior the structural detector is built for. The " is a soft floor. A bare diverges to as the scale goes to zero, which is precisely what happens on a near constant stretch, and would manufacture a huge spurious step at the first noisy window after it. Conversely, is finite and smooth at zero, behaves linearly () while the noise is small, and recovers the multiplicative behavior once the noise is appreciable. This gives graceful degradation instead of a singularity, and with no tuned epsilon to pick. Note that squaring the scale (using a variance instead) only doubles the dynamic range; it makes no difference to the detector either way, since differs only by a constant the threshold absorbs.

Detecting point anomalies as excursions from a local baseline

Spikes and dips are detected as point excursions from the local rolling-median baseline. Working from the local residual rather than raw values means level structure is removed and smooth curvature is tracked; even for time series that change significantly the detector is sensitive to significant local deviations.

The pipeline is a generous proposer followed by a strict gate:

1. Propose every point whose residual exceeds a threshold number of robust sigmas. The scale is the larger of the global first-difference noise (which stays meaningful on smooth data where most residuals are exactly zero) and a composite of robust scales of the residuals (which inflates once a frequent large-residual population appears).

2. Merge adjacent same-sign candidates into excursions, dropping any that span a full minimum segment: that is a regime, and is owned by the structural channels.

3. Rank and cap the excursions by peak z-score, keeping the top , so a pathological series cannot drown the output.

4. Gate using one shared null: build a Gaussian KDE from the series with all of the retained excursions removed, and keep an excursion only if its peak's Bonferroni-corrected upper-/lower-tail probability under that null clears the threshold.

Removing all the tested excursions from the single null at once is a key trick. The leave-one-out alternative – score each excursion against a null containing the others – lets the largest spike and dip mask everything else. Removing them together means several genuinely distinct excursions are each judged against the remainder and all survive, while a recurring population is still rejected.

The proposer and the gate ask deliberately different questions, and that distinction drives two further choices. The proposer works on residuals from the rolling median since it wants recall, and a residual is what tells you a point stands out from its local neighborhood. The gate is value-based: it asks "is this magnitude one we see at other times in the series?", so a spike to a level that recurs elsewhere — such as periodic batch jobs — is suppressed even though it is a large local residual. Those are the right semantics for an agent, but they expose a heteroscedasticity problem, because telemetry noise is almost always a function of magnitude. Periodic spikes can sit orders of magnitude above the background, and a single KDE bandwidth fitted to the whole value range is then far too narrow up in the high tail. So ordinary large values come back as significant, a steady source of false positives.

The fix is a variance-stabilizing transform. We run the value gate in space, where is a robust measure of the spread of the background. is linear for and logarithmic for , which turns a multiplicative (magnitude-dependent) spread into a roughly constant one, so a single bandwidth is valid across orders of magnitude. It is also odd and finite at zero, so exact zeros and sign changes (dips below a small baseline) need no special handling, unlike a bare log. Crucially, it is monotone and so does not change what is tested (for any monotone function , ) so it only fixes the estimate of that tail probability.

One subtlety closes the loop. The KDE null and the kernel bandwidth are taken from different scales, on purpose. The null is the stabilized background values: so it models any mode in the data, which is what makes a recurring large magnitude unsurprising. However, the bandwidth is taken from the stabilized residuals, not the stabilized values, because a genuine level change makes the value distribution bimodal, and a bandwidth computed from that bimodal spread would balloon, masking a real spike sitting on top of a shifted regime. The residual removes the step, so the bandwidth always reflects within-regime noise and the gate stays sensitive to a deviation that is extreme relative to its own neighborhood if it is also outside the envelope for the series as a whole.

Merging structural, distribution, and point anomaly events

Finally, an orchestration layer merges the structural, distribution, and point-anomaly event streams. Structural and distribution events that mark the same regime boundary are de-duplicated to the more significant one (a boundary that shifts both level and spread is one event, not two). Pulses are a separate stream added after de-duplication, because a spike that lands on a structural boundary is a real, separate finding and must not be suppressed. Everything is then mapped back from the internal value-array index space to source-bucket indices.

Handling extreme magnitudes and edge cases

What makes this usable as an unattended tool is a collection of defensive choices for the cases that break naive implementations:

• Variance computed as loses all precision at large magnitudes: a constant series at can manufacture phantom change points purely from floating-point error. We center every PELT input by a constant offset first; the polynomial RSS is invariant to that shift in exact arithmetic, but the working magnitudes drop from to .

• Using raw indices as the regressor results in poor condition polynomial fits: the largest moment is , which is about for a cubic over a 2000-point window. This trips the SVD singularity guard and silently degrades the fit. Mapping affinely onto leaves the fit identical (RSS is invariant under reparametrization) but every moment becomes .

• Scale-free cost, as described, means the segmentation cost doesn't depend on tuning a noise estimate.

• Down-weighting wants a primarily local scale: suppress whatever is anomalous in its own neighbourhood. It uses the maximum of MAD and a small global floor on the differences from the rolling median. Conversely, spike/dip detection wants a global scale: we care about global outliers. It uses a composite of robust scales of all differences. Using the wrong one in either place produces characteristic failures – irrelevant spikes in a quiet segment, or locally large excursions creating spurious breaks – and maintaining separate channels allows us to pick appropriately.

• Using one p-value threshold, Bonferroni-corrected by the number of candidates, applied consistently across all three detectors, means that "how surprised should I be" is consistent everywhere.

The recurring theme is that the difference between a detector that works in a notebook and one that works on a firehose of production time series is mainly in handling the edge cases gracefully.

Performance: ~1ms per series on a single core

Detection cost is dominated by PELT. Its segment cost is a profiled-variance linear fit, which we evaluate in constant time from prefix-summed weighted moments rather than maintaining a regression per candidate boundary, so a single segment cost is a handful of array reads and a 2×2 solve. Cost grows a little faster than linearly with series length since PELT's pruned candidate set does not stay constant on noisy data. Therefore, to bound the worst case on very long series, we downsample ahead of detection: above a cap (2000 samples), the series is collapsed into macro-buckets, keeping two samples per bucket: the median and the largest local deviation. This is inspired by the M4 downsampling scheme, but because we need only the median and the largest excursion for structural-change and outlier detection, respectively, we can then afford to double the bucket resolution. The downsampled series carries its original bucket indices, so every reported event still maps back to a real source bucket; below the cap it is a no-op. The whole analysis is a single pass over the (possibly downsampled) series with no per-series configuration. This lets the agent call it freely across many signals.

In absolute terms, this means the analysis is comfortably interactive. On a single core, post-warmup, a typical series of 140–350 buckets is analysed in about 1 ms (≈220,000 buckets/s), and a series long enough to hit the downsample cap (say 5,000 buckets, collapsed to 2,000) takes about 40 ms, which is the effective worst case per call. For an agent issuing a handful of these calls per investigation, and parallelising across the many series in a `BY` query, the latency is negligible.

Evaluation on synthetic and production telemetry

Synthetic benchmark

We evaluate first on a synthetic generator with known ground truth, because it lets us measure the things that matter precisely. The generator produces random time series with diverse behaviors, which we group into three families. The positive family injects known events of each type: clean and noisy step changes (up and down, SNR around 10, at several positions), trend onsets and ramps (including a flat–ramp–flat sequence with two boundaries), variance changes with a constant mean (single steps and a low–high–low bump), and isolated spikes and dips. The null family ideally produces no event: stationary noise, perfectly constant series, smooth quadratic drift and clean ramps, and periodic signals. (A variance change with a constant mean is not null: it is a distribution change, an abrupt step on the dispersion channel, and so it belongs in the positive family above. The related null requirement, that such a change does not surface on the value channel as a step, is checked separately.) The adversarial family stresses the robustness machinery: a perfectly flat series at a magnitude of (for which a naive variance arithmetic manufactures phantom breaks here through catastrophic cancellation) and, conversely, a genuine spike or step in a noisy baseline as high as (which must still be found and located, the high baseline notwithstanding); a spike on top of a step; a within-regime spike after a 100 level jump; a recurring train of equal peaks (a population, not individual spikes); wide sustained excursions (a structural change, not a spike or dip); and fuzzed random level-shift series. On these series we track recall per event type, precision and the false-positive rate on the null family, localization error, parsimony (events per series and adherence to the count limit), and invariance under constant offsets, rescaling and extreme magnitudes.

The table below summarizes what the suite tests.

Event family

Representative scenarios

Required outcome

Localization tolerance

Step

clean / noisy, up / down, single and multiple, several positions

detected

≤ 4–8 buckets

Trend

slope change and flat–ramp–flat, clean and in noise

detected

≤ 8–12 buckets

Distribution

variance step (mean constant), low–high–low bump

detected

≤ 1 dispersion window

Spike / dip

isolated, multiple distinct, on heavy-tailed and high-magnitude series, within-regime after a step

detected, capped at max(5, 2% of n)

≤ 2 buckets

Null series

stationary noise, constant, smooth drift / ramp, periodic

no event reported

n/a

Invariance / robustness

constant offset, rescaling, 10^5–10^9 magnitudes, spike-on-step, recurring population, wide excursion

result unchanged / no spurious event

n/a

Running this over 400 series per scenario, just over half a million buckets in total, and scoring with the same two categories we use for the real-data evaluation below (any regime change versus point spikes and dips) gives:

Event type

Recall

Precision

Median localization error

Structural change

0.994

0.664

0

Spike / dip

0.847

0.746

0

Two things stand out. When an event is detected, it is placed essentially exactly: the median localization error is zero buckets for both categories; and the point-wise accuracy, 0.998, is directly comparable to the 0.995 we report on real telemetry below: the overwhelming majority of buckets are correctly left unmarked.

The precision figures are lower than on real data, and understandably so. A sixth of this population is a hostile null family (periodic signals, smooth drift, clean ramps) chosen precisely because they tempt a detector into a spurious break, and every false alarm on them counts against precision. The resulting false-positive rate is 0.5% per bucket, with 15% of null series carrying at least one spurious event. We treat this as the conservative end of the range: on the real-telemetry mix below, where the null series are less adversarial, precision rises to 0.85 (structural) and 0.90 (spikes/dips). Finally, offset and scale invariance holds on all 400 series: the same events, to within a few buckets, whether the series is shifted by a constant or rescaled by up to three orders of magnitude.

That raw precision also misses how the result is consumed. The agent reads events most-significant-first, so a false positive only does harm if it outranks a genuine one, and by and large it does not. The median p-value of a true positive is about , against about for a false positive: the genuine events are typically overwhelmingly more significant. Concretely, if we keep only the top events per series by significance (with the true count) precision rises to 0.94, and a randomly chosen true positive is more significant than a randomly chosen false positive 93% of the time. It is not a perfectly clean separation: a strong periodicity or a sharp curve genuinely can contain a significant-looking break, which is why that figure is 0.94 rather than 1. However, the ranking is reliable enough that an agent reading from the top, or applying a stricter significance cut-off, sees the real events first and the false alarms as a lower-significance tail. This is also why exposing the detector's significance to the agent (see What's next) matters more than squeezing the raw precision higher.

Real cloud telemetry

To evaluate on real data, we scraped around 300 metrics from our production cloud environment. These cover HTTP status-code counts, failed memory allocations, memory usage, network usage, page faults, CPU usage, and throttling metrics, measured both per instance and aggregated across the fleet as a whole. Their values range over more than 12 orders of magnitude, and they display a variety of behaviors including ramps, periodicity, step changes, distribution changes, and trend changes. The figure below shows a sample of series together with the detections we make on them.

To get a sense of the accuracy on this set data, we labeled a subset of 150 most interesting time series, marking the visually clearest features in each. This labeling is not necessarily optimized for our target use case, where false negatives are typically more problematic than false positives: an agent consumes these results as part of a broader investigation and can pull additional information to corroborate them. Even so, we find excellent agreement with human judgment on these series. Since each series comprises between 140 and 350 points, the point-wise accuracy, at 0.995, is extremely high: the great majority of points are correctly identified as neither a change, a spike, nor a dip. But the more telling metrics are recall and precision on the human-labeled points, shown in the table below. The human labels did not attempt to categorize each change, so we break the results down only into "structural changes" and "spikes / dips" — the same two categories, and the same point-wise accuracy and recall/precision metrics, as the synthetic benchmark above.

Event type

Recall

Precision

Structural change

0.94

0.89

Spike / dip

0.97

0.92

It is worth covering exactly why we get disagreements. These largely fall into three categories: spikes and dips in context, isolated breaks, and small-magnitude breaks in stable series. We deliberately do not try to detect spikes and dips that are unusual only in their immediate context; that is, not globally unusual but visually significant relative to an inferred periodicity in the data, for example. Trying to account for these without fully modeling the seasonality in the data hurt precision more than it helped recall, and we have a separate persistent anomaly-detection process that builds more complete models of baseline behavior over time. Isolated breaks are an artifact we decided to live with: PELT's cost function tends to isolate a change point with a few values intermediate between the two regimes, because absorbing it into either neighboring span inflates that span's cost. Humans are good at judging such situations visually and assign a single change point. Finally, small-magnitude changes are simply not visually obvious. We detect them deliberately and regard this as a strength of a quantitative approach, since they are often the early precursors of an incident whose later, larger effects drown them out.

How the agent uses change point results in ES|QL

To the agent, all of this is one tool call: it points it at a collection of time series and gets back a typed, located, ranked list of events. That list is small by construction, which means it drops cleanly into the model's context without crowding out everything else it is reasoning about. Because the result contains multiple events, a single call over a window can hand the agent the whole local story – "error spike at 02:14, latency regime change at 02:30, throughput dip at 02:31" – and let it correlate across signals to a root cause.

Operationally, we expose this through both the ES|QL CHANGE_POINT command and the change_point aggregation. Currently, we have not extended the change_point aggregation to return multiple change points since it breaks backwards compatibility of the output schema. It just returns the most significant event. We don't have the same restriction for ES|QL since it returns change points annotated onto the table rows to which they apply. We do plan to revisit the output schema for both ES|QL and the aggregation in a later version. We'd like to migrate to optionally returning significance in log-space, which doesn't underflow, and including a short verbal description of each change, which we expect to help agents when seeing just the change points themselves.

ES|QL is Elasticsearch's piped query language, and CHANGE_POINT runs the detector as one stage in a pipeline. Its BY clause enables it to analyze many series at once (one per group) so the agent can, in a single query, segment every service's latency or every host's error rate side by side rather than issuing a call per series. The actual leverage, compared to the change_point aggregation, is composability: the events come back as ordinary rows in the pipeline, so the agent then has the entire ES|QL language to manipulate them downstream. It can filter to a window, join change points against deploy markers, count events per service, rank by significance, feed the survivors into a further aggregation, and so on.

For example, suppose the agent wants to find out whether any servers have recently seen a sudden CPU spike or a prolonged step change in CPU usage over the last 12 hours, and whether that might point to a load-balancing issue. It could use the following query:

Here it's using a STATS ... BY host.pod to see how the detected events cluster across other dimensions of the data, such as the Kubernetes pod, and so judge whether they share a common cause.

What's next for ES|QL time series analysis

As far as detecting events of interest in time series, the foundation is in place: a single, robust, parsimonious tool that turns a raw telemetry series into the short list of events that actually matter, which is exactly the kind of reach an agentic SRE needs. Going forward, we plan to explore the best mechanism for feeding the detector's uncertainty to the agent, so that a borderline event can be flagged as "worth a second look" rather than silently included or dropped. Also, this is the first of several analytical tools we plan to build into the ES|QL query language to enable agents to triage and RCA issues more effectively; so stay tuned for further updates.

What you'll discover

In this post, you'll learn how to:

• Set up OpenTelemetry in an example Python FastAPI back end.

• Add custom search.* attributes to your search spans in ~20 lines of code.

• Understand how OTel-native ingestion maps attributes to queryable data.

• Write six ES|QL queries against real trace data: top queries, zero-results rate, which queries return nothing, average and max latency, slow query investigation, and search volume over time.

• Turn those queries into saved Kibana visualizations.

What you'll need

• An Elastic Cloud deployment (or self-managed with OTel-native ingestion enabled). Examples tested on Elastic Stack 9.x.

From concept to code: Building the search analytics instrumentation

In the first post, we made the case for using OpenTelemetry to capture search analytics. The idea: Add search.* attributes to your existing OTel spans, send them to Elastic APM, and query them with ES|QL.

Now let's build it.

Want working code? A companion reference project accompanies this series. It's a minimal FastAPI app with the exact instrumentation described below. Clone it, add your Elastic Cloud credentials, and you'll have search analytics data flowing in 10 minutes. Each blog stage maps to a commented-out code block you can enable as you progress.

An OpenTelemetry primer for search API developers

If you've been building search systems but haven't worked with OpenTelemetry before, here's the minimum you need to know.

OTel is an open standard for collecting observability data, traces, metrics, and logs from your applications. It's vendor-neutral: You instrument your code once, and send data to any compatible back end.

The core concept is the span. A span represents a single operation, for example, an API call, a database query, or a search request. Every span has a start time, an end time (the difference is the span duration), and attributes, which are key-value pairs that describe what happened.

Spans nest inside each other to form traces. A trace is a tree of spans that represents one end-to-end request. When a user searches, the trace might look like: browser request → API handler → search logic → Elasticsearch query. Each step is a span, and the parent-child relationships show you exactly where time was spent. This is distributed tracing, which works across services and network boundaries, so a single trace can follow a request from front end to back end to database and back.

Why traces instead of logs? You could log "search query=headphones results=15 took=120ms" and parse it later. But a log line is flat; it can't show you that the 120ms Elasticsearch time sat inside a 250ms API call, revealing 130ms of overhead in your application layer. Traces give you hierarchy, timing, and correlation across services. For search analytics, that means you can see not just what happened but also where time was spent and how operations relate to each other.

For this post, we don't need to understand the full OTel ecosystem. We just need three things:

1. Create a span when a search request happens.

2. Add attributes to that span, describing the search (such as search.query or result_count).

3. Send the span to Elastic, where we can query it with ES|QL.

That's it. If you can call span.set_attribute("key", value), you can build search analytics.

What you'll build

By the end of this post, every search request in your API will emit an OTel span that looks like this:

And you'll run six ES|QL queries against real data to answer questions that your team is already asking, using about 20 lines of instrumentation code in total.

Install the OTel SDK

We're using Python and FastAPI here. The same pattern applies to any language with an OTel SDK; the concepts are identical, only the imports change.

Elastic provides the Elastic Distribution of OpenTelemetry Python (EDOT), which bundles the standard OTel SDK with sensible defaults, early access to Elastic-contributed improvements, and a single configure_opentelemetry() call that handles all the boilerplate. We recommend it:

Three packages, two roles:

• elastic-opentelemetry EDOT: The OTel API, SDK, and OpenTelemetry Protocol (OTLP) exporter in one package, preconfigured for Elastic.

• opentelemetry-instrumentation-fastapi: Auto-instruments HTTP endpoints (automatic spans for every request).

• opentelemetry-instrumentation-elasticsearch: Auto-instruments Elasticsearch client calls (automatic spans for every query).

The auto-instrumentation packages are doing real work here. Without writing a single line of tracing code, you already get HTTP request spans and Elasticsearch query spans. What we're adding is the search-specific context that turns generic traces into analytics.

Using the standard OTel SDK instead? Replace elastic-opentelemetry with opentelemetry-api, opentelemetry-sdk, and opentelemetry-exporter-otlp-proto-http. You'll need to wire up the TracerProvider, OTLPSpanExporter, and BatchSpanProcessor manually (about 10 extra lines). Everything else in this post works the same either way. See the Elastic OTel guide for the full setup.

Configure the connection

OTel uses environment variables for connection configuration. You need four to get started:

Variable

Purpose

Example

`OTEL_EXPORTER_OTLP_ENDPOINT`

Managed OTLP (mOTLP) endpoint URL

`https://my-deployment.ingest.us-central1.gcp.elastic-cloud.com`

`OTEL_EXPORTER_OTLP_HEADERS`

Authentication

`Authorization=ApiKey <your-api-key>`

`OTEL_SERVICE_NAME`

Service name (shown in Kibana APM)

`search-analytics-demo`

`OTEL_RESOURCE_ATTRIBUTES`

Other resource attributes

`service.version=1.0.0`

Where to find these values: In Elastic Cloud, your mOTLP endpoint follows the pattern https://<deployment>.ingest.<region>.gcp.elastic-cloud.com. You can find it in the Elastic Cloud console under your deployment's details or in Kibana at the APM integration page (/app/home#/tutorial/apm) under the OpenTelemetry tab. Your API key can be created from Kibana's Stack Management > API Keys or via the Elasticsearch Create API Key API. For self-managed deployments, you can use the EDOT Collector as an intermediary that receives OTLP and forwards to Elasticsearch.

Set them in your environment or .env file:

Initialize the tracer

With EDOT and environment variables configured, initialization wires up three things: the tracer provider and two auto-instrumentation packages that automatically create spans for every HTTP request and every Elasticsearch query:

configure_opentelemetry() reads the OTEL_* environment variables and sets up the tracer provider, exporter, and batch processor with Elastic-optimized defaults, including using the HTTP exporter automatically, which is what the mOTLP endpoint requires. If you see connection errors with vanilla OTel, the most common cause is accidentally using the gRPC exporter instead of HTTP.

The two instrumentors patch the FastAPI and elasticsearch-py libraries at startup so every request and every Elasticsearch call automatically generates a span, without any code changes to individual endpoints.

Instrument your search API

Here's where it gets interesting. This is the code that turns a generic API endpoint into a search analytics source:

A few things worth unpacking.

• start_as_current_span creates the span and sets it as the active span in the current context. This matters because the Elasticsearch client instrumentation picks up the active span and nests its own spans underneath it. You get a span hierarchy automatically.

• search.query_id is derived from the trace ID. Every trace already has a unique identifier, and we're reusing it as the query identifier. There’s no UUID generation or database sequence. When we add click tracking later, clicks will reference this same query_id to link back to the search that produced the results.

• search.result_count does double duty. It tells you how many results came back, and when it's zero, you know you have a content gap. There’s no need for a separate boolean flag: Just filter on result_count == 0 in your queries.

Note that the application name is no longer set as a span attribute; it's the service.name resource attribute, configured once via OTEL_SERVICE_NAME. This is the standard OTel approach: Resource attributes describe the service, and span attributes describe the operation.

Normalize search queries before analyzing them

Notice that we're storing request.query as is. That means "Laptop Bag", "laptop bag", and " laptop bag " will be counted as three different queries when you aggregate with STATS ... BY attributes.search.query.

For cleaner analytics, normalize before setting the attribute:

Lowercasing and trimming whitespace is enough for most cases. If you need the original phrasing (for display or debugging), store it in a separate attribute, like search.query.original. But start simple. You can always add the raw version later if you find you need it.

What a search API trace looks like

Once this is running, a single search request produces this trace:

The auto-instrumented spans give you HTTP latency and Elasticsearch query detail. Your search span in the middle ties them together with the business context: what the user searched for, how many results came back, how long Elasticsearch took.

The search span attributes you need to capture

Here's the full set of attributes we're capturing on the search span:

Attribute

Type

When set

Purpose

`search.query`

string

Before query

The query as the user entered it

`search.query_id`

string

Before query

Unique identifier, derived from trace ID

`search.result_count`

int

After query

Total matching results (0 = zero-result search)

`search.took_ms`

int

After query

Elasticsearch execution time in milliseconds

`search.query_response_hit_ids`

string[]

After query

Document IDs returned (optional; enables per-result analytics)

`feature_flag.key`

string

Before query

A/B test flag name (optional; pair with `feature_flag.result.variant` for the assigned variant; enables per-variant click-through rate (CTR) comparison)

We use the search.* namespace following OTel's convention of domain-specific prefixes (http.*, db.*, messaging.*). While there aren't standardized search conventions in OTel yet, search.* is self-describing and vendor-neutral. The naming is informed by the User Behavior Insights (UBI) Standard, which defines a schema for search events. We reference it for structure without coupling to it. Where established OTel conventions exist, like feature_flag.key (flag name) and feature_flag.result.variant (assigned variant) for A/B experiments, we reuse them rather than inventing custom attributes.

You'll notice that enduser.pseudo.id isn't in the search span table above. We don't need it for query analytics, but you'll add it as soon as you introduce click tracking in our third blog; it ties click events back to a specific browser session, enabling per-user CTR and Mean Reciprocal Rank (MRR). Blog 3 adds enduser.pseudo.id (browser-generated, persistent across sessions) to link clicks back to searches. Our fourth blog focussing on revenue attribution documents session.id and user.id as optional extensions for authenticated users who want cross-device attribution.

How OpenTelemetry attributes become queryable ES|QL fields

Before we start querying, you need to understand how OTel attributes map to Elasticsearch fields. With OTel-native ingestion into Elastic, the mapping is straightforward.

OTel attribute

Type

ES|QL field

`search.query`

string

`attributes.search.query`

`search.result_count`

int

`attributes.search.result_count`

`search.took_ms`

int

`attributes.search.took_ms`

`search.query_id`

string

`attributes.search.query_id`

`feature_flag.key`

string

`attributes.feature_flag.key`

With OTel-native ingestion, attribute names preserve their dot notation under attributes.*. All types live in the same namespace; there’s no split between string and numeric fields. Booleans are stored as native booleans, not strings. If you've used Elastic APM's classic ingestion before, you'll appreciate the simplicity: What you set in code is what you query.

Running ES|QL queries in Kibana Discover

With spans flowing to Elastic, open Kibana and go to Discover (in the left sidebar under Analytics, or use the global search bar and type "Discover"). By default, you'll see the KQL query bar, a filter language familiar from Kibana dashboards. Click Try ES|QL in the top right to switch to the ES|QL editor. Unlike KQL (which filters documents) or the JSON query DSL (which requires nested objects), ES|QL is a piped language: Each | step transforms the previous output, making aggregations like STATS count BY field read naturally from left to right.

The editor gives you a full-width text area where you type piped queries. Results appear as both a table and an auto-generated chart; Kibana picks a sensible visualization based on your query shape. For STATS ... BY queries, you'll get a bar chart automatically.

Set the time range wide enough to capture your data (top-right date picker). If you're just getting started, try "Last 30 days".

Six ES|QL queries for search analytics

Everything below runs against traces-generic.otel-default, the index where Elastic's OTel-native ingestion automatically stores trace data. You don't need to create this index; it's provisioned by Elastic when the first OTLP span arrives.

These queries ran against our live demo cluster: 62 searches, ~20 distinct queries, latency range 77ms–153ms.

Note: Results below are illustrative. When you run the reference project and generate traffic with python generate_traffic.py --blog 2 --sessions 50, your exact numbers and top queries will vary based on session count and random query selection. The query patterns and ES|QL syntax are what matter here.

Query 1: Are spans arriving?

Start simple. Count your search spans.

Result: 62.

If this returns zero, your spans aren't arriving. Check your OTLP endpoint and API key and that init_otel() is being called before any requests. The name == "search" filter ensures that you're counting your custom spans, not the auto-instrumented Elasticsearch client spans (which are also named "search").

Query 2: What are users searching for?

The first question every search team asks.

Results:

Query

Searches

Average results

laptop

9

8

headphones

7

12

running shoes

6

5

"laptop" was the most popular query, with nine searches. There were around 20 distinct queries total (including zero-result ones).

The avg_results column tells you whether popular queries are actually returning content. A query with high volume and low results is a relevance problem worth investigating. If your query has high volume and high results, check whether users are actually clicking. We'll get to that in the next blog focussing on measuring search quality with click data.

Query 3: What percentage of searches return nothing?

Result: 17.7% (11 out of 62 searches returned nothing).

We're using attributes.search.result_count == 0, a straightforward numeric comparison. No separate boolean attribute is needed when you already have the count.

A zero-results rate above 10% is worth investigating. Every zero-result search is a user who asked for something and got nothing back. Some of those are junk queries, but others reveal real content gaps or query parsing failures.

Query 4: Which queries return nothing?

The rate tells you there's a problem. This query tells you where.

Results:

Query

Occurrences

quantum physics calculator

4

unicorn saddle

3

holographic projector

2

time machine parts

2

Three different failure modes: "quantum physics calculator" and "unicorn saddle" are out-of-catalog queries you'll never be able to serve. This is useful to know but nothing to fix. "holographic projector" might be a real emerging category worth considering. "time machine parts" is probably noise. In a production catalog, these would be mixed with legitimate zero-result queries that are fixable, like missing synonyms, phrasing mismatches, or product gaps.

Repeated zero-result queries are the highest-priority fixes. One-off failures are usually noise.

Query 5: How fast is search?

Result: Average 81ms, max 153ms.

search.took_ms captures Elasticsearch's self-reported execution time, the took field from the search response. This is different from span duration, the wall-clock time from when the span started to when it ended (as we covered in the primer above). Span duration measures end-to-end time, including network round trips, serialization, and application logic. You want both: Comparing them tells you where overhead lives. If took_ms is 50ms but the span duration is 200ms, the extra 150ms is network or application overhead, not a query problem.

This attribute also keeps your analytics portable. If you're using OTel log records instead of spans (a lighter-weight alternative we mention in the first blog in the series), there's no span duration. took_ms is the only timing signal you have.

We'll go deeper on search performance monitoring (Service Level Objectives [SLOs], alerting on latency regressions, and using this data for operational dashboards) in the last blog in the series focussing on Search Reliability Engineering.

Want to find the slow queries specifically?

A query with high average latency and high result count is hitting many documents; consider query optimization. High latency with low results might mean complex filters or slow aggregations. Outlier max values are often cold caches or cluster issues.

Query 6: How does search volume change over time?

Counts, rates, and latencies tell you the what. Volume over time tells you the when: When did traffic spike, when did it drop, and when did that zero-results rate jump?

DATE_TRUNC(5 minutes, @timestamp) rounds each timestamp down to the nearest 5-minute boundary. The result is a time series that Kibana's Lens can render as a bar chart or line, showing your search traffic pattern for any time window.

Narrow the bucket for higher granularity (1 minute), widen it for trend analysis (1 hour, 1 day). When you add this to a dashboard alongside your zero-results rate, you can answer: Did zero-results spike because traffic changed or because something broke?

Verify that your search API instrumentation is working

If you're using the reference project, the full setup is:

Then trigger a search:

Wait 5–10 seconds for the BatchSpanProcessor to flush, and then open Kibana → Discover → switch to ES|QL mode and run:

You should see rows with attributes.search.query, attributes.search.result_count, and attributes.search.took_ms.

If no rows appear, check in order:

1. OTEL_EXPORTER_OTLP_ENDPOINT points to the mOTLP endpoint, not your Elasticsearch URL.

2. OTEL_EXPORTER_OTLP_HEADERS includes Authorization=ApiKey <your-key>.

3. OTEL_TRACES_SAMPLER=always_on is set (default sampler may drop spans).

4. Kibana → Observability → APM → Services shows search-analytics-demo (confirms export is working).

Turn ES|QL results into Kibana visualizations

The bar chart that Discover auto-generates from your ES|QL results is a good start, but you can customize it. Click the pencil icon in the top-right corner of the chart to open the inline Lens editor.

From here you can:

• Change chart type (bar, line, area, pie, table, metric).

• Adjust axes and add breakdown dimensions.

• Save the visualization to a dashboard.

This is the path from ad hoc ES|QL exploration to a persistent dashboard panel. You don't need to build visualizations from scratch; Discover and Lens handle the chart rendering from your query results.

Lens is Elastic's drag-and-drop visualization editor, and it's more capable than this quick workflow suggests. You can build multilayer charts, combine metrics with breakdowns, add reference lines, and design full dashboards that mix ES|QL panels with traditional aggregation-based visualizations. For search analytics, that means you can put top queries, zero-results trends, and latency percentiles side by side in a single view.

To go deeper:

• Lens documentation: A full guide to the visualization editor.

• ES|QL in Lens: Using ES|QL queries as data sources for dashboard panels.

• Kibana Dashboards: Building and sharing operational dashboards.

• Use an Agent built in Kibana to make visualizations for you

We'll build a full search analytics dashboard in a later post.

How sampling affects search analytics accuracy

Most application performance monitoring (APM) configurations sample traces to control costs, capturing 10% or 25% of requests. For application monitoring, that's fine. For search analytics, it's a problem.

If you're sampling at 10%, your "total searches" count is 90% lower than reality. Your zero-results rate is still accurate (it's a ratio), but volume counts are off.

Two approaches:

Approach 1: Configure 100% sampling for search endpoints. Your search API probably handles far fewer requests than your main application, so the data volume increase is manageable. The simplest way is through environment variables:

These are head-based sampling decisions made at the start of each trace. They apply globally to the service, which is fine if your search API is a dedicated service. If search shares a service with other endpoints and you need per-endpoint sampling rules, you can implement a custom Sampler in the OTel SDK that inspects the span name or attributes before deciding.

More sophisticated routing (sampling differently per endpoint, dropping noisy spans, or making decisions after a trace completes [tail-based sampling]) typically involves deploying an OTel Collector (such as the EDOT Collector) as an intermediary between your application and Elastic. That's a valuable architecture pattern, but it's beyond the scope of this post. See the OTel Collector documentation and Elastic's EDOT Deployment for more on collector-based sampling and routing architectures.

Approach 2: Upscale in your queries. If you know the sampling rate, multiply: EVAL estimated_total = total_searches * 10. Ratios and averages stay correct; only absolute counts need adjustment.

For more on sampling strategies generally, see the OTel sampling documentation.

For the queries in this post, we used 100% sampling.

What's next: Adding click tracking to search analytics

The six ES|QL queries in this post answer: What do users search for, what returns nothing, how fast is search, and when does traffic spike? They're all derived from a single instrumentation point: the search span.

But they can't tell you whether users are finding what they need. A search that returns 15 results looks healthy from the server side. But if nobody clicks any of those results, your ranking has a problem.

In the next post, we add click tracking, a second span that captures which result the user clicked and where it appeared in the list. If you've been running the reference project, you already have 62 search spans; the next post builds directly on that data. With searches and clicks linked together, we'll calculate:

• Click-through rate (CTR): What percentage of searches result in a click.

• Mean Reciprocal Rank (MRR): How far down the results users have to scroll.

• Click position distribution: The shape of where users click.

The pattern is the same: You add attributes to spans and query them with ES|QL. You get a richer view of the same data without introducing new infrastructure.

Resources to get started with search analytics on OpenTelemetry

• Reference project: Working code for the entire blog series; clone, configure, run.

• EDOT Python: Elastic distribution of OpenTelemetry for Python.

• OpenTelemetry with Elastic: How to send OTel data to Elastic APM.

• OpenTelemetry Python SDK: Upstream SDK documentation and instrumentation guides.

• ES|QL documentation: Query language reference.

• UBI Standard: Reference schema for search event structure.

This is the second post in a series on search analytics with OpenTelemetry and Elastic. Next up: Measuring search quality: Click tracking, CTR, MRR, and click position analysis.

Context

Elasticsearch is one of the top plugins used with the Grafana UI. Until now, the Elasticsearch plugin only supported Lucene and raw Query DSL for querying. ES|QL is Elastic's modern pipe-based query language, designed for analytics on log, metrics, and trace data. Its intuitive syntax makes it easier to filter, aggregate, and transform data compared to Query DSL or Lucene.

This has been tracked as a community request since 2024: grafana/grafana#81765.

How to enable ES|QL support in Grafana

The feature is behind the elasticsearchESQLQuery feature flag. To turn it on, add the following to your grafana.ini:

Restart Grafana after saving. The feature is available starting with Grafana 13.0.

ES|QL in the Grafana query editor

Once the flag is enabled, the Elasticsearch query editor gains a Query language selector. You can switch between Lucene, Raw DSL, and ES|QL from the same editor panel.

When you select ES|QL, the editor switches to a Monaco-powered code editor, the same engine that powers VS Code. You get syntax highlighting, error highlighting, and basic autocompletion out of the box.

Smart index pre-population makes getting started quick: if an index pattern is configured in your data source settings, clicking into the ES|QL editor for the first time auto-inserts FROM <index>. You can change or delete it freely. If no index is configured, the FROM clause is left blank.

Running your first queries

Count log entries by severity

This is a good first query to confirm ES|QL is working and to get a feel for the syntax.

Run it in the Raw Data panel type to see a table with counts per log level:

Browse the latest errors

WHERE, KEEP, SORT, and LIMIT make it easy to filter down to exactly the fields and rows you care about.

Log volume over time

Use BUCKET to group log counts into hourly intervals. This works well with the Metrics panel type, which can render it as a time series graph.

Top hosts by log activity

Identifying the most active hosts is a common operations task. STATS with BY and LIMIT makes it concise.

The TS command for time series metrics

For metrics data in Time Series Data Streams (TSDS), the TS source command in ES|QL enables metrics analytics and time series aggregation.

Examples:

• RATE(): rate of change over time

• AVG_OVER_TIME(): average value over a sliding window

• INCREASE(): total increase over a period

• DELTA(): difference between first and last value

• LAST_OVER_TIME(): most recent value in a window

The pattern follows a two-level aggregation: an inner function applied per individual time series, then an outer function aggregating across groups (for example, per host or per service).

You can also use AVG_OVER_TIME() to compute the average value of a metric over a sliding window, then split the results by host and 10-minute buckets:

TS also runs queries through the ES|QL vectorized compute engine. Internal benchmarks show performance improvements of an order of magnitude or more compared to equivalent Query DSL queries for TSDS-backed data.

Reference:

• TS command documentation

• Time series aggregation functions

Inline error messages

ES|QL queries run against the /_query HTTP endpoint on Elasticsearch. If your query has a syntax error or references a non-existent field, Elasticsearch returns a structured error response. The plugin surfaces this directly in the query editor as an inline message, so you see exactly what went wrong right in the Grafana UI.

In the example above, host.nam is missing the final e. Elasticsearch catches this as a verification exception and returns the field name that could not be resolved. That message appears inline, right below the query editor.

Technical details

Under the hood, the plugin handles ES|QL and other query types on separate code paths. ES|QL queries go to the /_query endpoint with Content-Type: application/json. Lucene and Query DSL queries continue to use /_msearch with Content-Type: application/x-ndjson.

This separation is intentional: /_query returns a different response shape that the plugin parses independently before passing data to Grafana panels.

Try it out!

That also means this is a good moment to try it and give feedback.

The upstream PR and the original tracking issue are public. If you run into problems or have requests, both are open for comments.

Next steps

• Enable the feature in your Grafana 13.0 instance with elasticsearchESQLQuery = true

• Try the example queries above against your own indices

• For metrics data, give the ES|QL TS command a spin, against an Elasticsearch 9.2 or Serverless data source.

• Read the full ES|QL overview to explore what else the language can do

If you are not yet on Elasticsearch, you can start a free trial at Elastic Cloud.

For a high-level tour of metrics analytics across ES|QL and Discover, see Explore and Analyze Metrics with Ease in Elastic Observability. This post zooms in on the mechanics.

Here is the mental model in five bullets:

• FROM treats every document as an independent row. That is right for events, but metric aggregations often need the time series that each row belongs to.

• TS adds that time series context: it groups and aggregates data points by time series before any other aggregation runs, and enables functions like RATE, AVG_OVER_TIME, and LAST_OVER_TIME.

• A TS | STATS query normally has two aggregation phases. The inner phase reduces samples inside each time series; the outer phase groups and combines those per-series results.

• The default inner aggregation is LAST_OVER_TIME, which is why TS metrics | STATS AVG(cpu_usage) and FROM metrics | STATS AVG(cpu_usage) can return different numbers.

• Use TS to query a time series data stream (TSDS). Use FROM for events and raw document inspection.

What is a time series, really?

A time series is a sequence of (timestamp, value) data points identified by the metric name and a unique set of dimension values.

For example, request_count reported every 30 seconds by host h1 in data center dc1 is one time series. The same metric on host h2 in dc1 is a different time series.

In a time series data stream, every metric document carries an internal _tsid field that uniquely identifies a time series. Samples that share a _tsid belong to the same time series and are stored sequentially, sorted by timestamp.

That storage layout enables efficient per-series aggregations. It also explains why TS only works on time series data streams. Other index modes have no notion of a time series, so the per-series operations TS relies on have no such identifier to attach to. FROM does not support those operations, which is what the next section is about.

Why FROM leaves metrics on the table

Consider a counter named request_count collected every 30 seconds from three hosts.

A counter is a cumulative metric: each sample is the running total since the process started reporting it. For request_count, a value of 1,000 means "this time series has observed 1,000 requests so far", not "1,000 requests happened since the previous sample". Counters reset to zero on process restart, so a sample of 4 right after 1,004 is a fresh count, not negative traffic. The ES|QL RATE function computes the per-second change within a time series and handles resets without glitches.

You want to calculate the total request rate across all hosts, bucketed by 5 minutes.

If you are used to writing ES|QL over event data, you might start with this query:

The chart it produces looks plausible at first: a line that goes up over time. But the number on the y-axis is the sum of every cumulative counter value reported in the bucket. Each host contributes its own running total, repeatedly, once per sample. Because the query uses SUM on those cumulative values, the result is not a rate, it is not the number of requests in the bucket, and it grows without bound even if the application stops receiving requests.

request_count is a monotonically increasing counter, so its raw values represent "how many requests have ever happened on this host", not how many happened in the bucket. The right computation is "how much did this counter increase per second on each host, then sum across hosts." FROM cannot express that operation directly. It can group rows by fields, but it has no built-in notion of "the same time series over time" and no way to ask for the change of a counter within each time series. It also cannot use sliding-window time series functions such as RATE(request_count, 5m), which we will come back to below.

TS was introduced for this purpose, providing a succinct syntax to express time series aggregations:

RATE(request_count) runs per time series and produces a per-second rate that handles counter resets correctly. SUM then adds those rates across hosts.

Two aggregation phases: inner and outer

Every TS | STATS query has two distinct aggregation phases.

Let's make that concrete with a query that calculates the request rate per data center:

The diagram below shows how TS evaluates this query. It first reduces samples inside each time series, then groups and combines those per-series values into one result per datacenter and time bucket.

The phases are:

Inner (within a time series). Runs separately for each time series. It collapses many (timestamp, value) data points within a bucket into a single value per time series per bucket by applying the inner aggregation function, such as RATE in the example above. Functions: RATE, AVG_OVER_TIME, MAX_OVER_TIME, LAST_OVER_TIME, STDDEV_OVER_TIME, and so on. The full list is on the time series aggregation functions page.

Outer (across time series, the "grouping" phase). Combines the per-series values into a single value per group per bucket. Functions: SUM, AVG, MAX, MIN, percentiles, and the rest of the regular ES|QL aggregates.

In SUM(RATE(request_count)) BY datacenter, TBUCKET(5m):

• RATE(request_count) is the inner aggregation. It runs per time series.

• SUM(...) is the outer aggregation. It combines time series within the same datacenter and bucket.

• TBUCKET(5m) defines the bucket boundaries (equivalent to BUCKET(@timestamp, 5m)).

The outer aggregation is optional. If you only need the per-time-series result, use the time series aggregation function directly:

That query keeps the per-series rate for each bucket instead of wrapping it in SUM, AVG, or another aggregate across time series.

The default inner aggregation: LAST_OVER_TIME

TS has to reduce raw samples inside each time series before it can run the outer aggregation. That means every metric field in a TS | STATS aggregation needs an inner aggregation, even when the query does not spell one out.

Consider a metric named cpu_usage. It is a gauge: a metric that captures a value at a point in time and can move up and down freely. A sample of 0.42 means "this host is at 42% CPU at this time". For a gauge, the natural "value in this bucket" is the most recent sample.

That is what ES|QL fills in for you. If you write TS metrics | STATS AVG(cpu_usage) BY host.name, TBUCKET(5m), the implicit inner aggregation is LAST_OVER_TIME(cpu_usage) and the query is equivalent to:

For each time series, LAST_OVER_TIME picks the latest sample in the bucket. Then AVG averages across time series.

It is also why the same-looking query against FROM and TS can return different numbers. FROM averages every individual document. TS averages one value per time series per bucket. If your hosts publish at slightly different rates, those averages diverge. For example, in a five-minute bucket, a host that publishes every second contributes 300 documents while a host that publishes every two minutes contributes only two or three. With FROM | STATS AVG(cpu_usage), the chatty host dominates the average. With TS, each time series is reduced to one bucket value first, so the outer average gives each host one value to contribute.

If you want the average value during the bucket instead of the latest value, make the inner aggregation explicit:

AVG_OVER_TIME averages all CPU utilization samples within each time series. The outer AVG then averages those per-series values across matching hosts. That makes the result sample-weighted within each time series, then equally weighted across time series. Use this when you care about how the value behaved during the bucket, not just where it ended up.

The same rule applies to peaks and troughs. For a peak CPU chart, use MAX(MAX_OVER_TIME(cpu_usage)), not just MAX(cpu_usage). The inner MAX_OVER_TIME finds the peak within each time series; the outer MAX finds the peak across matching time series.

Counters work the other way around. Their sample value is a running total, so the latest sample on its own is rarely meaningful. For a counter, the inner aggregation you almost always want is RATE for a per-second rate, or INCREASE for the total change in the bucket. Falling back on the default LAST_OVER_TIME gives you the most recent cumulative value, which is the trap the FROM query in the previous section walked into.

Pick the inner function deliberately. The outer function is the easy part.

When to use TS, when to use FROM

A practical rule of thumb:

• Use TS for metric aggregations against a time series data stream. It is the source command designed for that data, and it applies per-series semantics by default.

• Use FROM for events: logs, traces, audit records, transactions. Each row is independent. There is no time series context.

FROM still works on TSDS indices and is occasionally useful, for example when you want to inspect raw metric documents without per-series grouping. For dashboards, alerts, and any kind of charting, TS is the right default.

If you first need to discover which metrics or time series exist in the data, use METRICS_INFO or TS_INFO after TS and before STATS. See ES|QL METRICS_INFO and TS_INFO: Catalog your time series data for a deeper walkthrough.

Post-process TS results with ES|QL

The first STATS command is the boundary between time series processing and regular ES|QL processing. Before that first STATS, TS needs to keep the data grouped by _tsid, so commands that change row order or shape are not allowed. After that first STATS, the output is a regular ES|QL table. You can sort it, limit it, join lookup data, enrich it, or compute derived columns.

For example, this query calculates average CPU per host and bucket, finds the maximum bucketed average for each host, and returns the ratio:

Sliding windows for the inner aggregation

Time series aggregation functions accept a second argument: the window size for the inner phase.

This computes the rate over a 5-minute sliding window, but reports a value every minute. It is useful when you want a smoother chart at fine bucket sizes.

The window is the ES|QL counterpart to a PromQL range vector selector: RATE(app.requests, 5m) serves the same purpose as rate(app_requests[5m]).

Gotchas worth knowing

A few things in TS can seem surprising, especially when coming from the events-based FROM mental model. None of these are bugs; most are direct consequences of the per-series model. Here is what to watch for.

COUNT(*) is rejected. Say you want to know how many samples were collected per service in each bucket. The instinct from FROM is COUNT(*), but TS rejects it: there is no plain "row" once data is grouped by time series, so a row count has no defined meaning. Pick what you actually want to count:

• Number of samples per service: STATS samples = SUM(COUNT_OVER_TIME(cpu_usage)) BY service.name, TBUCKET(5m). The inner COUNT_OVER_TIME counts samples per time series; the outer SUM adds them across the time series in the group.

• Number of distinct hosts reporting per service: STATS hosts = COUNT_DISTINCT(host.name) BY service.name, TBUCKET(5m). This counts unique label values across time series.

You cannot sort, limit, lookup join, or enrich before STATS. TS metrics | SORT @timestamp | STATS ... will fail. The grouping by _tsid must happen first, before anything else can run. Filter with WHERE if you need to narrow the scope. After the first STATS, the output is regular ES|QL and you can pipe it through any command, as shown in the previous section.

Gauge vs counter mapping. Time series functions are sensitive to the metric type set in the field mapping. RATE only works on counters; *_OVER_TIME functions are intended for gauges. If you build TSDS mappings by hand, pay special attention to this part.

This can be a source of friction for Prometheus users. Prometheus metric type metadata is not always available in the data Elasticsearch receives, so the metric type may have to be inferred from naming conventions (_total for counters, and so on). Those heuristics are imperfect, and a misclassified metric is rejected by the function that should accept it. The deeper mechanics, including how Prometheus Remote Write maps metric types into TSDS, are covered in How Prometheus Remote Write Ingestion Works in Elasticsearch.

Explicit converter functions (gauge-to-counter and counter-to-gauge) are on the roadmap to make these cases easier to recover from at query time.

Kibana charts go empty when you zoom in too far. In Kibana, TBUCKET adapts to the date picker, so zooming in shrinks the bucket size. When the bucket size drops below the data's collection interval, every other bucket has no sample, RATE and the rest return null, and the chart silently goes blank. Elastic is evaluating mitigations such as a runtime warning when the bucket size is too small, a configurable minimum bucket size, or automatic widening of the window or bucket size.

Wrap up

For metric queries, start with TS unless you specifically need raw documents. Then choose the inner aggregation based on what the value should mean inside each time series: RATE for counters, LAST_OVER_TIME for current gauge values, and explicit *_OVER_TIME functions for peaks, averages, minimum values, or distributions.

Once the per-series value is right, the outer aggregation is the familiar part: group and reduce those time series into the chart, alert, or table you need.

For the full reference, see the TS command docs and the list of time series aggregation functions.

One line: Speed that scales with your data

On billions of documents, analytical queries push against a real efficiency-precision trade-off. We’ve been hard at work pushing back. Our native columnar support is one of the best there are. ES|QL itself is a fast, purpose-built analytical engine, getting ever smarter at aggregation execution. And Elasticsearch ships a steady stream of efficiency innovations, like Block k-dimensional (BKD) tree pruning, with more landing all the time.

But even with all of that, native approximate queries really shine through. Starting in Elasticsearch 9.4, ES|QL supports approximate query execution. All you have to do is add one line to your queries: Prepend SET approximation = true. Now Elasticsearch will automatically sample a subset of your data, run the aggregation on that sample, extrapolate the results, and report confidence intervals. All transparently.

Your existing query stays unchanged. The SET directive tells Elasticsearch to handle the sampling, extrapolation, and statistical validation for you. No query rewriting, no manual sampling math, no guessing at sampling probabilities.

SET approximation = true is a forward-compatible directive. Today, it speeds up the most heavily used aggregations. As we expand support to more capabilities, your existing queries benefit automatically. Queries that aren’t yet approximated run exactly without errors; a warning header explains why.

How much faster?

On the ClickBench benchmark, well-behaved analytical queries ran on average 23x faster with confidence intervals enabled. Individual queries hit ~100x. Disabling confidence interval computation, the highest-leverage queries land near ~300x.

The advantage grows as your datasets grow. Approximate-mode cost is capped by the configured sample size, while exact-execution cost scales with row count. Doubling your index doubles exact-query time but barely changes approximate-query time for the same accuracy! This is a beautiful property of the underlying math, not an engineering trick, and it’s why approximation gets more valuable as you scale.

Speedup also depends on query shape, grouping cardinality, and sample size. See the FAQ for the full set of factors and tuning tips. Read “Fast approximate ES|QL” in two parts ([Part 1], [Part 2]), straight from the creators of the feature.

What you get back

The response includes your original aggregated values, automatically scaled to represent the full dataset; a COUNT on a 1% sample comes back as the estimated total, not the sample count. Column names and types are preserved (backwards-compatible). Plus two additional columns per approximated value:

• Confidence interval: A range that bounds the true value at the configured confidence level (default 90%). For example, a count of 268,473 with interval [264,444–273,179] means you can be 90% confident the true count falls in that range.

• Certified flag: A Boolean indicating whether the confidence interval for that value meets formal statistical guarantees. When certified is true, the data distribution allows us to rely on the results at face value. When false, the approximation is still often close but we can’t claim the same formal guarantees, typically because the distribution may be highly skewed or involve too few documents in a group. Think of it as the difference between "statistically proven" and "best estimate."

This is a deliberate design choice: Consumers that don't care about the confidence metadata can choose to not compute them at all (see “Granular control when you need it”) or ignore the extra columns and use the results exactly as before. Consumers that do care (like AI agents that read the results programmatically) get everything they need without a second query.

Use cases for approximate queries in ES|QL: Where this matters

AI agents and agentic workflows

Approximate queries don’t just speed up agent queries; they enable a scan-then-enhance investigation pattern that wasn’t practical at scale before. An agent can sweep billions of documents in sub-second time, identify candidates, and zoom in for exact answers, all inside a single reasoning loop. The certified flag turns approximation into a decision signal: Proceed at face value when it’s true, escalate to an exact query when it’s false and the step needs a tight guarantee. As ES|QL becomes the foundation for agentic analytics in Elastic, approximation is the speed layer that makes investigation possible at this scale.

Dashboards and charts on large datasets

Dashboards that aggregate weeks or months of data can become sluggish as data volumes grow. With SET approximation = true, the same dashboard loads faster. In the future, Kibana will inject the setting transparently, so users won't need to know it's happening; they’ll just see faster charts.

Log pattern analysis in ES|QL at scale

CATEGORIZE, GROK, and regex-heavy conditions are among the most compute-intensive parts of ES|QL because they require nontrivial compute per document. With approximate execution enabled, these large-scale pattern and exploration workflows become practical on very large indices.

Exploratory analysis and hypothesis testing

When you're exploring data to form hypotheses, for example, "Which services have the highest error rates this week?", you rarely need exact counts. You need shapes, relative magnitudes, and outliers. Approximate mode gives you those at interactive speed, and the confidence intervals tell you when to switch back to exact mode for the final answer.

How approximate queries work in ES|QL, without the math

The speedup is real engineering, not a query-planner trick. Sampling happens at the Lucene layer: Elasticsearch reads only the documents in the sample, so I/O and compute savings are proportional to the sampling rate. The aggregation runs on the sample, and the result is automatically scaled to represent the full dataset.

Confidence intervals are computed by a bootstrap procedure over multiple sub-partitions of the sample: statistically rigorous, not a heuristic or a guess. This is what backs the certified flag: When the methodology’s assumptions are met, the intervals carry formal guarantees.

Granular control when you need it

The defaults are designed to work well out of the box, but you can tune them:

• rows: How many documents to sample (default: 100,000 for ungrouped queries, 1,000,000 for grouped). More rows means higher accuracy and longer runtime.

• confidence_level: The confidence level for intervals. Defaults to: 0.9. Set it to a higher level for an increased probability that the value is within the confidence interval.

• Skip confidence intervals for maximum speed: Set confidence_level to null, and Elasticsearch returns just the point estimates, adding another 2–5x speed on top of approximate execution. This is how the highest-leverage queries land near 300x.

What's next

SET approximation = true is a forward-compatible directive. As we add support for FORK, JOIN, chained STATS, and additional aggregations, your existing queries automatically benefit.

Future work also includes tighter integration with Kibana so dashboards and Discover can enable approximation automatically and improved handling of highly skewed grouping fields.

Additionally, we’ll make approximate queries natively accessible to agents, so they can opt into fast execution as part of their analytics tools and reasoning loop.

Get started

Approximate queries are available in Elasticsearch 9.4 as a technical preview on the Enterprise subscription tier. Add SET approximation = true; to the beginning of your query, and see the difference. Check the ES|QL SET command reference for configuration options.

FAQ

What is approximate query execution in Elasticsearch?

Approximate query execution is a mode where Elasticsearch samples a subset of your data, runs the aggregation on the sample, and extrapolates the result to represent the full dataset. You get back the estimated value plus a confidence interval showing how much to trust it. It's controlled by a single SET directive prepended to your existing ES|QL query; no query rewriting required.

How do I speed up ES|QL aggregations without reducing my data retention?

Just add SET approximation = true to your query. Approximate execution samples at query time, not at index time. Your data stays fully indexed, fully retained, and queryable both exactly and approximately. Elasticsearch handles sampling and extrapolation on the fly. Drop the directive any time you want exact results; nothing about the underlying data changes.

How much faster are approximate queries?

On the ClickBench benchmark, aggregation-heavy ES|QL queries that are well-suited to sampling typically run 10–40x faster with confidence intervals enabled, with individual queries reaching 100x or more. Disabling confidence interval computation (SET approximation = {"confidence_level": null}) adds another 2–5x on top, so the highest-leverage queries hit nearly 300x. The advantage grows with dataset size: Sampling cost is capped by the configured sample size, while exact execution cost scales with the row count, so the bigger your index, the bigger the win for the same precision.

How accurate are approximate queries? Can I trust the results?

Each approximated value comes back with two signals: a confidence interval (a range bounding the true value at a configurable confidence level) and a certified Boolean flag. When certified is true, the confidence interval carries formal statistical guarantees. When false, the result is still often close, but the data distribution didn't meet the assumptions required for a formal guarantee. Accuracy depends on data characteristics and query shape, not on document count, so speedup gains increase as your dataset grows.

What does the speedup depend on?

Five main factors:

• Dataset size. Larger datasets produce larger speedups, for the reason described above (exact scans grow with N; sampled scans don’t).

• Query shape. Queries that scan a lot to compute relatively little (large STATS, especially MEDIAN and PERCENTILE) benefit most. Queries that are already cheap (small WHERE filters matching few rows, or simple counts that hit indexed summary statistics) see little speedup.

• Grouping cardinality and distribution. Well-distributed BY fields with healthy per-group sample counts benefit cleanly. Very sparse or highly skewed grouping (for example, a near-unique field or a long tail of rare values) can erode the gain because rare groups end up with too few sampled documents.

• Confidence interval computation. Computing intervals adds overhead. Set confidence_level to null, and you trade interval reporting for an additional 2–5x speedup.

• Sample size. The defaults (100k for ungrouped STATS, 1M for STATS … BY) work well for most queries. Increasing rows improves accuracy on high-cardinality grouping at the cost of some speedup; decreasing it does the reverse.

Can I use approximate queries for log analysis and pattern detection?

Yes. CATEGORIZE, GROK, and regex-heavy conditions are among the most compute-intensive operations in ES|QL because they require per-document processing. With SET approximation = true, these operations run on a sampled subset instead of the full index, making large-scale log pattern analysis and exploration fast on very large datasets.

Do I have to rewrite my ES|QL queries to use approximate mode?

No. Prepend SET approximation = true to your existing query. The aggregation expressions, column names, and output types stay the same. The response adds two columns per approximated value (the confidence interval and the certified flag), but existing consumers that don't use those columns see no breaking change.

What aggregations does approximate mode support in 9.4?

COUNT, SUM, AVG, WEIGHTED_AVG, MEDIAN, PERCENTILE (except extremes), MEDIAN_ABSOLUTE_DEVIATION, and STD_DEV (with caveats for highly skewed distributions). More coverage on the way.

Will I get the same result twice for the same query?

Not exactly. Approximate execution randomly samples documents at query time, so successive runs of the same query return slightly different point estimates and confidence intervals. The variation between runs is small relative to the confidence interval each run reports. If you need bit-for-bit reproducibility, run the exact query. For dashboards, depending on the use case, the variation can typically be smaller than the visual resolution of the chart.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Just send your OTel metrics to the Elasticsearch OTLP/HTTP endpoint and they're stored using the new exponential_histogram type and queryable immediately. Already have historical data stored in the classic histogram type? A simple ::exponential_histogram cast in your ES|QL queries handles the migration transparently. Already using downsampling? Both histogram field types are now fully supported.

Histogram metrics

When dealing with metrics (in OpenTelemetry or Prometheus, for instance), counters and gauges are the most common metric types. Gauges allow you to monitor values that rise or fall (e.g., CPU utilization). Counters allow you to, well, count things, such as the total number of HTTP requests your service is handling. Counters normally just increase in value, with a few exceptions when they reset, like when a server reboots.

In the case of counters, you can additionally collect a counter measuring the total sum of your HTTP response times, which allows you to derive the average response time by dividing that sum by the total number of requests. However, average response times provide limited insights into the collected data and the system behavior. The best insights are gained by analyzing the collected metric distribution, e.g., through median and percentile calculations. This is where counters fall short.

In the past, workarounds have been applied: For example, classic Prometheus-style histograms attempt to capture the distribution using a set of counters. By defining fixed buckets (e.g., one for response times in the range [0s, 1s), one for [1s, 4s), and so on) and associating a counter with each, we can at least estimate percentiles broadly. However, the key problem here is that we have to know the distribution of our data up front to properly define these buckets.

To that end, the OpenTelemetry community has come up with a better solution: exponential histograms. Exponential histograms assign collected values to buckets, just like classic Prometheus-style histograms. The key differentiator is that these buckets vary dynamically based on the collected values. The name "exponential" comes from the fact that the bucket sizes increase exponentially: we use small buckets for small values and wider buckets for larger values. You can find an excellent introduction in the OpenTelemetry exponential histograms introduction.

Note that in addition to classic histograms, Prometheus also added native histograms, which directly map to OTel exponential histograms. Native histograms have their own PromQL syntax. We are actively working on adding support for that syntax to the Elasticsearch PromQL implementation, so that you can directly query exponential histograms using PromQL.

Demo setup

Let's start by collecting some histogram metrics to show how they can be stored and analyzed in Elasticsearch using ES|QL.

We'll focus on a Java JVM metric: garbage collection durations. OpenTelemetry defines the jvm.gc.duration, which is a histogram-typed metric. The OpenTelemetry Java agent natively supports collecting this metric.

We'll spin up a JVM running a Renaissance benchmark to put it under stress. We'll start that JVM with the vanilla OpenTelemetry Java agent attached and have it send the metrics directly to Elasticsearch.

You can find the ready-to-run Docker-compose file here. You'll just need to insert your Elasticsearch OTLP/HTTP endpoint and API key in the docker-compose.yml:

Note that you don't have to use this demo setup. We even encourage you to try it with your own application. Here are the other important OpenTelemetry agent settings the demo already includes, which you should include too if you're bringing your own app:

Let's step through them:

• Temporality preference: OpenTelemetry supports both cumulative and delta-based histograms. Cumulative means that the histogram is only cleared after an application restart, while delta clears it after each export. At the time of writing, Elasticsearch only supports delta temporality for histograms. We are actively working on supporting cumulative histograms as well.

• Default Histogram Aggregation: By default, OpenTelemetry exports histograms in the Prometheus-style fixed bucket format. Since we want to reap the benefits of exponential histograms, we tell the agent to use them instead.

• Runtime Telemetry enabled: This tells the agent to actually collect the detailed JVM metrics, which include jvm.gc.duration.

Now we are ready to go! We'll let the application run in the background and switch over to Kibana to analyze the GC metric.

Querying with ES|QL

Now let's open up Kibana and navigate to "Discover". There we'll switch to ES|QL mode, and start querying the collected data:

As a response, we now see the metric panel shown below. If you don't see any data, make sure to double-check the Kibana time range filter.

This number represents the total number of garbage collection operations that happened in our test application during the selected time frame.

Similarly, we can query the total time spent on those garbage collection operations:

So we have roughly 270k garbage collections, which in total took 713 seconds. Given these two numbers, we can now compute the average if we are still fluent in primary school-level math. Even if not, you can just let ES|QL do that for you:

Now we know that the average garbage collection operation took about 3 milliseconds. However, Java experts might know that there are different kinds of garbage collections happening, which can have significantly different pause times. Fortunately the OpenTelemetry metric comes with attributes, which allow us to slice the data accordingly:

As expected, major garbage collections take a lot more time per collection than minor ones, at least on average. So far, we have done nothing you couldn't also achieve by just using counters. Let's now use histograms to understand the actual distribution of the GC latency. We'll look at the data over time (by grouping using TBUCKET) and focus on the major garbage collections:

The graph now shows us the minimum, maximum, median and 99th percentile for major garbage collections. Note that we aren't bound to only querying the median and the 99th percentile. We can query any percentile we'd like to see, as these are estimated at query time from the raw exponential histograms.

A note on backwards compatibility

So far, we have seen how you can use the new shiny toy in Elasticsearch and ES|QL: exponential histograms. However, since this has just reached general availability (GA) in the 9.4 release, what about your historical data?

Before exponential histograms were added, Elasticsearch was already capable of storing OpenTelemetry histograms in the histogram field type. To do so, we converted them to a different data structure supported by the histogram field type: T-Digest. T-Digest provides good accuracy for extreme percentiles (e.g., 99th percentile) at the cost of accuracy for percentiles in the middle of the distribution, such as the median. In contrast, exponential histograms provide a guaranteed upper bound on the relative error for every percentile. As conversions always introduce errors, we are happy to now have native support for exponential histograms, allowing you to collect and analyze your metrics end-to-end without unnecessary conversions.

But still, what should you do if you have historical data and still want to query it? Thanks to ES|QL union types, the answer is actually easy: You just have to add a ::exponential_histogram suffix to the histogram metrics in your queries:

When this query encounters histogram fields, it will attempt to convert them to exponential histograms. When operating on exponential_histogram fields, the ::exponential_histogram cast has no effect. Note that this also works with mixed data sets: if your backing indices use both types, the query will just do the right thing.

So if you are building queries or dashboards that you expect to run on pre-9.4 ingested data, we recommend that you simply add: ::exponential_histogram casts.

Wrapping up

Native support for OpenTelemetry exponential histograms in Elasticsearch gives you better metric fidelity and more flexible analysis in ES|QL. In this blog post, we have shown you how to easily ingest and analyze your histogram metrics with ES|QL using various aggregations and the impact exponential histograms have.

Exponential histograms are generally available in Elasticsearch basic starting with the 9.4.0 release. They will be available in Elastic Cloud Serverless a few weeks after the 9.4.0 release, once mOTLP (the managed observability OTLP intake) switches to use the Elasticsearch OTLP endpoint. We'll update this blog post and add a note on the Elastic Cloud Serverless release notes when that happens.

The problem: Heterogeneous data, one query

Consider a production incident investigation. Errors are spread across three microservices: an API gateway, a payments service, and an auth service, each with different field names and different conventions. Before subqueries, combining them in a single ES|QL query meant cramming everything into one FROM with CASE chains:

This is brittle and slow. The disjunctive OR prevents predicate pushdown; every index scans every condition. Every CASE chain grows with every source. Copy it into five dashboards and three alert rules, and you have eight places to update when anything changes.

The fix: Independent pipelines

Subqueries replace the monolithic FROM + CASE pattern. Each data source gets its own complete pipeline:

The gateway branch only scans for HTTP 500s. The payments branch only looks at transaction statuses. The auth branch only checks login failures. Because each branch has its own WHERE, the optimizer pushes filters independently into each index, restoring the predicate pushdown that a single FROM with OR conditions prevents. Fields that exist in one branch but not another are filled with null.

Adding a fourth service means adding a fourth branch. Existing branches don't change.

Save it as a view

This is where subqueries and logical views combine. Wrap the subquery above in a named view, with one API call:

Now consumers just write FROM error_triage | STATS error_count = COUNT(*) BY service. Three indices, three pipelines, one name. If you have 10 dashboards and five alert rules consuming this pattern, that's 15 copies of the same logic today; with a view, it's one definition and zero consumer-side edits when you add a fourth service. See Elasticsearch ES|QL Views for the full views deep dive.

What you can do inside a branch

Each branch supports the full ES|QL pipeline: WHERE, EVAL, STATS, LOOKUP JOIN, ENRICH, and more. See the subquery documentation for the complete list.

Aggregate different metrics, and then combine

Each branch can compute its own summary before results are merged. This is useful when different indices track the same concept under different field names:

Both branches produce avg_latency and hour, but each computes it from a different source field. The combined result is a single table you can chart or alert on, without normalizing field names at ingest time. This pattern is impossible with a single FROM; you can't compute different aggregations per index without subqueries.

Subqueries vs. FORK

ES|QL also has FORK (now generally available), which creates parallel execution branches from the same input. The distinction:

Different indices → subqueries. Same data, different analyses → FORK.

How this compares

If you're coming from other query languages, here's how ES|QL subqueries stack up at the time of writing:

Splunk SPL/SPL2 has append and multisearch in classic SPL, and SPL2 adds a union command that merges events from multiple datasets (the closest analogue to ES|QL subqueries). Federated Search extends this across remote Splunk deployments (analogous to CCS). The differences are in how the engine handles each branch: ES|QL subqueries give each branch independent predicate pushdown, meaning filters are pushed into each index's shard-level structures separately. SPL2 union merges datasets but optimization across branches is limited to what the search scheduler can parallelize. Wrapping ES|QL subqueries in a view gives you engine-level encapsulation with role-based access control (RBAC); Splunk's equivalent is saved searches and macros, which are text substitution expanded at parse time.

SQL databases have UNION ALL, which is the closest analog. The difference is that SQL UNION ALL typically requires matching column counts and types at parse time. ES|QL subqueries are more forgiving; columns that exist in one branch but not another get null-padded automatically, which matters when your sources have different schemas (the norm in observability data). SQL views solve the reuse problem similarly, but ES|QL views are cluster-level objects, not database-scoped; they work across cross-cluster search boundaries.

Grafana / Datadog / other dashboarding tools handle multisource composition at the visualization layer: Run separate queries, merge in the panel. This works for display but breaks for alerting, downstream queries, and anything that needs a single result set programmatically. ES|QL subqueries push the composition into the engine, so alerts, views, and API consumers all get the same unified result.

Capability

Splunk SPL/SPL2

SQL UNION ALL

Dashboard-layer merge

ES|QL subqueries

Independent filters per source

SPL2 `union` merges datasets; optimization is scheduler-level

Yes

N/A (separate queries)

Yes; parallel with pushdown

Schema mismatch handling

Manual field normalization

Strict column matching

Manual in panel config

Automatic null-padding

Engine-level reuse

Text macros (parse-time expansion)

Database-scoped views

Dashboard variables

Cluster-level views with RBAC

Works for alerts + API

Limited (summary indexing)

Yes

No; display only

Yes

Add a source

Edit every macro/saved search

Add a UNION branch

Add a panel query

Add a branch; existing branches unchanged

Current constraints

In the Tech Preview release, subqueries are non-correlated; branches run independently and can't reference the outer query. They're supported in FROM only (not TS), and FORK can't be used inside or after subqueries. See the subquery documentation for details.

What's next for subqueries

WHERE subqueries — WHERE field IN (FROM other_index | ...) and other correlated forms — will extend the composition model from FROM into filtering. This brings the familiar SQL pattern of nested filtering to ES|QL.

Try it

Subqueries in FROM are available as a Tech Preview. Try them in Kibana Dev Tools or Discover. We'd love your feedback; file a GitHub issue with the ES|QL label.

ES|QL subqueries in FROM are a Tech Preview feature. Tech Preview features are subject to change and are not covered by the support SLA of GA features. The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Views don't store data; they re-execute on every read, so results always reflect the current data and the current definition. If you've used views in SQL databases, this will feel familiar. The difference: ES|QL views are engine-level virtual indices stored at the Elasticsearch cluster level, not saved query text that gets expanded client-side. They appear in Kibana autocomplete, support cross-cluster search (CCS), and are governed by dedicated role-based access control (RBAC) privileges.

A simple view

A view can wrap any ES|QL query. Start with a straightforward filter — HTTP 500 errors from the API gateway:

Now anyone can write FROM error_triage without knowing the index pattern or filter condition:

The query is defined once. Consumers reference a name.

Views support full create, read, list, update, and delete (CRUD) via the _query/view REST API.

Update propagation

Say the team decides error_triage should also capture client errors, not just 500s. Update the definition in place:

Every dashboard panel, alert rule, and ad-hoc query using FROM error_triage immediately reflects the broader filter. No saved objects to hunt down. No stale copies. Change once, update everywhere.

Nested views

Views can reference other views, enabling layered abstractions. Create views for suspicious IPs and threat intelligence, and then compose them:

Security teams query FROM security_overview without knowing the underlying data model. They're also shielded from any changes made to suspicious_ips by its owner; the abstraction boundary is real, not syntactic.

Multisource views with subqueries

A view can wrap any ES|QL query, including multisource compositions, using subqueries in FROM. Each subquery branch queries one service independently (its own filters, its own field normalization), and the results combine automatically:

Consumers just write:

Two indices, two independent pipelines, one name. To add a third service later, add a third branch; existing branches don't change, and every downstream dashboard and alert reflects the update automatically. For a deep dive on subquery syntax and what you can do inside each branch, see Three Indices Walk Into a FROM Clause.

How views work under the hood

When you write FROM view_name, ES|QL resolves the view's stored query and executes it inline. Views are re-executed on every read, so results always reflect the current data and the current definition.

Views share a namespace with indices, aliases, and data streams. A view cannot have the same name as any of these (enforced at creation time). This keeps FROM my_name unambiguous regardless of whether the name resolves to a view, an index, or an alias.

Security model

Views are governed by four dedicated RBAC privileges: create_view, read_view_metadata, delete_view, and manage_view. Elasticsearch checks the privileges of the user running the query (invoker security), not the user who defined the view. The user querying a view needs permissions on both the view and its underlying indices.

Kibana integration

Views appear in Discover's ES|QL editor autocomplete alongside indices. ES|QL-based dashboard panels work with views transparently. In the initial Tech Preview release, view management is API-only. A Kibana UI for creating and managing views is planned.

Cross-cluster search

A view's definition can reference remote indices using CCS syntax:

Consumers query FROM cross_cluster_errors without knowing which clusters are involved.

Current constraints

In the Tech Preview release, view management is API-only and SET directives can't appear inside view definitions; the caller applies them when querying. Subquery-based views can't be nested inside other multisource FROM expressions. See the views documentation for the full list.

What's next for views

Views today are always fresh; they re-execute on read. Materialized views flip that tradeoff: Pre-compute once, read instantly. Think pre-aggregated rollup views for Service Level Agreement (SLA) dashboards that load in milliseconds instead of scanning raw data on every refresh. A Kibana CRUD UI for views, including a "Save as View" workflow in Discover, is also planned.

Try it

Logical views are available as a Tech Preview. Try them in Kibana Dev Tools or Discover. We'd love your feedback; file a GitHub issue with the ES|QL label.

ES|QL logical views are a Tech Preview feature. Tech Preview features are subject to change and are not covered by the support SLA of GA features. The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Learn how to get started with practical examples you can use right away.

Elasticsearch ES|QL query builder for JavaScript and TypeScript

If you've ever built an ES|QL query in JavaScript, you've probably written something like this:

It looks fine until hostname isO'Brien's server and the whole thing blows up with a parse error. Or until a user passes "; DROP INDEX logs into a search field and you realize you've been building queries with raw string concatenation this entire time.

There's a better way. The ES|QL query builder for JavaScript and TypeScript lets you write queries like this instead:

Values are escaped automatically. You get autocomplete in your editor. And you can see exactly what the query does, without mentally parsing a template literal.

ES|QL query builders are already available across Elastic's language clients, including Python, Ruby, and others. This article focuses on the JavaScript and TypeScript version, walking through practical examples you can start using today.

Getting started

Install the package:

Here’s a minimal query:

This renders:

To run it against Elasticsearch:

That’s it. No string interpolation, no manual escaping.

Building a real query, step by step

Let's walk through a realistic scenario: You're building a dashboard that analyzes web server error logs. We'll start simple and layer on features.

Step 1: Filter error logs

Step 2: Add a computed column

Your timestamps are in milliseconds, but you want response time in seconds:

Step 3: Aggregate errors by status code

The f namespace gives you access to 150+ ES|QL function wrappers: aggregations, string functions, date functions, math, geo, and more. They all return chainable expressions, so you can use them anywhere you'd use E().

Step 4: Use date functions for time-based analysis

Step 5: Branch queries safely

Every method returns a new query object. The original is never mutated. This means you can build a base query and branch it for different views:

Three different queries, one shared base. Change the filter on base, and all three update. This is especially useful for dashboards where multiple panels query the same dataset with different aggregations.

Three ways to write expressions

The domain‑specific language (DSL) gives you flexibility in how you write conditions. Here's the same WHERE clause written three different ways:

Raw strings: When you're writing a quick one-off:

The E() expression builder: When you want type safety and autocomplete:

The esql template tag: -When you want safe interpolation of dynamic values:

All three produce the same ES|QL. Pick whichever fits your situation: raw strings for simple cases, E() when building expressions programmatically, and the template tag when mixing literal ES|QL with dynamic values.

Keeping queries safe

If any part of your query comes from user input, you need to think about injection. ES|QL supports parameter binding, and the DSL makes it straightforward:

The ? placeholder is replaced server-side by Elasticsearch, so the user's input never touches the query string. No escaping, no injection risk.

Beyond the basics

Once you're comfortable with the core commands, the DSL supports every advanced ES|QL feature:

Hybrid search with FORK and FUSE:

Data enrichment:

Conditional aggregation:

AI/machine learning (ML) integration:

For the full list of commands and functions, check out the ES|QL query builder documentation.

What's next

This is the initial release of @elastic/elasticsearch-esql-dsl. You can find the package on npm, explore the source on GitHub, and read the full documentation in the repository. If you run into issues or have feature requests, open an issue; we're actively developing this and want to build what JavaScript and TypeScript developers actually need.

In order for you to be able to trust our estimates, we provide error estimates. Furthermore, since there are edge cases in error estimation, we certify when the estimated value and error are trustworthy. In this blog post, we’ll dive into the theory for approximating and estimating the error in such queries, as well as discuss the testing we’ve done.

Background

In order to estimate ES|QL STATS queries efficiently, we make use of a property that’s shared by many statistics: Their estimates computed from a large number of independent samples from a dataset approach their true value. In the case of an index with some field we can think of the true value of a statistic as its value computed for a random variable with uniform discrete distribution on . In the following we denote this quantity ; it can be things like AVG, MEDIAN, and so on. If we make independent draws from , denoted , such that each value is selected with probability , we have independent copies of this random variable. The property we rely on means that a sample statistic value computed from approaches as becomes large. For example, if is the mean of some metric values then as becomes large. Indeed, for many statistics the limiting error distribution is known to be normal. Furthermore, it only depends on the distribution of , the size of the sample and the type of the aggregation . This means supported STATS queries can be approximated with fixed accuracy independent of the index size .

It is easy to pick values at random from a Lucene index: create a filter that takes exponentially distributed jumps through the dataset, where the expected jump size is controlled by the desired sample probability. The AND of this filter and any other Lucene query can be performed extremely efficiently, since AND’ing filter queries is one of the things for which it is well optimized. In our other post, we discussed some real-world query examples to give a sense of the speedup we obtain for different levels of accuracy.

So far, we've only discussed obtaining an estimate of a query. While such a point estimator can be useful, without knowing anything about its error those uses are limited. We found that ES|QL has existing capabilities that make it relatively easy to incorporate cheap, flexible, and accurate error estimation at the same time. We'll discuss this next.

Error estimates

We view providing an accurate understanding of the uncertainty in our estimates as crucial for users to be able to trust the approximation. While having the option to quickly estimate an ES|QL query alone can be useful in certain situations, we wanted to provide a richer feature that allows clients to make intelligent choices. For example, if an approximate query is being used to preview a chart and the error is only a couple of pixels, there’s little point in running another expensive query to redraw it.

The way we've chosen to represent error is by a confidence interval: the -central confidence interval, to be precise. This can be expressed in terms of the cumulative density, , of the statistic being estimated. Specifically, it's the interval which contains the true value of the statistic with probability whose endpoints are and . Confidence interval calculations are surprisingly subtle. There are also important constraints for our use case that make standard approaches undesirable. Next, we’ll take a look in more detail at the motivation and the design for the approach we’ve adopted.

A key requirement of the whole project is to dramatically accelerate expensive analytics queries. It’s therefore vital that the overhead of estimating uncertainty isn’t too large compared to estimating the query result itself. We also want the feature to be as general as possible, but “isolated” within the language. In other words, ES|QL is a flexible language, and we want estimation to work with as much of it as possible. At the same time, we don’t want to introduce a cross-cutting feature that incurs development costs on every new feature we ship.

With these considerations in mind, we chose to estimate confidence intervals by partitioning the sample set and computing the query output on each subsample. This is reminiscent of bootstrap; however, since we ensure that each partition receives a disjoint random subset of the sample data, we know that they comprise true estimates of the statistic distribution. To achieve the best possible estimate of the statistic itself, we still compute its value on the full sample. For example, to estimate the mean and its distribution the process can be expressed as follows:

This introduces a complication to account for the discrepancy between the count of values used to estimate a query statistic and used to sample its distribution. This is a downside; however, there are some significant advantages.

Most of the work in analytic queries resides in computing the aggregate statistics: post-processing after a STATS reduction acts on a far smaller table, and the cost is often relatively small. In this scheme, every row in the input data to the STATS command is processed exactly twice compared to just estimating the statistic. Therefore, roughly speaking we pay a fixed overhead that's the same order of magnitude as the cost of estimating the query in order to estimate its uncertainty. Since we often achieve multiple orders of magnitude speedup on the exact query, this is acceptable.

Because this process uses a plain old table, with extra columns for the distribution samples, we can pass the whole table through any ES|QL pipeline and compute confidence intervals on the final results. For example, if we include EVAL square_avg = avg * avg in the pipeline above, we'd have exactly the same square_avg, square_avg_0, …, square_avg_B-1 extra values. At the end of the pipeline, we have samples from the distribution of the original statistics and all quantities that are computed using them. Therefore, we can apply our standard confidence interval machinery to reduce the table and convert samples into confidence intervals for derived quantities as well. This whole process is essentially transparent to the rest of the ES|QL language, and as we showed above, can be achieved by query rewriting.

The confidence interval calculation

We have independent samples of the statistic distribution . However, they're computed with fewer values than our estimate . We also have a relatively small number of distribution samples, to avoid the count discrepancy being too large, and so we don’t inflate the table too much. We therefore prefer a parametric approach for estimating confidence intervals.

The errors in the statistics for which we support estimation tend to normal distributions in the limit they're computed from many values. So a natural choice, the standard interval, is to estimate the mean and standard deviation from the samples and report the corresponding normal confidence intervals . Here, denotes the standard normal distribution function. For heavy-tailed data and statistical functions that are sensitive to outliers, such as STD_DEV, convergence to normality can be slow, resulting in poorly calibrated intervals.

Briefly, in order to assess the quality of the intervals, one can examine their calibration. Specifically, one computes a quantity called the coverage. For a central confidence interval, it should contain the true statistic value roughly times for trials. In fact, since we seek the central confidence interval, we can make the stronger statement that the true value should be above, or below, the confidence interval endpoints in roughly out trials. The empirical coverage is this fraction computed for a large number of trials. It allows us to compare alternative approaches by simulation. We return to this when we report our test results.

In order to obtain better confidence intervals, we tried a couple of different approaches: the Cornish-Fisher correction of quantiles and an adaptation of bias-corrected accelerated (BCa) confidence intervals. Simulation showed BCa provided more robust calibration across a range of confidences, so this is the approach we selected. The basic idea, which was introduced by Efron, is to assume that there exists a monotonic transformation of the underlying statistic which, when applied to a distribution sample normalizes its distribution:

Here, , and is the standard normal random variable. This is clearly a relaxation of the assumption that the statistic itself is normally distributed, which is used to derive the standard interval. In fact, this family includes many distributions, since is only constrained to be monotonic. (You can think of as a first-order Taylor expansion of the case that the variance is an arbitrary function of the true parameter value. This further relaxes the assumption that the normalizing transformation also stabilizes the variance.) The nice thing about this ansatz is that never needs to be explicitly computed, and there exist standard approaches for estimating the parameters and from the distribution samples.

To handle one simply arranges for the estimate to land at the median of transformed distribution. If we assume the cumulative distribution function in theta space is then , where is the estimated statistic value, and as before is the standard normal distribution function. Typically, is approximated by the empirical distribution function, computed indirectly by bootstrap. However, somewhat surprisingly, extensive simulation showed that we obtained better calibrated intervals using a normal approximation to our sample values, i.e. with and their empirical mean and standard deviation, respectively.

To complete the procedure, one can rearrange (1) to derive quantiles for as follows:

where is the standard normal z-score for quantile . Typically, one uses the inverse empirical cumulative density estimate of to convert quantiles back to a confidence interval. However, because we have a mismatch between the count of values used to compute distribution samples and the query estimate, we need to do some sort of scaling. Exploring options by simulation, we again found it best to use a normal approximation, , where is the number of distribution samples we use. This is just applying the usual scaling of variance by .

Efron showed that in the case is distributed as , i.e. that it depends only on the true value , then the acceleration can be estimated without any knowledge of . In particular, . By assumption, our statistics tend to normal distributions with mean . Since skew is translation and scale invariant, this gives that , i.e. one sixth of the skew of our distribution samples. One thing this glosses over is the dependence of skew, and therefore acceleration, on sample size. We know it tends to zero as the count increases. In fact, skew also asymptotes to zero as and so we also adjust acceleration to be to account for the count mismatch between the samples and estimate .

Although we significantly improve the calibration of confidence intervals by using a better methodology, we still see issues in the case that the underlying distribution has very heavy tails for some of the supported STATS functions. Therefore, we introduce some additional guard rails we discuss next.

Guard rails

To avoid the user having to understand too much about edge cases, we provide additional safeguards that surface when we've been unable to confirm that the distribution samples behave as we expect. This typically happens when the statistic isn’t computed from a sufficient number of values given the metric distribution. It's exacerbated by very skewed metric data and certain aggregation functions, such as the STD_DEV, which are sensitive to outliers.

We have some global constraints on the minimum count of values used to estimate a statistic for which we'll certify it. For example, if any bucket is empty, then we can’t rely on the distribution samples. This is because ES|QL allows mixing approximate statistics, which treat empty buckets differently. For example, consider the following query:

There is no self-contained way of correctly assigning a value to mix for empty buckets, since summing requires that we treat them as zero, in which case we bias our estimate of avg. Alternatively, ignoring empty buckets introduces bias in the sum. There is also a global minimum count of values for which we’ve verified our certification method is sufficiently reliable; this is 10.

We explored a variety of additional tests to certify the results. These were based on both tests of the underlying data distribution, specifically Hill’s estimator, as well as the statistic’s distribution properties. If the true distribution of the statistic is sufficiently normal, then our estimate and confidence interval calculation behaves as we expect: The interval is well calibrated and the interval width is representative of the actual error. Therefore, in the end, we chose to use a test based on the p-value for distribution samples’ skewness and kurtosis versus a normal distribution null hypothesis. To certify a result, we require that the two tail p-values are greater than 0.05 for both tests. As we show below, we found this test was well aligned to our actual needs: to distinguish results for which the estimate and its confidence interval are more and less reliable.

There's a simple trick we can use to boost the accuracy of the accuracy of the test: Create multiple independent distribution samples and use a vote. Given a test to certify results with a failure rate , the distribution of the count of failures for tests is for the case the null hypothesis, that the estimate is trustworthy, is true. For example, for the majority vote assuming and then the significance of the test is , i.e. we fail to certify fewer than 1% of trustworthy results. Note that we can compute multiple trials relatively easily using different seeds for the RANDOM bucket identifier.

This additional check allows us to certify that we trust our estimates and their errors. We surface this information in the approximate query results. When we can’t certify results, they won’t necessarily be inaccurate, but they should be treated with more caution.

Testing

The two main aims of the testing we discuss here were to understand the calibration of the confidence intervals and to see how well they characterize the statistics' estimation errors. The count function is particularly well behaved, its error distribution is binomial, so the majority of our testing focused on metric aggregations. We study smooth distributions but make sure we cover a range of tail behaviors. The presence of outliers is the key factor that reduces the accuracy of estimated statistics. For example, if an outlier isn’t sampled at all, it can significantly affect the value of some statistics.

We explored a range of light-tailed distributions, such as uniform and normal, and skewed and heavy-tailed distributions, such as exponential, log-normal, Cauchy, and Pareto. For each family of distribution, we used multiple parameterizations, focusing primarily on varying the scale parameter. In total, we had 24 distinct data distributions. Figure 1 shows some example sample distributions from this set. Note that we’ve truncated the charts to remove extreme outliers, which are present for both the Cauchy and log-normal distributions.

For each data distribution, we evaluated 14 different sample sizes, ranging from 1000 to 500000. Then, for each sample set, we evaluated AVG, COUNT, MEDIAN_ABSOLUTE_DEVIATION, MEDIAN, PERCENTILE([25, 75, 90, 95, 99]), SUM and STD_DEV at two levels of confidence, 50% and 90%. In total, we have around 7500 distinct experiments. For each experiment, we assessed the interval calibration using 100 runs and counting the number of times the true statistic lands in the confidence interval. This gives us a binomially distributed estimate for the true confidence interval coverage. The variation we expect in the estimated coverage changes slightly with the level of confidence; for example, at 50% we expect to see values mainly between 0.44 and 0.56, and for 90% we expect to see values mainly between 0.86 and 0.94 using 100 trials.

Figure 2 shows box plots for the empirical coverage for the two confidence levels computed from all experiments. In all cases, the confidence intervals are reasonably well calibrated. Extreme percentiles are biased for small sample sizes, which leads to increased outlier counts for small sample sizes. As a rule of thumb, you’d want roughly samples to ensure that you have enough samples in the appropriate tail.

Next, we examine the degree to which the confidence intervals capture the typical size of the estimate error. To do this, we examine the distribution of the ratio of the estimated statistics' error and half the confidence interval width for all certified results. The higher the confidence, the wider the interval, so different confidence levels shift the mean of this distribution. Figure 3 shows this distribution computed for the 90% confidence interval. As expected, the distribution is roughly normal, albeit with a tail of some larger errors. We see in all cases the confidence interval width gives the order of magnitude of the estimated statistics' actual errors.

We’ve shown that certified results are nearly always reliable; however, we’d also like some insight into the proportion of results which we fail to certify that are actually reliable, to confirm that the test aligns with our objective. We use reliable here in the fairly strong sense that the confidence interval is well calibrated. Specifically, for the 50% and 90% confidence intervals, we count the proportion of uncertified results for which the confidence interval empirical calibration has an acceptable margin of error, given the number of trials used to estimate it. Using this procedure, the false positive rate across all experiments is around 1%. This agrees well with the failure rate we expect by chance, given our test parameters, and confirms the assumption underlying the test.

Finally, to better understand the difference between certified and uncertified results, Figure 4 shows the error distribution of the ratio of the estimated statistics' errors and half the 90% confidence interval for the reliable and unreliable results separately. Note that we truncated the range for uncertified intervals.

Wrapping up

In this post, we present the background behind our approach for quickly estimating ES|QL queries and providing an indication of their errors. To do this, we developed an effective confidence interval mechanism that allows us to provide error estimates. Our approach also allows us to estimate confidence intervals for quantities derived from sampled statistics via other pipeline operations. Quantifying the error comes with a relatively small overhead compared to just estimating the query. Finally, we developed a statistical test to certify results we return. Values that aren’t certified can still be accurate, but we’re less confident in them.

As well as testing the feature on a range of real-world use cases, which we discuss in our companion post, we tested the error estimation by extensive simulation across a range of data characteristics, sample sizes, aggregation functions, and confidence levels. This showed confidence intervals are well calibrated, and the interval itself provides a good approximation of the actual error we observe in the estimates. Finally, we showed that we were able to certify intervals with a low false negative rate.

We’re planning to integrate this feature into other stack capabilities in the future, so stay tuned.

One of the key requirements for a pleasant user experience is that these operations are performed quickly. Large language model–based (LLM) agents also introduce new higher bandwidth and speculative query patterns that can potentially benefit from different optimization strategies.

In this two-part blog series, we discuss an optimization approach we’re introducing to ES|QL in version 9.4 of Elasticsearch and the Elastic Stack, which exploits a relaxation of the problem. Rather than trying to get exact values for aggregates, we allow ourselves to return approximate values, together with some characterization of their error. A key benefit of approximation is that it breaks the dependency between performance and dataset size: The accuracy with which one can approximate a query doesn’t depend on the original dataset size but, principally, its data characteristics and the query itself. As we’ll see later, this allows us to achieve some dramatic performance improvements.

In our next blog post, we will discuss the theory behind our approach and the validation we’ve done of its statistical properties. Here, we introduce the syntax and give a sense of how it’s achieved using standard ES|QL and query rewriting. You can explore its performance on a subset of the popular ClickBench benchmark. Finally, we discuss some limitations and gotchas that are worth understanding when you use query approximation.

Syntax and behavior

So how do you actually use it?

That’s it. You simply introduce the new line SET approximation=true; and write your STATS query pipeline as usual. Below, we discuss some advanced configuration options and some limitations around the agg(...) and commands. However, essentially, we choose defaults so that this will typically provide useful approximations while achieving significant speedups.

With this change, you’ll see some differences in the query results. Let’s look at a concrete example to illustrate this. Suppose the raw query is as follows:

The results might look something like this:

Approximating this query introduces some extra columns for each quantity that’s estimated:

The count column now contains an estimate, and you’ll see it’s somewhat different from the exact values above. The _approximation_confidence_interval(count) column defaults to the central 90% confidence interval for the count estimate and the _approximation_certified(count) column indicates if we’re highly confident that the results and their confidence interval are trustworthy. In outline, the confidence interval is an interval we expect has a high probability (0.9) of containing the true value for the quantity being estimated. The certified column indicates the distribution of the approximation is behaving as we expect. When the result isn’t certified, it’s often still accurate, but our test of the properties of its distribution hasn’t been able to confirm this. These quantities are discussed in more detail in our second post.

Implementation

An approximate query is rewritten before query execution using random sampling and extrapolation. Let’s take a look at the query of the previous section. The part of the rewritten query responsible for obtaining the best estimate looks like:

The query samples a fraction of the data, and therefore the final count has to be extrapolated by scaling up with the inverse of the sample probability. Extrapolation clearly depends on the underlying aggregation function, and we handle this appropriately for all functions we support.

To obtain the sample probability, we're setting a fixed number_of_rows to be processed by the STATS command. In this case, the probability is calculated as follows:

This query is executed before the final approximate query is executed.

As well as this best estimate, confidence intervals and a statistical test used to certify that the value distribution is behaving as we expect also need to be computed. The intervals are computed using a variant of the bias-corrected and accelerated bootstrap confidence interval (BCa) method. Therefore, the data needs to be partitioned into B buckets, which are used in turn to compute the intervals. Omitting some implementation details, this approximate query looks like:

To certify the estimate and confidence interval, there should be enough data, and the distribution of the bucket values should tend to normality.

Some queries can be efficiently computed using only summary statistics maintained in the index. To handle these correctly, where sampling is both slower and inaccurate, we updated the physical query planner, since detecting this case requires information that’s only available where the data resides. When the planner detects this is possible, it simply executes the query as normal. Such queries are typically fast anyway, and there’s no real side effect, so you don’t need to worry about this when using approximation; however, you’ll see that confidence intervals for such queries always have zero length, indicating the results are exact.

Results

To explore the performance improvements, we use ClickBench. This is a benchmark for analytics workloads for database management systems (DBMS). It comprises approximately 100 million rows, with a focus on clickstream and traffic analysis, web analytics, machine-generated data, structured logs, and events data. The benchmark also defines 43 queries that are typical of ad-hoc analytics and real-time dashboards.

Some of the queries aren’t suitable for approximation. For example, we don’t support approximating the unique count of a categorical value or computing the minimum and maximum of a metric value. We also don’t care about queries targeting search alone, for which Elasticsearch has excellent performance in any case. We therefore exclude these types of query from our evaluation. Finally, we also want to test a few additional aggregation functions, such as percentiles, which are not well represented in the original query set, so add some variants of the original metric queries to this end.

Queries in the benchmark are written using standard SQL and so need porting to use ES|QL syntax. This translation is fairly straightforward. Here’s an example:

becomes:

when rewritten in ES|QL.

For running all benchmarks, we use an Elastic Cloud Hosted instance with 870GB disk, 29GB Ram, and 4 vCPUs, in effect, an Amazon Elastic Compute Cloud (EC2) i3.xlarge instance. In the following results, we simply compare ES|QL with and without query approximation. Extensive results on a range of different hardware setups and datastores can be found here. Even with significantly constrained test hardware (matching the vCPUs of the smallest setup), our approximation approach achieves competitive results against much larger systems.

We run each query and its approximation five times in a random order, clearing the query cache between each run. We report the average run time over all five runs. While clearing the cache should be sufficient to avoid most of the advantage of running second, we wanted to avoid any possible accidental prewarming effects, which is why we alternate.

The results break down into four categories:

1. Queries which are rewritten to use index summary statistics (three queries).

2. Queries that perform well (13 queries).

3. Queries with high cardinality partitioning (seven queries).

4. Queries with restrictive filters (12 queries).

Roughly speaking, for these four categories, approximate querying is: equivalent (1); faster and accurate (2); faster but unreliable (3); and slightly slower (4), compared to exact querying, respectively.

For category 1, the planner automatically detects that we’re able to perform the query using summary statistics, and we end up executing the queries in the same way. To do this, we need information that’s only available on the data nodes, so we perform the rewrite only after we've estimated the sample probability. Because we're able to do this very efficiently, the overhead is small (around 10–15%). In both cases, the results are exact.

Queries in category 2 run on average 23 faster if estimating the values and computing confidence intervals and 72 faster if just estimating the values, which you can select as follows: SET approximation={"confidence_level":null}. These headline figures hide quite some variation in the impact of approximation on performance. The table below shows some queries sampled from the range of speedups we see:

Query

Baseline / ms

Approximate with CI / ms

Approximate without CI / ms

3

1725

145

15

10

4340

1721

56

13

32912

6106

3821

21

46739

3284

2139

22

252505

6478

5019

Here are the corresponding queries:

We'll return to the accuracy of the approximation in the next blog post, but to give a sense of this, we plot below the exact and approximate values for a sample run for query 13:

For category 3, we get an average speedup of . However, the results of queries in this category can miss some partitions and often have large estimation errors. Approximation can still be valuable for such queries, particularly in the context of agentic workflows, but requires larger sample sizes than out default if accuracy is important. As we discuss in the next section, we provide an API to explicitly control the sample size. If the source dataset is sufficiently large, this can be increased and approximation will still yield significant performance improvements. The table below shows a couple of query examples for this category:

Query

Baseline / ms

Approximate with CI / ms

Approximate without CI / ms

15

8256

1187

124

17

70641

2109

982

Here are the corresponding queries:

Finally, category 4 queries use selective filters and end up being executed exactly, but they run slightly slower because of the work done in the query rewrite stage. Typically, all these queries run fast anyway, so the absolute slowdown is small. On average, they run approximately 14% or 370ms slower than the “without” sampling for our test setup.

Limitations and best practices

It’s worth explicitly mentioning some limitations. In particular, the following queries are not currently supported:

1. Queries using the TS source command.

2. Queries using the FORK or JOIN processing command.

3. Pipelines which use two or more STATS commands.

4. The ABSENT, PRESENT, DISTINCT_COUNT, MIN, MAX, TOP, ST_CENTROID_AGG and ST_EXTENT_AGG aggregation functions.

We plan to lift some of these restrictions in future releases, such as approximating queries using TS, FORK and JOIN; however, some are intrinsic. For example, while there’s prior art for estimating the minimum and maximum of a metric dataset or the count of unique values of a categorical dataset (see, for example, this paper), they require making certain distributional assumptions, either explicitly or implicitly. In summary, we view trying to automatically provide estimates of these statistics as being too open to accidental misuse.

For the expert user, we provide another route: ES|QL supports using the SAMPLE command directly. This allows one to obtain “point estimates” of any query, albeit with no attempt to correct for the impact of sampling or quantify error. For example:

computes the unique count of the value field on a sample of roughly 1/100th of the dataset. The sample probability can be adjusted to get a sense of how this is asymptoting, or more sophisticated estimation procedures can use STATS COUNT() BY value to estimate the frequency profile of the data.

There are a couple of cases that are more problematic for sampling. If a very restrictive filter is applied in the query, then sampling is of little value, since few rows match anyway. In such cases, we discover that we’d have to sample too large a proportion of the rows to estimate the query in the rewrite phase. In this case, we revert to running the query without sampling and its result is exact. However, the search procedure to determine the fraction of rows to sample comes with some overhead. One therefore pays a penalty, albeit less than the original query cost, for no benefit. If you know in advance that the query is expected to match relatively few rows, it's best to run it without approximation.

The second case only applies when computing STATS partitioned by some expression. If the cardinality of this expression is very high, then even if many rows are searched, individual statistics may be computed from a small number of rows. Some cases are more problematic than others. Sorting by ascending count, that is, finding the rarest partitions, can be impossible to estimate in a single query if heavy hitters would require us to sample most of the dataset to find them. For this particular case, heavy hitting partitions can be estimated first and sometimes efficiently excluded by updating the query. In general, infrequent partitions may be lost in the sampling process, and their statistics' estimation errors can be high. It’s worth noting that we won’t attempt to estimate any statistic for which we have fewer than 10 samples, and we simply drop them from the result set. In the case of very high cardinality BY clause, for example, a field whose value is unique for every row, this means the query can return no results. If you find approximate query results are too inaccurate, you have the option to increase the sample size, which by default is 1,000,000 for STATS, which uses grouping and 100,000 otherwise. Currently, this needs to be done manually, and we provide the following API for this:

Occasionally, functions significantly alter the distribution characteristics of the quantities they act on. A contrived example is the following:

If the variation in the estimate sl is much larger than we expect the distribution of csl to be mainly flat in the interval with peaks near both endpoints. In this particular case, it’s not clear that the central confidence interval is a particularly useful concept, since the modes of the distribution lie outside almost all central confidence intervals. In any case, just observing the samples of csl, our standard confidence interval machinery won’t reliably characterize this distribution and it will underestimate the variability of csl. However, our statistical test should detect this problem, and the result won’t be certified.

Finally, we note that Elasticsearch implements some query optimization strategies that ideally need to account for the fact that sampling is taking place. These rewrite the query at the Lucene level and the preprocessing involved in this rewrite can be relatively expensive. Accelerating an expensive string matching operation by first building a suitable data structure makes sense if the query needs to process every row, but if it processes only a small fraction of them, the trade-off is different. This is something we plan to enhance in future.

Conclusions

In this blog post, we introduced a new form of query optimization we’re bringing to ES|QL that enables dramatically faster querying by relaxing the constraint that the results are exact. We found on ClickBench that we were able to accurately estimate query values and their confidence intervals up to 100 times faster and values alone up to 250 times faster than we can compute them exactly. Furthermore, we expect this advantage to grow as the dataset size increases, because the approximation accuracy is independent of the dataset size. This feature works with many features of the ES|QL language and is enabled by simply prepending SET approximation=true; to the query to estimate.

As well as providing a point estimate, we also estimate confidence intervals and indicate whether we think that the underlying assumptions used to compute these are satisfied. This allows us to certify the results if the results are reliable. We explain the theory behind this feature and discuss the testing of its accuracy in our next post.

The Elasticsearch Java client supports ES|QL queries through the DSL, but currently it treats queries as simple strings, with no dedicated helper; and while Kibana offers an excellent UI to build ES|QL queries, we’re aware that sometimes having everything needed to write applications in the integrated development environment (IDE) offers a better experience. So, until the Java client extends its type support to ES|QL, we wrote an Intellij IDEA plugin that autocompletes, syntax checks, shows documentation, and executes ES|QL queries.

The plugin currently supports Java, Kotlin, and plain text files, in case the Java Virtual Machine (JVM) isn’t your thing.

Check it out in the JetBrains Marketplace page and in the GitHub repository, for more information.

Prerequisites

• IDE: Intellij IDEA version >= 253 (community or ultimate)

Usage

Install the plugin in Intellij IDEA like you would with every other plugin, so either from the JetBrains marketplace or by going to Settings -> Plugins -> Marketplace and searching “esql”.

The following examples are written using Java, but Kotlin is also supported and the usage is pretty much the same.

Create a text block string, write “ES|QL” in a simple comment above it, and you’re done.

If you see the Elastic logo appearing on the left:

then everything is working, and you’re ready to write your queries.

Why text blocks and not simple strings? The ES|QL syntax accepts quotes in various contexts, and escaping them would trigger other errors in the syntax checker, so we decided on text blocks to keep things simple.

It’s even simpler for txt files, as you can just add the comment and start writing the query right below:

Connecting to a server instance

The plugin can be connected to an Elasticsearch server instance to fetch indices and field names, which will then be added to the autocompletion options. Look for the Elastic logo on the bottom left of of the screen (or wherever you keep your tools), and configure your connection to any server instance:

Autocomplete

Start typing while in the text block to automatically open the autocompletion popup, which will return a list of acceptable commands/values to continue writing the query correctly. If you want to manually trigger autocompletion, ctlr+space is the IDE’s shortcut to use:

Syntax check

The plugin will highlight errors in queries, explaining what to fix:

Documentation

Hovering with the cursor over commands will display documentation describing what the command can be used for and its correct syntax:

Running the query

Once connected to a server instance, you can run queries by clicking on the green button beside the Elastic icon: The results will be displayed in the tool window:

Or if you’re writing an application, you can use the Java client like so:

Check our previous ES|QL Java Client article for a complete example of mapping ES|QL results to Java objects.