Just send your OTel metrics to the Elasticsearch OTLP/HTTP endpoint and they're stored using the new exponential_histogram type and queryable immediately. Already have historical data stored in the classic histogram type? A simple ::exponential_histogram cast in your ES|QL queries handles the migration transparently. Already using downsampling? Both histogram field types are now fully supported.

Histogram metrics

When dealing with metrics (in OpenTelemetry or Prometheus, for instance), counters and gauges are the most common metric types. Gauges allow you to monitor values that rise or fall (e.g., CPU utilization). Counters allow you to, well, count things, such as the total number of HTTP requests your service is handling. Counters normally just increase in value, with a few exceptions when they reset, like when a server reboots.

In the case of counters, you can additionally collect a counter measuring the total sum of your HTTP response times, which allows you to derive the average response time by dividing that sum by the total number of requests. However, average response times provide limited insights into the collected data and the system behavior. The best insights are gained by analyzing the collected metric distribution, e.g., through median and percentile calculations. This is where counters fall short.

In the past, workarounds have been applied: For example, classic Prometheus-style histograms attempt to capture the distribution using a set of counters. By defining fixed buckets (e.g., one for response times in the range [0s, 1s), one for [1s, 4s), and so on) and associating a counter with each, we can at least estimate percentiles broadly. However, the key problem here is that we have to know the distribution of our data up front to properly define these buckets.

To that end, the OpenTelemetry community has come up with a better solution: exponential histograms. Exponential histograms assign collected values to buckets, just like classic Prometheus-style histograms. The key differentiator is that these buckets vary dynamically based on the collected values. The name "exponential" comes from the fact that the bucket sizes increase exponentially: we use small buckets for small values and wider buckets for larger values. You can find an excellent introduction in the OpenTelemetry exponential histograms introduction.

Note that in addition to classic histograms, Prometheus also added native histograms, which directly map to OTel exponential histograms. Native histograms have their own PromQL syntax. We are actively working on adding support for that syntax to the Elasticsearch PromQL implementation, so that you can directly query exponential histograms using PromQL.

Demo setup

Let's start by collecting some histogram metrics to show how they can be stored and analyzed in Elasticsearch using ES|QL.

We'll focus on a Java JVM metric: garbage collection durations. OpenTelemetry defines the jvm.gc.duration, which is a histogram-typed metric. The OpenTelemetry Java agent natively supports collecting this metric.

We'll spin up a JVM running a Renaissance benchmark to put it under stress. We'll start that JVM with the vanilla OpenTelemetry Java agent attached and have it send the metrics directly to Elasticsearch.

You can find the ready-to-run Docker-compose file here. You'll just need to insert your Elasticsearch OTLP/HTTP endpoint and API key in the docker-compose.yml:

OTEL_EXPORTER_OTLP_ENDPOINT: https:///_otlp
OTEL_EXPORTER_OTLP_HEADERS: "Authorization=ApiKey "

Note that you don't have to use this demo setup. We even encourage you to try it with your own application. Here are the other important OpenTelemetry agent settings the demo already includes, which you should include too if you're bringing your own app:

OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE: delta
OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION: BASE2_EXPONENTIAL_BUCKET_HISTOGRAM
OTEL_INSTRUMENTATION_RUNTIME_TELEMETRY_ENABLED: "true"

Let's step through them:

• Temporality preference: OpenTelemetry supports both cumulative and delta-based histograms. Cumulative means that the histogram is only cleared after an application restart, while delta clears it after each export. At the time of writing, Elasticsearch only supports delta temporality for histograms. We are actively working on supporting cumulative histograms as well.
• Default Histogram Aggregation: By default, OpenTelemetry exports histograms in the Prometheus-style fixed bucket format. Since we want to reap the benefits of exponential histograms, we tell the agent to use them instead.
• Runtime Telemetry enabled: This tells the agent to actually collect the detailed JVM metrics, which include jvm.gc.duration.

Now we are ready to go! We'll let the application run in the background and switch over to Kibana to analyze the GC metric.

Querying with ES|QL

Now let's open up Kibana and navigate to "Discover". There we'll switch to ES|QL mode, and start querying the collected data:

TS metrics-* | STATS COUNT(jvm.gc.duration)

As a response, we now see the metric panel shown below. If you don't see any data, make sure to double-check the Kibana time range filter.

This number represents the total number of garbage collection operations that happened in our test application during the selected time frame.

Similarly, we can query the total time spent on those garbage collection operations:

TS metrics-* | STATS SUM(jvm.gc.duration)

So we have roughly 270k garbage collections, which in total took 713 seconds. Given these two numbers, we can now compute the average if we are still fluent in primary school-level math. Even if not, you can just let ES|QL do that for you:

TS metrics-* | STATS AVG(jvm.gc.duration)

Now we know that the average garbage collection operation took about 3 milliseconds. However, Java experts might know that there are different kinds of garbage collections happening, which can have significantly different pause times. Fortunately the OpenTelemetry metric comes with attributes, which allow us to slice the data accordingly:

TS metrics-* | STATS AVG(jvm.gc.duration) BY jvm.gc.action

As expected, major garbage collections take a lot more time per collection than minor ones, at least on average. So far, we have done nothing you couldn't also achieve by just using counters. Let's now use histograms to understand the actual distribution of the GC latency. We'll look at the data over time (by grouping using TBUCKET) and focus on the major garbage collections:

TS metrics-* 
| WHERE jvm.gc.action == "end of major GC"
| STATS MAX(jvm.gc.duration),
        PERCENTILE(jvm.gc.duration, 99),
        MEDIAN(jvm.gc.duration),
        MIN(jvm.gc.duration)
 BY TBUCKET(100)

The graph now shows us the minimum, maximum, median and 99th percentile for major garbage collections. Note that we aren't bound to only querying the median and the 99th percentile. We can query any percentile we'd like to see, as these are estimated at query time from the raw exponential histograms.

A note on backwards compatibility

So far, we have seen how you can use the new shiny toy in Elasticsearch and ES|QL: exponential histograms. However, since this has just reached general availability (GA) in the 9.4 release, what about your historical data?

Before exponential histograms were added, Elasticsearch was already capable of storing OpenTelemetry histograms in the histogram field type. To do so, we converted them to a different data structure supported by the histogram field type: T-Digest. T-Digest provides good accuracy for extreme percentiles (e.g., 99th percentile) at the cost of accuracy for percentiles in the middle of the distribution, such as the median. In contrast, exponential histograms provide a guaranteed upper bound on the relative error for every percentile. As conversions always introduce errors, we are happy to now have native support for exponential histograms, allowing you to collect and analyze your metrics end-to-end without unnecessary conversions.

But still, what should you do if you have historical data and still want to query it? Thanks to ES|QL union types, the answer is actually easy: You just have to add a ::exponential_histogram suffix to the histogram metrics in your queries:

TS metrics-* | STATS AVG(jvm.gc.duration::exponential_histogram)

When this query encounters histogram fields, it will attempt to convert them to exponential histograms. When operating on exponential_histogram fields, the ::exponential_histogram cast has no effect. Note that this also works with mixed data sets: if your backing indices use both types, the query will just do the right thing.

So if you are building queries or dashboards that you expect to run on pre-9.4 ingested data, we recommend that you simply add: ::exponential_histogram casts.

Wrapping up

Native support for OpenTelemetry exponential histograms in Elasticsearch gives you better metric fidelity and more flexible analysis in ES|QL. In this blog post, we have shown you how to easily ingest and analyze your histogram metrics with ES|QL using various aggregations and the impact exponential histograms have.

Exponential histograms are generally available in Elasticsearch basic starting with the 9.4.0 release. They will be available in Elastic Cloud Serverless a few weeks after the 9.4.0 release, once mOTLP (the managed observability OTLP intake) switches to use the Elasticsearch OTLP endpoint. We'll update this blog post and add a note on the Elastic Cloud Serverless release notes when that happens.

Place controls anywhere in the dashboard

Most dashboarding tools lock filters into a fixed bar at the top of the page. Kibana 9.4 breaks that constraint: controls are now regular panels you can drag anywhere to place them where you need them. Pin them to the top so they stay visible while you scroll, or place them inside a collapsible section where they automatically scope their filters to that section only — so a single dashboard can serve multiple use cases without one filter resetting all the panels.

Fewer clicks to a readable chart

Building a time series in Kibana Dashboards used to take 14 clicks. Now, it takes three. The new defaults give you a line chart with a bottom legend and cleaner x-axis labels — the way you'd configure them by hand. A right-hand legend still makes sense past ten series, but for fewer, the bottom list avoids the empty margin a side legend leaves behind. For top-value breakdowns, the default jumps from three series to nine — matching how many distinct colors the eye can parse at once — and the colors themselves are darker and more separated, so series stand out at a glance

Before (bar chart by default, legend to the right, redundant x-axis label):

Now (line chart by default, list legend at the bottom, more contrasted color palette for lines readability):

A calmer look for dense dashboards

Dense dashboards in most tools still look like a wall of boxes. We redesigned Kibana's dashboard visuals to feel flatter and more cohesive — more like a single page than a grid of disconnected panels. See some of these changes highlighted in the picture below compared to what the same dashboard looks like in 9.3.

Other improvements

Collapse filter pills

Another common filtering approach is using filter pills at the top of the dashboard. When you stack many pills, they used to consume vertical space; they now scroll automatically after a threshold, and you can collapse the row to reclaim the space.

Closing the gap for ES|QL panels

Elasticsearch Query Language (ES|QL) panels are closer to parity with the rest of dashboards: Drilldowns between dashboards work for ES|QL visualizations, and click-to-filter from the chart keeps improving. Upcoming releases aim to close remaining gaps, such as annotations and saving ES|QL visualizations to the library.

Panel size visibility

Before 9.4, it was hard to match panel height and width because there was no readout for panel size. Panel sizing is now shown while you resize, so you can align dimensions across panels for a more harmonious layout.

Easier section reordering

Collapsible sections picked up usability fixes, too: You can reorder sections while they’re expanded, and you can drag a section from its header without hunting for the small drag handle.

ES|QL support for Vega visualizations

Vega lets you build custom visualizations beyond what built-in chart types — radar charts, chord diagrams, or anything the grammar supports. The hardest part has always been the data query: writing verbose Elasticsearch Query DSL with nested aggregations and format paths just to feed data into the chart. Vega now accepts ES|QL as a data source, so you can get the data with a single readable query. Less time wiring up the data, more time on the visualization itself.

What’s next

This is just the start. Upcoming releases will keep pushing on smarter defaults and closing the gap between ES|QL and data-view visualizations so the two feel interchangeable.

Got a pain point or a feature request? Hit Submit feedback in the top menu — we're listening.

How to try it

If you use Elastic Cloud Serverless, you may already be on these changes. Otherwise, upgrade to 9.4 and then create a dashboard or open an existing one. Many updates apply automatically to new visualizations, while layout and panel options appear in edit mode. If you aren’t on Elastic Cloud yet, start a trial and explore the latest Kibana Dashboards there.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elasticsearch has supported storing metrics in time-series data streams (TSDS) since version 8.7. This offering mainly focused on storage gains as explained in an earlier blog post. Still, performance was not on par with specialized systems for storing and querying metrics, in terms of storage efficiency, indexing throughput and query latency.

In the past year, we revisited the storage layer, optimized ingestion for OTel metrics and extended the ES|QL compute engine with vectorized processing for time series data. These efforts led to substantial performance wins across the board, compared to earlier versions of TSDS:

1. Up to 6.6x improvement in storage efficiency, reaching 3.75 bytes per data point in OTel metrics
2. Up to 50% improvement in indexing throughput for OTel data
3. Up to 160x improvement in query latency, including blazing fast counter rate evaluation and window support in time series aggregations

Elasticsearch has thus become a leading columnar metrics engine, matching or exceeding the competition (like Prometheus, Mimir, and ClickHouse) in indexing throughput and exceeding it by up to 2.5x in storage efficiency and 30x in query performance. All while maintaining the ability to store logs and other data and fully use the rich querying capabilities of ES|QL (e.g. inline stats, lookup join) — which other PromQL-based systems lack. Elasticsearch can thus serve as a unified storage and query engine for all user data, with no compromises for metrics and observability applications.

How TSDS is organized

TSDS has the following properties that help improve the performance of time-series codecs and produce correct results when aggregating data points per time series:

• The metric name and the dimension names and values are used to calculate the _tsid, a unique identifier per time series.
• TSDS get sorted by [_tsid ascending, timestamp descending] order. Each time series is thus stored in sequence on disk, with newer data points appearing first. Since the _tsid is calculated over dimension values, the latter are also clustered on disk.
• Shard routing is based on _tsid, with each _tsid value appearing in one shard only.
• Backing indices are time-bound, with no overlap over time between them.

The rest of this post explains how we use these properties to improve storage, indexing, and query performance.

Storage optimizations

TSDS already achieved a very competitive storage footprint, reaching 0.9 bytes per data point, when it is possible to combine many metrics in a single doc, sharing the same dimension values. However, when most data points have a unique set of dimensions (which is typical for OTel or Prometheus metrics), docs end up containing a single data point. In this setup, storage required 25 bytes per data point, with dedicated metrics stores requiring less than 10 bytes per data point.

To further reduce the storage footprint, we applied a series of optimizations over the past year:

Replace inverted indices and BKD trees with doc value skippers

Elasticsearch creates inverted indices (for text values) or BKD trees (for numeric values) by default for all non-metric fields, i.e. for @timestamp and dimensions. These indices improve performance for queries including filters on these fields, but have significant impact to storage — effectively doubling the footprint for each field. More so, they are also processed during segment merging, increasing the cpu, memory and storage overhead and slowing down the system — especially in high ingest throughput scenarios, as is often the case with metrics.

Lucene has been extended with doc value skippers, a form of hierarchical sparse indices that store the minimum and maximum value of blocks of documents. Range queries can check these min and max values and ignore blocks that don't fall into the requested range. Skippers work particularly well on sorted fields. Since TSDS are sorted by [_tsid, timestamp desc], dimension values get also clustered on disk. It's therefore possible to replace indices on @timestamp and dimension fields with doc value skippers that amplify the columnar layout — each field stored in its own files, with no duplicate tracking of each doc for indexing purposes.

Doc value skippers have negligible storage overhead — replacing indices with them led to a reduction of 10 bytes out of the initial 25 bytes per data point in OTel. Moreover, they work very well in practice when queries include filters on time ranges or dimension values (including prefixes and regex) — there was no noticeable regression in query performance in our benchmarks when they replaced separate indices. Doc value skippers are enabled for TSDS by default since version 9.3.

Enable synthetic IDs

The _id metadata field was another big contributor to the storage footprint. TSDS has already been extended to trim the doc values once they were no longer needed for replication, but the inverted index was kept around to efficiently support the id-based APIs (Get, Delete, Update).

The ID value for TSDS is synthesized by combining the _tsid and @timestamp values that uniquely identify each data point. Since these fields are configured with doc value skippers, it's possible to replace the inverted index on _id with (a) retrieval of the _tsid and @timestamp value from the _id value, and (b) checks for matches using doc value skippers respectively. Care has to be taken to avoid expensive checks for duplicate IDs during metric ingestion, with segment-level bloom-filters keeping the overhead at bay.

Supporting synthetic IDs in metrics is a first for Elasticsearch. It led to a reduction of 5 bytes out of the initial 25 bytes per data point for OTel metrics, with no loss of functionality. Synthetic IDs are enabled for TSDS by default in version 9.4. We plan to extend their uses in logs and other applications after further evaluation.

Trim sequence numbers

Sequence numbers are used as part of replication, but also to provide strong consistency semantics on doc modification operations through Optimistic Concurrency Control (OCC). While such semantics are applicable to certain scenarios, they don't fit in metrics where concurrent updates are very rare, with no practical need for guarding against concurrent operations on data points with matching ids. We therefore decided to disable the use of sequence numbers in all APIs, along with OCC support, for TSDS, in version 9.4. This leads to a substantial storage reduction of 4 bytes out of the initial 25 bytes per data point for OTel data, as there's no inverted index and sequence numbers get trimmed once no longer needed for replication. Update and delete by query operations are still supported, albeit with weaker consistency semantics.

If OCC is still deemed important for a particular metrics application, the old behavior can be restored by setting index.disable_sequence_numbers: false in the index template of the involved TSDS.

Use large numeric codec blocks

TSDS already uses an advanced codec, as explained in an earlier article. The codec works very well in most cases, but has poor performance in case of repeated sequences of keywords and numbers, leading to an inflated storage footprint for dimensions containing IP and MAC addresses. We identified that the existing logic for identifying repeated sequences requires larger codec blocks to work well, especially as the sequence length increases. After experimentation, the numeric block size was increased from 128 to 512 elements in version 9.3, leading to a reduction of 2 bytes out of the initial 25 bytes per data point for an OTel dataset containing IP and MAC addresses as dimensions. We're also working on a more configurable codec layout that will allow more flexibility with block sizes and other parameters, based on field type and cardinality.

Indexing throughput

Elasticsearch has support for bulk ingestion of documents. This entrypoint has long been optimized for leniency, ensuring that all docs get accepted. This flexibility, however, incurs additional processing cost during indexing. Metric applications proved good candidates for using different approaches to reduce this overhead, as explained below.

Introduce OTLP protobuf entrypoint

OTel metrics and Prometheus have established protocols for metrics ingestion, using protocol buffers. In the past, a translation step was required to convert collected protobuf messages to bulk requests that Elasticsearch can consume.

Elasticsearch was recently extended with endpoints accepting messages from OTel metrics collectors and over Prometheus remote write. Parsing and processing these (binary) messages is cheaper, compared to json parsing, while hash operation over dimensions for _tsid calculations get reused and amortized across more data points within a single protobuf message. Furthermore, _tsids get evaluated once per doc in the coordinator nodes and propagated to data nodes for indexing, thus deduplicating an expensive step per indexed doc. These improvements led to up to a 20% speedup in indexing throughput for OTel metrics. The OTLP entrypoint was added in version 9.2 (tech preview) and reached GA in version 9.3. We've added similar entrypoints for Prometheus remote write in version 9.4 (tech preview) and are actively working to cover OTel Logs and Traces.

Reduce indexing CPU with doc value skippers

In addition to a substantial storage footprint, inverted indices require a lot of cpu to build and reconstruct during segment merging. The use of doc value skippers in their place helps also reduce cpu load at ingestion and thus improves indexing throughput by 10%, a welcome bonus on top of the aforementioned storage wins.

Synthetic recovery source

The original source of a document, as provided at index time, is never stored for metrics. Still, Elasticsearch needed to temporarily store it for replication purposes. That changed in version 9.1, where the source gets synthesized on demand for replication purposes. This is known as synthetic recovery source and reduces disk I/O by 50%, with a significant impact to metrics indexing performance. Check out this article for more details.

Query execution

Replacing inverted indices with doc value skippers leads to a pure columnar storage layout for TSDS, with metric and dimension fields stored as Lucene doc values, each field encoded and compressed in their own file. Combined with the introduction of the ES|QL compute engine that uses vectorized execution internally, it became possible to introduce a fully columnar storage and query processing engine for metrics in Elasticsearch. We pushed this idea to the extreme and implemented a columnar metrics processing engine that comfortably outperforms dedicated metrics engines and other columnar stores in query performance.

Time series integration in compute engine

Time series processing is largely based on applying aggregation functions per time series (or _tsid), such as a gauge average or a counter rate. These partial results are then reduced by a secondary function to produce results for the grouping dimensions, e.g. per host and process. Observability dashboards are built on top of this execution model, providing summary views of how metrics evolve over time and allowing for quick deep-dives by filtering on dimension values and time ranges.

To support this execution model, we introduced the TS source command, providing a simple yet powerful syntax for executing such queries that combine an inner aggregation function per time series with an outer aggregation over the partial results of the former. For instance, the following query calculates the hourly sum of rate of search requests per host over the last day:

TS metrics
  | WHERE TRANGE(1d)
  | STATS SUM(RATE(search_requests)) BY TBUCKET(1h), host

To execute this query, the compute engine is aware of how data is stored and applies the inner aggregation function per _tsid value. Since data are sorted by _tsid, time series aggregation functions process metric values as they get fetched, until the _tsid changes or the timestamp belongs to the next time bucket. This leads to vectorized execution of these functions over the fetched columns of metric values, while dimension values are only fetched (once) when the _tsid changes. The evaluation of the secondary aggregation function is also efficient, with partial aggregates stored in arrays of primitive values that get populated when _tsid values change.

The compute engine has inherent support for parallel query evaluation, taking full advantage of the available processing cores. Time series aggregations fully use this feature and process data points in parallel as applicable, reducing response times through improved cpu utilization.

Time series processing in ES|QL was introduced in version 9.2 as tech preview and reaches GA in version 9.4. We expect all metrics applications to adopt it and benefit from the much improved query performance wins.

Zero-copy data decoding and loading

Vectorized processing of time series data delivered immediate performance wins (8x for some queries), compared to aggregations through the /_search API, but the performance was still inferior when compared to competitive metrics stores. Benchmarking and profiling showed that there were too many array copies within the compute engine, between data decoding and evaluation of aggregation functions. To that end, the following optimizations were introduced:

• The codec for TSDS was extended to decode on-disk data directly into primitive arrays inside blocks that the compute engine uses to evaluate time series aggregations. No additional copies required, as the compute engine can bulk-read these blocks and process their arrays, one column at a time.
• Blocks containing a single value N times are represented as constant blocks with these 2 values, as opposed to an array with length N, a form of in-memory run-length encoding. Filtering and aggregation operations were extended to efficiently consume these blocks. This reduced memory pressure and cpu overhead for the _tsid and dimension fields, as their values get clustered due to index sorting.
• Documents with null values for the aggregated metric fields are filtered out at the Lucene level, before they get decoded and copied into blocks.
• All filters and regular expressions on the timestamp and dimension fields get pushed down to Lucene that makes use of doc value skippers to efficiently filter out non-matching docs.

Combined, these optimizations led to query execution speedups exceeding 10x (totaling 80x when combined with the 8x speedup from vectorized execution). They were included since the introduction of the TS source command in version 9.2, and fine-tuned ever since.

Optimized counter rate evaluation

While most time series aggregations can be trivially parallelized and evaluated, rate evaluation of cumulative counters is tricky as it requires processing all data points in order to detect counter resets (e.g. when a host restarts). To address this, the compute engine uses the _tsid prefix to shard time series across threads. Care has been taken to assign in-order ranges of _tsid values to each thread, as opposed to hash-partitioning _tsids, so that each thread can scan on-disk data in order, still making use of efficient decoding and zero-copying into blocks. The performance wins are impressive, with rate evaluation performance far exceeding dedicated metrics stores as we shall see in the next section.

Another interesting problem for cumulative counters is how to properly calculate counter increases for the entire time bucket when there are no data points at the bucket boundary timestamps. Metrics systems often use extrapolation, extending the first and last data points of each time bucket to the boundaries, or calculate the delta between the last data point of adjacent buckets. We posit we can do better, by interpolating between the last data point of each bucket and the first of the next, to get an estimate of the value on each boundary. The delta is then calculated over the interpolated values of the lower and upper boundary of each time bucket.

Sliding window support

Elasticsearch has long supported aggregations bucketed by time, but it was not possible to extend the window of processed data beyond each time bucket. Using windows larger than the time bucket, e.g. a window of 5 minutes for per-minute bucketing, helps smoothen out spikes and observe the underlying trend per time series with reduced noise:

All time-series aggregation functions have been extended with window support, as an optional argument. In case the window is a multiple of the time bucket (e.g. 1h window with TBUCKET(5m)), the compute engine first aggregates data points over intervals matching the time bucket span, and then combines these partial results per window span. This 2-phase approach eliminates repeated scans of data points and makes optimal reuse of intermediate results, improving response times. Window support was introduced as tech preview in version 9.3 and reaches GA in version 9.4.

Efficient datetime rounding

Queries on time-series data commonly include time bucketing. While data points can be trivially assigned to sub-hour time buckets, larger buckets start interfering with issues like time zones, daylight savings, variable days per month etc. Elasticsearch has elaborate logic for datetime rounding that takes these peculiarities into account, but that has relatively high cpu cost when processing time series data.

To mitigate this, the compute engine has been extended to identify cases where simpler logic can be employed to assign data points to time buckets. For instance, it can identify when the buckets are sub-hour or when timezones and daylight savings don't affect a particular query, and switches to simple modulo operations for datetime rounding. This led to a further 30% improvement in response times for certain queries. This change is introduced in version 9.4.

Performance evaluation

To evaluate the performance of our offering and track how it evolves and improves over time, we focused on OTel metrics since (a) Open Telemetry is the industry standard for collecting metrics, with universal adoption by all cloud providers and (b) they lead to a storage layout with 1 metric per doc, a setup that traditionally hurt performance for Elasticsearch.

We rely on Metricsgenreceiver to generate datasets. This tool is inspired by TSBS, producing data simulating the data points collected by the OTel hostmetricreceiver. We used two datasets:

1. A low-cardinality setup, with 100 hosts sending metrics every 1s, containing 14k time series in total
2. A high-cardinality setup, with 10k hosts sending metrics every 10s, containing 1.4M time series in total

We benchmarked on single-node deployments on EC2, using c6i.4xlarge and c8g.8xlarge machines for the low- and high-cardinality datasets respectively.

For competitive comparison, we used Prometheus (v.3.11.1), Mimir (v.3.0.6.) and ClickHouse (v26.3.9.8-lts). Prometheus and Mimir have proper time series processing, e.g. for counter rate, whereas ClickHouse lacks such support and only provides approximate values at best (for instance, it can't track counter resets consistently). We still report response times for ClickHouse to showcase that, once we optimize Elasticsearch for columnar query processing, it can exceed competing columnar engines even when they don't process the data per time series as expected.

We strived to use the default configuration for every system (including Elasticsearch), without tweaking them to optimize performance for the particular workload. This helps understand the user experience when systems are deployed by novice users, without much experience (or time) to tweak before receiving metrics traffic and setting up dashboards. We focused on single-node runs to keep noise low and accommodate all systems (Prometheus doesn't offer a multi-node setup out of the box). Elasticsearch performance provably scales well with the number of nodes; we plan to share scalability results in a future post.

Storage efficiency and indexing throughput

Our efforts to improve storage efficiency paid big dividends. Performance on OTel metrics dropped from 25 to 3.75 bytes per data point, in a year. Such an improvement, on top of an offering already optimized for time series, is really impressive and very rare in the industry.

The competitive picture looks favorable at this point, with Elasticsearch:

• Slightly outperforming Mimir in storage efficiency and indexing throughput
• Outperforming Prometheus by 2.5x in storage efficiency and by a small margin in indexing throughput
• Outperforming ClickHouse by 2x in storage efficiency and by 40% in indexing throughput

Query performance

The novel columnar engine for metrics processing proves very efficient in practice. We used a mix of queries based on gauge averages and counter rates, the most common operations that require different optimization approaches. The queried interval was 4 hours of data, covering all time series per metric.

ClickHouse doesn't support time series aggregations, so the results have limited value and are not directly comparable to Prometheus or Mimir that natively support time series processing. We used the published guidelines to adjust each query to get similar results to the extent possible. The point is to show how our columnar engine compares to generic columnar stores.

Here is a summary of the results:

†ClickHouse lacks native support for time series aggregations and counter reset detection.

Gauge average

We compared performance of evaluating the per-host hourly average of average memory utilization per time series, using the following queries:

# PromQL
avg by (host.name) (avg_over_time(system.memory.utilization[1h]))

# ES|QL
TS metrics-hostmetricsreceiver.otel-default
| STATS AVG(AVG_OVER_TIME(system.memory.utilization)) BY host.name, TBUCKET(1h)

Elasticsearch comfortably outperforms the other systems by up to 30x, in both low and high cardinality datasets.

Counter rate

We next compared performance of evaluating the per-host hourly average of cpu rate, using the following queries:

# PromQL
avg by (host.name) (rate(system.cpu.time[1h]))

# ES|QL
TS metrics-hostmetricsreceiver.otel-default
| STATS AVG(RATE(system.cpu.time)) BY host.name, TBUCKET(1h)

Despite processing data points per time series in order, counter rate performance matches calculating gauge average (the involved time series have 6.6x more docs than the query above). Elasticsearch maintains its wide advantage compared to the other systems and outperforms Mimir and Prometheus by 30x in the low cardinality dataset and by 16x in the high cardinality one.

It's really impressive that, for the high cardinality dataset, Elasticsearch is able to process 4 hours of data for half a million time series in less than 2 seconds, while the other systems take more than 30 seconds, leading to unresponsive dashboards for such queries. ClickHouse is also slower, despite having no logic to detect counter resets and extrapolate/interpolate deltas across buckets.

Prefix filter on host name

We next compared performance of filtering on host names based on their prefix, using the following queries:

# PromQL
avg by (host_name)
  (avg_over_time(system_cpu_load_average_1m{host_name=~"host-.*"}[5m]))

# ES|QL
TS metrics-hostmetricsreceiver.otel-default
| WHERE host.name LIKE "host-*"
| STATS AVG(AVG_OVER_TIME(system.cpu.load_average.1m)) BY host.name, TBUCKET(5m)

Elasticsearch manages to maintain an advantage of up to 5x compared to the other systems, despite replacing the inverted index on host.name with a doc value skipper.

Gauge average with window

We compared the performance of time series aggregations with a window of 90 minutes and time buckets of 30 minutes, using the following queries:

# PromQL
avg by (host.name) (avg_over_time(system.memory.utilization[90m]))&step=30m

# ES|QL
TS metrics-hostmetricsreceiver.otel-default
| STATS AVG(AVG_OVER_TIME(system.memory.utilization, 90m))
    BY host.name, TBUCKET(30m)

Elasticsearch maintains an advantage that reaches 25x for the low cardinality dataset and 8x for the high cardinality one. ClickHouse is outperformed by close to 4x, denoting the efficiency of our approach for windowed query operations.

What's next for Elasticsearch metrics

Elasticsearch has been extended with metrics storage and processing capabilities that outperform Prometheus, Mimir, and ClickHouse. We're making fast progress with supporting PromQL and Prometheus remote write, also available as tech preview in version 9.4. These extensions enable users familiar with Prometheus and relevant systems to switch their applications to Elasticsearch — no need to migrate existing dashboards. Since Prometheus integration reuses the same storage and query engine that has been presented in this article, the same performance wins are also expected for Prometheus. Furthermore, collected metrics can be queried with PromQL and ES|QL, side-by-side or in ES|QL query pipelines, further boosting the analytics capabilities far beyond what was conceivable so far with Prometheus-based systems.

The improvements in storage efficiency, indexing throughput and query performance are already impressive, but we're not done. We'll be introducing more refinements to the codec for time series data, further reducing bytes per data point. Batch processing of ingested metrics will be further improved, reducing synchronization overhead and redundant processing layers that are not needed for well-formatted collected metrics. We're also planning to make wider use of doc value skippers, storing pre-computed aggregates like sum and count per block of values, to shortcut data point loading and processing where applicable, as well as use more cpu-friendly partitioning and grouping operations.

From policy logic to retrieval architecture

Part 3 and Part 4 provided a technical deep dive into the governed control plane and its implementation using the Elasticsearch percolator. Once the logic layer has identified which policies to apply, the system must address the retrieval strategy used to execute the search.

Managing the transition from precision to recall is a critical function of any ecommerce search engine. For example, a basic search implementation often defaults to broad keyword matching. If a shopper searches for "organic Pink Lady apples", this can lead to irrelevant results, such as apple-scented dish soap, apple juice, or organic pink grapefruit, appearing at the top of the list simply because they share a common term. While these items are technically matches, they fail to satisfy the user's intent and typically lead to high bounce rates. However, a "No results" page is equally detrimental to conversion. This conflict is resolved by implementing a three-tier execution model, which uses the governed control plane to orchestrate a principled fallback strategy.

The three-tier execution model

This architecture executes up to three retrieval tiers in a sequence, each with a specific matching logic.

Highest tier: Strict matching

Strict matching is a lexical match that requires that all query terms appear in the product metadata.

• The logic: A search for "organic navel oranges" returns only products containing all three terms.
• Application: This tier provides the highest precision. When a customer types a precise product name, such as "organic navel oranges", they’re typically seeking that exact item rather than an alternative.

Mid-tier: Relaxed matching

If the strict tier fails to return sufficient results, the system expands the search parameters.

• The logic: This tier allows for a subset of terms to lexically match, using Elasticsearch's minimum_should_match logic.
• Application: Relaxed matching maintains lexical grounding. A search for "organic navel oranges" might surface "navel oranges" (missing the "organic" term) or "organic oranges" (missing the "navel" term). These represent intuitive, keyword-based alternatives for the shopper.

Lowest tier: Semantic matching

• The logic: This tier uses vector/semantic embeddings (such as Elastic Learned Sparse EncodeR [ELSER], E5, or Jina) to retrieve conceptually related products, regardless of direct keyword overlap.
• Application: A search for "organic navel oranges" might surface "mandarins" or "clementines”. This serves as the final retrieval tier, intended to provide relevant options when literal keyword matches are unavailable.

To see this multi-tier orchestration in action and how the Engine steps down from lexical to semantic matching, watch the video: Eliminating Zero-Result Pages: PRISM’s Multi-Tier Search Fallback.

Tier orchestration: The "bucket filling" logic

While the governed control plane provides the logic and the queries for each tier, the application layer is responsible for the execution. The application executes these tiers sequentially and excludes lower tiers once the accumulated result count on the first page reaches or exceeds 10 items (or whatever number of results you want to display on the first page). This threshold ensures a full first page of results while prioritizing the most accurate retrieval method.

Scenario 1: High-intent search ("oranges")

The first tier returns 15 hits. Since 15 is more than 10, the current result set is locked to only strict matches (which can be paged through) and subsequent tiers are not executed.

Strict tier:   [##########]##### (>= 10 found: Exact matches)
Relaxed tier:  [          ]      (Tier bypassed)
Semantic tier: [          ]      (Tier bypassed)

Scenario 2: Specific but limited results ("organic blood oranges")

The strict tier finds only four items. Since this is less than 10, the system triggers the relaxed tier, which finds 12 more relevant products. The combined total (16) meets the threshold of 10, so the current result set is locked to the strict and relaxed tiers. Subsequent paging will only surface results from these two tiers (preventing lower-quality semantic hits from appearing on later pages).

Strict tier:   [####      ]       (4 found)
Relaxed tier:  [    ######]###### (>= 6 found)
Semantic tier: [          ]       (Tier bypassed)

Scenario 3: Abstract or intent-based search ("high vitamin C snacks")

Keyword matches are limited (only five hits between tiers 1 and 2). The system triggers the semantic tier to find conceptually relevant items, such as kiwis, guavas, or red peppers, to fill the result set. The result set for this query includes products from all tiers.

Strict tier:   [##        ]             (2 found)
Relaxed tier:  [  ###     ]             (3 found)
Semantic tier: [     #####]######################...

This orchestration optimizes for latency, as the computational cost of the semantic tier is only incurred when the keyword-based tiers are insufficient. Additionally, this allows fast-responding keyword results to be displayed while semantic results are integrated shortly after, maintaining a responsive user interface.

Determining intent via tier activation

The logic used to fill the first page serves a critical secondary purpose: It acts as a diagnostic for user intent. The application uses the logic returned by the governed control plane to determine which tiers remain active for the current result set and paging.

If the strict and relaxed tiers together yield fewer than 10 results, the query is likely exploratory or abstract. In this case, activating the semantic tier is a benefit. Because the query is diagnosed as exploratory, the system allows the shopper to page through the entire depth of the semantic results. This provides access to conceptually related alternatives that lexical matching would have missed, which is appropriate for an abstract search.

Conversely, if the strict tier returns a robust set of results (for example, 30 hits), it confirms that the system has found high-precision matches. The user can page through those 30 hits and will likely find what they’re looking for. In this scenario, there’s no need to provide additional, less relevant exploratory hits. By disabling lower tiers for these high-precision queries, we ensure that a shopper deep diving into specific results isn’t distracted by irrelevant semantic fallback as they paginate through the current result set.

Governance across tiers

A critical component of this architecture is that policies apply globally across all tiers. If a user has a "vegan" preference profile, the governed control plane injects that constraint into the strict, relaxed, and semantic queries. This ensures that even when the system uses semantic fallback to return "mandarins" for an orange search, the results remain compliant with the user's broader dietary preferences or business constraints.

The problem of facet instability

A challenge with multi-tier search is maintaining consistent faceted navigation (sidebar filters). If a search for "chocolate" yields 12 strict results, the sidebar filters might show "dark" and "milk". If a user selects "dark" and the result count drops, a naive system might trigger the semantic tier to fill the page, which could suddenly introduce "red wine" into the filters due to a semantic relationship.

The governed control plane identifies which tiers contributed to the initial search and locks the facets to those tiers. This prevents the sidebar from changing unexpectedly during a filtered session, ensuring a stable user experience.

The pagination challenge: Seamless multi-tier paging

Pagination in a tiered system requires precise state management. As established, the first page determines the scope of the current result set. If the first page required semantic results, the user can page through all available results from all three tiers. On the other hand, if the first page was satisfied by high-intent keyword matches, the semantic tier is not retrieved for that specific result set.

The governed control plane manages this through:

• Tier locking: The response includes an array identifying the contributing tiers. The front end returns this on subsequent requests to keep the tier composition consistent across all pages.
• Dynamic offset calculation: The back end calculates an offset based on the requested page and the total products returned in preceding tiers.Example: If the first page has returned seven strict matches and three relaxed matches, a request for page 2 (starting at index 10) would execute a relaxed tier query with an offset of three.
• ID exclusion for lower tiers: The system retrieves IDs from the higher tiers (which, by definition, will always be fewer than the page size threshold) and explicitly excludes them from lower-tier results using an ID-only query (which avoids the overhead of a full fetch phase for excluded items).

Summary

The multi-tier approach ensures search results are precise when data is available and helpful when it is not. By providing a governed fallback sequence for the application to execute, the architecture maintains high relevance while eliminating "no results" scenarios.

What's next in this series

The next posts in this series extend the governed control plane into new territory. Part 6 explores personalization (using purchase history boosting and cohort-aware policies), and Part 7 demonstrates per-query economic optimization. Stay tuned!

Put governed ecommerce search into practice

The search architecture described in this post, where retrieval tiers, economic weights, and governance constraints compose into a single request, was designed and built by Elastic Services Engineering as part of our repeatable ecommerce search accelerators.

To learn more about applying these patterns to your business, Contact Elastic Professional Services.

The problem: Heterogeneous data, one query

Consider a production incident investigation. Errors are spread across three microservices: an API gateway, a payments service, and an auth service, each with different field names and different conventions. Before subqueries, combining them in a single ES|QL query meant cramming everything into one FROM with CASE chains:

FROM svc-gateway-*, svc-payments-*, svc-auth-* METADATA _index
| WHERE http.response.status_code >= 500
    OR transaction.status IN ("failed", "timeout")
    OR (event.action == "login" AND event.outcome == "failure")
| EVAL
    service = CASE(
      _index LIKE "svc-gateway*", "gateway",
      ... /* one branch per source */),
    error_detail = CASE(
      _index LIKE "svc-gateway*", CONCAT("HTTP ", http.response.status_code::string),
      ... /* one branch per source */)
| KEEP @timestamp, service, error_detail, source.ip
| SORT @timestamp DESC

This is brittle and slow. The disjunctive OR prevents predicate pushdown; every index scans every condition. Every CASE chain grows with every source. Copy it into five dashboards and three alert rules, and you have eight places to update when anything changes.

The fix: Independent pipelines

Subqueries replace the monolithic FROM + CASE pattern. Each data source gets its own complete pipeline:

FROM
  (FROM svc-gateway-*
   | WHERE http.response.status_code >= 500
   | EVAL service = "gateway",
         error_detail = CONCAT("HTTP ", http.response.status_code::string)
   | KEEP @timestamp, service, error_detail, source.ip),
  (FROM svc-payments-*
   | WHERE transaction.status IN ("failed", "timeout")
   | EVAL service = "payments",
         error_detail = transaction.status
   | KEEP @timestamp, service, error_detail, source.ip),
  (FROM svc-auth-*
   | WHERE event.action == "login" AND event.outcome == "failure"
   | EVAL service = "auth",
         error_detail = CONCAT(event.action, " ", event.outcome)
   | KEEP @timestamp, service, error_detail, source.ip)
| SORT @timestamp DESC
| LIMIT 20

The gateway branch only scans for HTTP 500s. The payments branch only looks at transaction statuses. The auth branch only checks login failures. Because each branch has its own WHERE, the optimizer pushes filters independently into each index, restoring the predicate pushdown that a single FROM with OR conditions prevents. Fields that exist in one branch but not another are filled with null.

Adding a fourth service means adding a fourth branch. Existing branches don't change.

Save it as a view

This is where subqueries and logical views combine. Wrap the subquery above in a named view, with one API call:

PUT _query/view/error_triage
{
  "query": "FROM (FROM svc-gateway-* | WHERE ...) , (FROM svc-payments-* | WHERE ...) , (FROM svc-auth-* | WHERE ...)"
}

Now consumers just write FROM error_triage | STATS error_count = COUNT(*) BY service. Three indices, three pipelines, one name. If you have 10 dashboards and five alert rules consuming this pattern, that's 15 copies of the same logic today; with a view, it's one definition and zero consumer-side edits when you add a fourth service. See Elasticsearch ES|QL Views for the full views deep dive.

What you can do inside a branch

Each branch supports the full ES|QL pipeline: WHERE, EVAL, STATS, LOOKUP JOIN, ENRICH, and more. See the subquery documentation for the complete list.

Aggregate different metrics, and then combine

Each branch can compute its own summary before results are merged. This is useful when different indices track the same concept under different field names:

FROM
  (FROM svc-gateway-*
   | STATS avg_latency = AVG(http.response.time_ms) BY hour = BUCKET(@timestamp, 1 hour)
   | EVAL service = "gateway"),
  (FROM svc-payments-*
   | STATS avg_latency = AVG(transaction.duration_ms) BY hour = BUCKET(@timestamp, 1 hour)
   | EVAL service = "payments")
| SORT hour DESC, service

Both branches produce avg_latency and hour, but each computes it from a different source field. The combined result is a single table you can chart or alert on, without normalizing field names at ingest time. This pattern is impossible with a single FROM; you can't compute different aggregations per index without subqueries.

Subqueries vs. FORK

ES|QL also has FORK (now generally available), which creates parallel execution branches from the same input. The distinction:

Different indices → subqueries. Same data, different analyses → FORK.

How this compares

If you're coming from other query languages, here's how ES|QL subqueries stack up at the time of writing:

Splunk SPL/SPL2 has append and multisearch in classic SPL, and SPL2 adds a union command that merges events from multiple datasets (the closest analogue to ES|QL subqueries). Federated Search extends this across remote Splunk deployments (analogous to CCS). The differences are in how the engine handles each branch: ES|QL subqueries give each branch independent predicate pushdown, meaning filters are pushed into each index's shard-level structures separately. SPL2 union merges datasets but optimization across branches is limited to what the search scheduler can parallelize. Wrapping ES|QL subqueries in a view gives you engine-level encapsulation with role-based access control (RBAC); Splunk's equivalent is saved searches and macros, which are text substitution expanded at parse time.

SQL databases have UNION ALL, which is the closest analog. The difference is that SQL UNION ALL typically requires matching column counts and types at parse time. ES|QL subqueries are more forgiving; columns that exist in one branch but not another get null-padded automatically, which matters when your sources have different schemas (the norm in observability data). SQL views solve the reuse problem similarly, but ES|QL views are cluster-level objects, not database-scoped; they work across cross-cluster search boundaries.

Grafana / Datadog / other dashboarding tools handle multisource composition at the visualization layer: Run separate queries, merge in the panel. This works for display but breaks for alerting, downstream queries, and anything that needs a single result set programmatically. ES|QL subqueries push the composition into the engine, so alerts, views, and API consumers all get the same unified result.

Current constraints

In the Tech Preview release, subqueries are non-correlated; branches run independently and can't reference the outer query. They're supported in FROM only (not TS), and FORK can't be used inside or after subqueries. See the subquery documentation for details.

What's next for subqueries

WHERE subqueries — WHERE field IN (FROM other_index | ...) and other correlated forms — will extend the composition model from FROM into filtering. This brings the familiar SQL pattern of nested filtering to ES|QL.

Try it

Subqueries in FROM are available as a Tech Preview. Try them in Kibana Dev Tools or Discover. We'd love your feedback; file a GitHub issue with the ES|QL label.

ES|QL subqueries in FROM are a Tech Preview feature. Tech Preview features are subject to change and are not covered by the support SLA of GA features. The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

A brief quantization primer

Vector search indexes often store millions or billions of embedding vectors, each hundreds or thousands of floats wide. Scalar quantization compresses each float coordinate independently to a small integer, typically 1, 2, or 4 bits, reducing storage by 8-32× and enabling fast integer-arithmetic scoring.
Why does per-coordinate quantization work at all? Because the expected squared error decomposes as a sum of independent per-coordinate terms: $\mathbb{E}[\|x - q(x)\|^2] = \sum_i \mathbb{E}[(x_i - q_i(x_i))^2]$, by linearity of expectation. Each coordinate can be quantized independently regardless of the joint distribution. The remaining question is how to make each per-coordinate quantizer good.

BBQ and Optimized Scalar Quantization

Better Binary Quantization (BBQ), and its underlying algorithm Optimized Scalar Quantization (OSQ), have been part of Elasticsearch for multiple releases. OSQ is an evolution of several techniques to make scalar quantization more accurate, specifically for vector search.

Each vector's components are mapped to a uniform grid over an interval $[a, b]$. The interval is initialized from the vector's statistics (assuming approximately normal residuals), using the exact same optimization objective as TurboQuant but with an additional constraint on centroid positions. They are then refined by coordinate descent to minimize an anisotropic loss $L = (1-\lambda)(x \cdot e)^2/\|x\|^2 + \lambda\|e\|^2$, where $e$ is the quantization error vector. With the production default $\lambda = 0.1$, this deliberately sacrifices some MSE to concentrate accuracy along the query direction. This is the direction that matters for ranking.

Before quantization, the segment (or cluster in the case of Inverted Vector File (IVF) indices) centroid $c$ is subtracted from every vector. This removes the dominant shared component that would otherwise waste the quantizer's dynamic range. Both the symmetric and asymmetric dot-product paths also center the query using the same segment centroid, so the only quantized inner product is between centered residuals. The correction terms, $\langle c, x \rangle$ for each vector and $\|c\|^2$, depend on only a single vector each, and can be precomputed exactly. Centering therefore adds no per-pair cost.

Documents are quantized at 1-bit (32× compression), queries at 4-bit (cheap since there are only a handful per search). The storage constraint binds on documents, not queries, so spending more bits on the query side recovers float-query accuracy while keeping per-document footprint minimal.

A block-diagonal orthogonal preconditioner equalizes coordinate variances and normalizes their distribution before quantization. This is the same goal as a full Hadamard rotation used by TurboQuant, but with no power-of-2 padding overhead.

Because the grid is uniform, quantized dot products decompose into integer dot products with scalar corrections. This enables NEON/SVE and SSE/AVX popcount and multiply-accumulate pipelines: bit-plane decomposition for 1-bit and 2-bit, nibble multiply for 4-bit, and a RaBitQ-style mixed 4×1 kernel that decomposes to four 1-bit kernels.

For a deeper dive into the sparse rotation and what it brings to robustness, see the Robust Optimized Scalar Quantization blog. For a full walkthrough of optimized scalar quantization, see this OSQ deep dive.

TurboQuant

TurboQuant (Google, ICLR 2026) takes a slightly different path to the same starting observation: that concentrated, predictable per-coordinate distributions are easy to quantize well.

Rather than adapting the quantizer per vector, TurboQuant normalizes the vector and applies a shared randomized Hadamard rotation to the entire dataset. This general sort of idea was first proposed and formalized by RaBitQ, which showed that the random rotation yields worst-case bounds over the data, holding for any fixed unit vector, for their quantization scheme. The idea of implementing via a Hadamard rotation was suggested by Weaviate for rotational quantization. After normalization and rotation, each coordinate's distribution almost always converges to $\mathcal{N}(0, 1/d)$ in high dimensions, regardless of the original data. TurboQuant builds on this foundation: with the distribution pinned down, it solves for the optimal Lloyd-Max scalar quantizer, a 1-D $k$-means problem on the known density. The resulting non-uniform centroids bunch up where the density is highest (near zero) and spread out in the tails. This achievies provably near-optimal MSE: within ~2.7× of the information-theoretic lower bound in general, and as tight as 1.45× at 1-bit.

For inner products, MSE-optimal quantizers introduce a multiplicative bias (which is most severe at 1-bit: $2/\pi \approx 0.64$). TurboQuant corrects this with a two-stage design ($Q_\text{prod}$): spend $b-1$ bits on the MSE quantizer, then use the remaining 1 bit for a Quantized Johnson-Lindenstrauss (QJL) sketch of the residual, yielding a provably unbiased inner-product estimator.

The paper's nearest-neighbour experiments were conducted on GPU (NVIDIA A100), where the lookup-table access pattern maps naturally onto shared memory.

The key design divergence: integer arithmetic vs. lookup tables

The difference between uniform and non-uniform centroids may seem minor, but it creates a large computational gap.

OSQ's uniform grid means each quantized coordinate is an integer whose arithmetic meaning is preserved. The dot product of two quantized vectors decomposes into an integer dot product, directly exploitable by SIMD: vpdpbusd on x86, multiply-accumulate and vcnt (popcount) on ARM NEON. The pipeline is branch-free and the data access pattern is sequential.

TurboQuant's non-uniform centroids break this. Each coordinate pair requires looking up a centroid value from a shared codebook, and the access pattern is data-dependent with each index selecting a different table entry. On NEON, which lacks a float gather instruction, this means scalar loads to build each vector register before the Fused Multiply-Add (FMA). Precomputing per-coordinate product tables ($d \times 2^b$ entries, amortized over all documents) doesn't help either: the FMA is relatively cheap on modern cores, so the bottleneck remains the data-dependent gather, not the arithmetic. Our benchmarks confirm this: precomputed ADC tables are no faster (and sometimes slower due to the larger working set) than inline centroid lookup.

Terminology used in the comparisons

The results sections below refer to several OSQ scoring configurations. All use uniform-grid quantization with scalar correction terms to recover the exact dot product up to quantization noise.

Symmetric $n$-bit quantizes both query and document at $n$ bits per coordinate.

Asymmetric keeps the query as a full float vector and quantizes only the document. The dot product is a float-times-integer sum. This is more expensive per pair than symmetric, but avoids any query quantization noise. TurboQuant's scoring is always asymmetric (float query dotted against quantized document via centroid lookup).

1-4 is the production configuration for OSQ: documents at 1-bit (32× compression), queries at 4-bit. This exploits the asymmetry of search: there is one query but millions of documents, so query storage is free but document storage is the binding constraint.

Centered means the segment centroid $c$ has been subtracted from all vectors (and the query) before quantization, with the exact correction recovered from precomputed scalar terms. Centering focuses the quantizer's dynamic range on the information-bearing residual rather than the shared mean.

$\lambda$ controls the anisotropic loss tradeoff: $\lambda = 1$ minimizes pure MSE, $\lambda = 0.1$ (production default) sacrifices some MSE to concentrate accuracy along the query direction, the direction that determines ranking.

How do they compare in practice?

The following results were obtained on an Apple M2 Max. The code to reproduce all these results is available here.

Head-to-head: MSE

On reconstruction MSE, the metric TurboQuant was designed to optimize, TurboQuant outperforms plain OSQ at every bit-width.

Relative MSE ($\|x - \hat{x}\|^2 / \|x\|^2$) on $d = 768$ Gaussian vectors (1,000 vectors, lower is better):

The $\lambda$ columns reveal that OSQ's production setting ($\lambda = 0.1$) deliberately sacrifices MSE for dot-product accuracy. With $\lambda = 1$ (pure MSE), the gap narrows, to just 1.18× at 1-bit.

But where does TurboQuant's remaining MSE advantage actually come from, the Lloyd-Max centroids, or the Hadamard rotation? We can answer this directly by applying the same randomized Hadamard rotation to OSQ (zero-pad 768→1024, random sign flips, Walsh-Hadamard butterfly, quantize in rotated space, invert). Theory predicts the MSE improves by a factor of $d/d' = 768/1024 = 0.75$:

OSQ + Hadamard matches TurboQuant almost exactly at 1-bit (0.306 vs 0.307) and 2-bit (0.092 vs 0.092). TurboQuant's MSE advantage is the rotation, not the centroids. At 3–4 bits the Lloyd-Max placement contributes a modest ~1.1× edge, real but small.

The convergence of the improvement ratio applying the Hadamard transformation to OSQ is itself informative: at 4-bit it hits the theoretical 1.33 exactly, but at 1-bit it's only 1.19. The shortfall quantifies the value of OSQ's data-dependent interval refinement: it already captures ~40% of the dimension expansion and component equalization benefit that Hadamard provides. The coordinate-descent is doing some of the same work as the rotation, adapting to each vector rather than relying on a data-oblivious transform. However, the real advantage, as we discuss below, is this formulation allows us to concentrate accuracy along the query direction.

This raises a natural question: how does OSQ's block-diagonal sparse preconditioner compare to the full Hadamard rotation in practice?

Head-to-head: sparse preconditioner vs Hadamard

OSQ's sparse preconditioner applies a block-diagonal random orthogonal transformation: dimensions are randomly permuted into blocks (64×64 in production), each block is multiplied by an independent random orthogonal matrix. This equalizes coordinate distributions within each block. The Hadamard rotation achieves the same goal globally but requires zero-padding to the next power of 2.

We test on anisotropic Gaussian data ($d = 768$, $\sigma_i$ ramping from 1 to 5 across coordinates), a challenging distribution where some coordinates carry far more variance than others.

Transform latency ($d = 768$, 1,100 vectors, ARM NEON, lower is better):

Hadamard is the fastest non-trivial option thanks to $O(d \log d)$ butterflies vs $O(d \times b)$ for block size $b$, though all block-diagonal variants are fast enough to be negligible in practice with even the 64×64 block at 4.9 μs is tiny compared to typical search latencies. The full dense $d \times d$ rotation is impractical at 244 μs/vec but serves as a theoretical reference. Note that the block-diagonal transform works for arbitrary dimensions: no power-of-2 padding is required, and the effective dimension stays at $d$.

MSE (relative MSE, $\lambda = 1$, anisotropic data, lower is better):

Even 32×32 blocks recover most of the gap from no-transform (0.443) to full rotation (0.362), 93% at 1-bit. Block 64×64 closes the gap further. On isotropic data (not shown), all methods produce identical MSE (~0.362 at 1-bit), confirming there is nothing to equalize when coordinates already have equal variance.

Dot-product accuracy (1-4 centered, raw relative dot-product error; note these are raw RMSE including multiplicative bias, which is appropriate for comparing preconditioner variants against each other since the bias structure is similar, lower is better):

On isotropic data the block-diagonal methods and the full dense rotation produce the same dot-product error as no transform since there is nothing to fix. Hadamard is the outlier, improving from 0.723 to 0.629. But this improvement is not from better preconditioning: the full dense rotation, which is an equally good random orthogonal transform, shows no improvement at all. The difference is the padding. Hadamard operates in 1024 dimensions, so 1-bit documents store 1024 bits instead of 768. This is 33% more storage. The improvement ratio (0.723 / 0.629 = 1.15) matches $\sqrt{d'/d} = \sqrt{1024/768} = 1.155$ almost exactly, confirming that the entire dot-product advantage is attributable to the extra bits, not the rotation.

On anisotropic data, the block-diagonal rotation does help dot-product accuracy (0.690 → 0.602 for block 64), which is the real value from coordinate equalization. Hadamard goes further (0.566), but the incremental improvement over a full dense rotation at the same dimension (0.595 → 0.566) is again consistent with the padding benefit.

The practical implication: for CPU-based search where storage efficiency matters, the block-diagonal preconditioner delivers the same MSE improvement as Hadamard at the same effective bit rate, works for any dimension without padding, and the dot-product gap we see in our experiments is a padding artifact, not a preconditioning advantage.

Head-to-head: dot-product accuracy

MSE measures reconstruction quality, but search engines rank by dot products. These are different objectives, and the gap between them is where OSQ's design choices pay off.

We measure relative dot-product error: $\sqrt{\sum(q \cdot x - \hat{q} \cdot \hat{x})^2 / \sum(q \cdot x)^2}$, varying the angle between query and document. The small-angle regime (0°–20°) matters most: real transformer embeddings occupy a narrow cone rather than spreading uniformly on the sphere (Ethayarajh 2019). Furthermore, near-parallel vectors, corresponding to the nearest neighbours of a query in the dataset, are where ranking accuracy is critical.

Our production configuration is 1-bit documents, 4-bit queries, centroid centering, with integer scoring.

Raw dot-product error conflates two distinct components: a multiplicative bias (a global scale factor that preserves ranking order) and noise (random per-pair deviations that can swap rankings). For search, only the noise matters: a biased estimator that consistently scales all scores by the same factor produces the same ranking as the exact scores. TurboQuant's MSE quantizer at 1-bit has a well-known multiplicative bias of $2/\pi \approx 0.64$, meaning raw dot-product errors of ~0.36 are almost entirely this ranking-irrelevant scale factor. To give a fair comparison, we report the debiased RMSE after fitting and removing the best multiplicative scale: $\alpha = \sum(\hat{d} \cdot d) / \sum(d^2)$, then measuring $\sqrt{\sum(\hat{d}/\alpha - d)^2 / \sum d^2}$.

Zero-mean corpus ($d = 768$, 500 vectors, 5 queries per vector, lower is better):

Shifted corpus (shift = 2.0, modeling real embedding bias, lower is better):

On zero-mean data, the raw error numbers (omitted for brevity, TQ @1-bit's is ~0.363, almost entirely due to the $2/\pi$ multiplicative bias) are misleading; only the debiased ranking noise matters. The asymmetric column (float query, 1-bit document) is the most directly comparable to TQ since both quantize only the document: at 0° OSQ achieves 2.4× lower noise (0.0035 vs 0.0083). This is the payoff of the anisotropic loss ($\lambda = 0.1$), which concentrates accuracy along the query direction at the expense of off-axis components. The symmetric 4-bit query recovers some of this advantage (0.0035 → 0.0067), showing that query quantization is now the dominant noise source at small angles. Even so, OSQ symmetric still beats TQ @1-bit by 1.2–1.4× through 10°. The tradeoff is visible at wider angles where TQ @1-bit has lower noise than OSQ (0.042 vs 0.049 at 60°): the Hadamard rotation distributes information uniformly across all directions, while OSQ deliberately favors the directions that matter for search.

What about TurboQuant's $Q_\text{prod}$ variant? TurboQuant's inner-product variant ($Q_\text{prod}$) was designed to address exactly this bias, spending $b-1$ bits on the MSE quantizer and 1 bit on a QJL sketch of the residual to produce a provably unbiased estimator. At 1-bit $Q_\text{prod}$ is not viable (0 bits for MSE), so the minimum is 2-bit. But for ranking, the cure is worse than the disease: $Q_\text{prod}$ trades ranking-irrelevant bias for ranking-relevant noise. At 60°, $Q_\text{prod}$'s debiased noise is consistently higher than MSE-only at the same total bit width, 0.031 vs 0.025 at 2-bit and 0.011 vs 0.007 at 4-bit, because each bit spent on QJL correction would have been better spent on quantization. Since search cares only about ranking, MSE-only is the better choice. The bias is harmless and the extra quantization bit reduces the noise that actually matters.

The picture changes on shifted data, where centroid centering gives OSQ a decisive advantage. At 0° the debiased noise drops to 0.0008, which is 9× lower than TQ @1-bit's 0.0073, and 7× lower than TQ @4-bit's 0.0054. Centering removes the dominant shared component before quantization, letting the quantizer focus its bits on the information-bearing residual. TurboQuant's data-oblivious rotation cannot exploit this structure. The advantage persists through 20° (OSQ 0.0041 vs TQ @1-bit 0.012) and only narrows at wide angles (60°: OSQ 0.022 vs TQ @1-bit 0.043), where OSQ remains competitive.

On shifted data, OSQ at 1-bit per document (debiased noise 0.001) beats TurboQuant at 4-bit per document (debiased noise 0.006): better ranking accuracy at over 5× less storage (768 bits vs 4,096 bits, since TQ pads 768→1024 for the Hadamard transform). This is the payoff of the data-dependent design: centering and anisotropic interval refinement extract structure that a data-oblivious rotation cannot.

TQ @4-bit MSE is consistently the lowest-noise option on zero-mean data (debiased 0.005-0.008 across all angles), but at 5× the storage cost per document. On shifted data it is actually substantially worse than OSQ symmetric for angles less than 20°.

Head-to-head: throughput

Throughput is where the uniform grid constraint really shines. Here are the throughput figures on $d = 768$, 10k documents, Apple M2 Max, 100 repetitions:

OSQ's symmetric kernels are 10–40× faster than TurboQuant!

We made a fair effort to optimize both implementations to use ARM NEON instructions effectively, but do not claim these are optimal. The key techniques:

The 1-bit kernel reduces to popcount(a AND b) via NEON's vcntq_u8, processing 32 bytes per iteration with dual accumulators for pipeline parallelism. For $d = 768$ the entire packed vector is 96 bytes, a single pass yields 7 ns/doc.

The 2-bit kernel decomposes each 2-bit index into two bit-planes (precomputed at quantize time), reducing the dot product to 4 AND+popcount passes over the same 96-byte planes: $\sum(2x_1+x_0)(2y_1+y_0) = 4 \cdot \text{pc}(x_1 \wedge y_1) + 2 \cdot \text{pc}(x_1 \wedge y_0) + 2 \cdot \text{pc}(x_0 \wedge y_1) + \text{pc}(x_0 \wedge y_0)$. At 14 ns/doc this is 2× the 1-bit time rather than the naive 4× because all four plane pairs share the same data loads, each 96-byte plane is read once and reused across passes.

The 4-bit kernel uses direct NEON nibble multiply with vandq/vshrq to split packed bytes into lo/hi nibbles, multiply, and accumulate via vpaddlq_u8 widening adds. At 22 ns/doc, this is faster than the 16-popcount bit-plane alternative ($4^2 = 16$ plane combinations).

A mixed 4×1 kernel is the production workhorse. It precomputes the 4-bit query's 4 bit-planes at quantize time (each 96 bytes in the same 1-bit packed layout as the document). Per-document scoring is then 4 AND+popcount passes, i.e. the RaBitQ decomposition: $\sum \text{idx4}_i \times \text{idx1}_i = 8 \cdot \text{pc}(\text{plane3} \wedge \text{bits1}) + 4 \cdot \text{pc}(\text{plane2} \wedge \text{bits1}) + \ldots$ At 14 ns/doc this is 21.3× faster than TurboQuant's 1-bit path at 3/4 the document storage.

TurboQuant's bottleneck is the data-dependent gather: each coordinate requires a scalar load from the centroid table to build a NEON float vector. The arithmetic (FMA) is essentially free in comparison.

Conclusion

TurboQuant is a theoretically elegant construction that builds directly on the OSQ formulation. The provable MSE bound, the unbiased inner-product estimator, and the clean data-oblivious design are real contributions. For applications requiring calibrated scores (not just rankings), or running on GPU hardware where gather operations are cheap, TurboQuant's architecture is well-motivated. The calibration-free design is also a natural fit for settings where quantization must happen on the fly with zero training overhead, KV cache compression during LLM inference is a prime example. There, every vector is quantized once as it enters the cache and discarded after the forward pass, so there is no opportunity to amortize a per-vector coordinate descent. A fixed codebook derived from the known post-rotation distribution is exactly the right tool: rotate, snap, store.

But for CPU-based vector search, the setting where Elasticsearch and most operational systems execute queries, the empirical picture is clear across all three axes:

MSE: TurboQuant's advantage comes from the Hadamard rotation, not the Lloyd-Max centroids. OSQ with the same rotation matches TurboQuant at 1–2 bits and comes within 1.1× at 3–4 bits. OSQ's sparse preconditioner already provides this benefit without padding overhead.

Dot-product accuracy: After removing ranking-irrelevant multiplicative bias (including TQ's $2/\pi$ scale factor at 1-bit), OSQ has 1.2–1.4× lower ranking noise than TQ @1-bit at small angles on zero-mean data even with a quantized query and without the 25% pad, thanks to the anisotropic loss concentrating accuracy along the query direction. On shifted data, the regime that matters in practice because embeddings typically have a non-zero mean, centering amplifies the advantage further: debiased noise of 0.0008 at 0° vs TQ @1-bit's 0.0073 and even TQ @4-bit's 0.0054. OSQ at 1-bit beats TurboQuant at 4-bit on ranking accuracy at less than 1/5 the storage. TurboQuant's $Q_\text{prod}$ variant addresses bias explicitly but trades it for higher noise, making MSE-only the better choice for search.

Throughput: 10–40× faster symmetric scoring, with the mixed 4-1 kernel at 14 ns/doc versus TurboQuant's 293 ns/doc using NEON intrinsics. This reflects a fundamental architectural divide between integer arithmetic and lookup-table gather, not a constant factor that disappears with batching.

The uniform grid, far from being a compromise, turns out to be the right trade: it sacrifices a theoretical MSE margin that almost vanishes under equivalent rotation, and in return unlocks the integer-arithmetic pipeline that makes sub-millisecond search at scale practical.

Views don't store data; they re-execute on every read, so results always reflect the current data and the current definition. If you've used views in SQL databases, this will feel familiar. The difference: ES|QL views are engine-level virtual indices stored at the Elasticsearch cluster level, not saved query text that gets expanded client-side. They appear in Kibana autocomplete, support cross-cluster search (CCS), and are governed by dedicated role-based access control (RBAC) privileges.

A simple view

A view can wrap any ES|QL query. Start with a straightforward filter — HTTP 500 errors from the API gateway:

PUT _query/view/error_triage
{
  "query": """
    FROM svc-gateway-*
    | WHERE http.response.status_code >= 500
    | KEEP @timestamp, http.response.status_code, url.path, source.ip
  """
}

Now anyone can write FROM error_triage without knowing the index pattern or filter condition:

FROM error_triage
| STATS error_count = COUNT(*) BY url.path
| SORT error_count DESC

The query is defined once. Consumers reference a name.

Views support full create, read, list, update, and delete (CRUD) via the _query/view REST API.

Update propagation

Say the team decides error_triage should also capture client errors, not just 500s. Update the definition in place:

PUT _query/view/error_triage
{
  "query": """
    FROM svc-gateway-*
    | WHERE http.response.status_code >= 400
    | KEEP @timestamp, http.response.status_code, url.path, source.ip
  """
}

Every dashboard panel, alert rule, and ad-hoc query using FROM error_triage immediately reflects the broader filter. No saved objects to hunt down. No stale copies. Change once, update everywhere.

Nested views

Views can reference other views, enabling layered abstractions. Create views for suspicious IPs and threat intelligence, and then compose them:

PUT _query/view/suspicious_ips
{
  "query": """
    FROM svc-auth-*
    | WHERE event.action == "login" AND event.outcome == "failure"
    | STATS attempts    = COUNT(*),
            first_seen  = FIRST(@timestamp, @timestamp),
            latest_user = LAST(user.name, @timestamp)
        BY source.ip
    | WHERE attempts > 3
  """
}

PUT _query/view/known_threats
{
  "query": """
    FROM threat-intel
  """
}

PUT _query/view/security_overview
{
  "query": """
    FROM suspicious_ips, known_threats
  """
}

FROM security_overview
| WHERE source.ip IS NOT NULL
| EVAL is_known_threat = threat.category IS NOT NULL
| KEEP source.ip, attempts, threat.category, threat.severity, is_known_threat
| SORT is_known_threat DESC, attempts DESC

Security teams query FROM security_overview without knowing the underlying data model. They're also shielded from any changes made to suspicious_ips by its owner; the abstraction boundary is real, not syntactic.

Multisource views with subqueries

A view can wrap any ES|QL query, including multisource compositions, using subqueries in FROM. Each subquery branch queries one service independently (its own filters, its own field normalization), and the results combine automatically:

PUT _query/view/all_errors
{
  "query": """
    FROM
      (FROM svc-gateway-*
       | WHERE http.response.status_code >= 500
       | EVAL service = "gateway",
              error_detail = CONCAT("HTTP ", http.response.status_code::string)
       | KEEP @timestamp, service, error_detail, source.ip),
      (FROM svc-payments-*
       | WHERE transaction.status IN ("failed", "timeout")
       | EVAL service = "payments", error_detail = transaction.status
       | KEEP @timestamp, service, error_detail, source.ip)
  """
}

Consumers just write:

FROM all_errors
| STATS error_count = COUNT(*) BY service
| SORT error_count DESC

Two indices, two independent pipelines, one name. To add a third service later, add a third branch; existing branches don't change, and every downstream dashboard and alert reflects the update automatically. For a deep dive on subquery syntax and what you can do inside each branch, see Three Indices Walk Into a FROM Clause.

How views work under the hood

When you write FROM view_name, ES|QL resolves the view's stored query and executes it inline. Views are re-executed on every read, so results always reflect the current data and the current definition.

Views share a namespace with indices, aliases, and data streams. A view cannot have the same name as any of these (enforced at creation time). This keeps FROM my_name unambiguous regardless of whether the name resolves to a view, an index, or an alias.

Security model

Views are governed by four dedicated RBAC privileges: create_view, read_view_metadata, delete_view, and manage_view. Elasticsearch checks the privileges of the user running the query (invoker security), not the user who defined the view. The user querying a view needs permissions on both the view and its underlying indices.

Kibana integration

Views appear in Discover's ES|QL editor autocomplete alongside indices. ES|QL-based dashboard panels work with views transparently. In the initial Tech Preview release, view management is API-only. A Kibana UI for creating and managing views is planned.

Cross-cluster search

A view's definition can reference remote indices using CCS syntax:

PUT _query/view/cross_cluster_errors
{
  "query": """
    FROM cluster-west:logs-*, cluster-east:logs-*
    | WHERE log.level IN ("error", "crit")
  """
}

Consumers query FROM cross_cluster_errors without knowing which clusters are involved.

Current constraints

In the Tech Preview release, view management is API-only and SET directives can't appear inside view definitions; the caller applies them when querying. Subquery-based views can't be nested inside other multisource FROM expressions. See the views documentation for the full list.

What's next for views

Views today are always fresh; they re-execute on read. Materialized views flip that tradeoff: Pre-compute once, read instantly. Think pre-aggregated rollup views for Service Level Agreement (SLA) dashboards that load in milliseconds instead of scanning raw data on every refresh. A Kibana CRUD UI for views, including a "Save as View" workflow in Discover, is also planned.

Try it

Logical views are available as a Tech Preview. Try them in Kibana Dev Tools or Discover. We'd love your feedback; file a GitHub issue with the ES|QL label.

ES|QL logical views are a Tech Preview feature. Tech Preview features are subject to change and are not covered by the support SLA of GA features. The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Agent Builder in 9.4 makes context the agent's problem, not yours. Skills provide reusable instructions that load on demand, so only what the current task needs is in context. Large result sets go into a conversation context store rather than sitting in the prompt. For long-running tasks, context gets compacted selectively so the agent doesn't drift. Token and turn counts are monitored as the conversation runs. And connectors handle reaching enterprise data where it lives.

The goal of all of it is the same. Load the context you need, when you need it. Internally we've seen this cut token costs by up to 40%, and the agent's context stays reliable across datasets when it would have degraded before.

Getting agents to know what you know

Three context problems show up again and again: managing bloated prompts, executing complex actions, and controlling enterprise data sources.

Firstly, prompts get bloated because every instruction has to live in them. Skills fix that by loading on demand and with fewer input tokens. Skills provide structured descriptions of how an agent should operate and act in a specific task. Agent Builder ships with built-in skills for common data analysis patterns, but the real value is that users can also build custom skills. A security team can encode its triage playbook as a skill. An SRE team can describe how they want root cause analysis to proceed. A developer can encode their API design conventions and error handling patterns. Skills are reusable and shareable across agents, which means a pattern that works for one team's deployment doesn't have to be reinvented by the next.

In practice, this looks like: a team lead defines a "Summarize this incident" skill with the process they care about, the severity classification their org uses, and the output format their runbook expects. Anyone on the team invokes it by typing in the chat input and selecting it from autocomplete. Skills follow the Agent Skills open format, so you can pull them from a shared library, write your own, or craft skills using an agent of your choice.

In internal testing, we found that removing instructions from the agent prompt and placing them in dynamically loaded skills showed a 21 to 39% reduction in input token usage across test datasets. The key architectural improvement is that skills and their associated tools are loaded only when the agent needs them. All other skills stay as lightweight stubs with just a name and description, consuming almost no context.

Chat with your data and act on it too (dashboards, workflows, queries and more): Agentic tasks don't stay simple for long. Agent Builder now has contextual awareness of objects in Kibana. With agentic dashboard creation, a user can describe what they want to see in plain language, and the agent generates a dashboard with panels, visualizations, queries, and everything that's needed. Users can refine it conversationally: "break that out by region," "add a filter for the last 7 days," "swap the bar chart for a line chart."

Dashboards, alerts, and rules also work as inputs. Once a dashboard exists, it can be retrieved from the Agent’s context. This unlocks the "act" side of agents. Once a dashboard or alert is in context, the agent can modify it, extend it, or create new ones. The agent can reason about what the data shows, suggest follow-up analyses, or modify the dashboard based on what it sees. It's a feedback loop: the user describes intent, the agent produces a visual artifact, and both the user and the agent can reason over that artifact together.

For business analysts and operations teams, this collapses the gap between "I have a question about the data" and "I have a dashboard I can share with my team," reducing hours of manual work to a few minutes of conversation.

Lastly, using enterprise data for context creates governance you didn't ask for. Connectors close the loop for data that lives outside Elastic. We added prebuilt OAuth-based connectors for sources like Google Drive, Salesforce, and Slack. The design principle here is worth calling out: data stays at the source. The agent searches data via the connector with the user's own permissions enforced. Agents do not accumulate copies of enterprise data in new locations just to be able to answer questions about it.

This matters more than it might seem. Enterprise data governance isn't just a compliance checkbox; it's a load-bearing infrastructure that most teams don't notice until it fails. When agents start routing around it, accumulating copies in vector stores and context windows, you've quietly created a new class of data sprawl that your security team didn't sign off on, and your audit logs don't capture. The connector approach eliminates this risk by constraint: if the data never moves, it can't end up somewhere it shouldn't. The user's permissions travel with every query because the query goes to the source, not to a cached copy. You get agents that are genuinely useful on enterprise data.

Ensure agents don't outgrow the context window

Giving agents too much context creates a new problem. A security analyst investigating a complex threat might pull in dozens of alerts, correlate across multiple indices, and go back and forth with the agent for twenty or thirty turns. At some point, you're pushing past what the context window can hold and degrading the quality of the model's responses. The problem is that each retrieval call adds latency to the user's request and pushes infrastructure costs higher, and a single user interaction can trigger dozens of these calls.

We built a context store for retrieval results. As the agent retrieves data from indexes, the results can grow large and crowd the context window. We introduced a temporary store that holds the results of a query in an in-memory “file store” and only pulls the results into the active context when needed. This allows for conversations to extend and deal with multiple related data sets without blowing out the context. We are also optimizing the retrieval results themselves, applying top snippets retrieval, which demonstrated a 27 to 34% reduction in token usage.

We also added intelligent context compaction for longer interactions: As a conversation progresses, the agent manages what stays in the active context and what gets compressed into a summary that can be retrieved if needed. This isn't a simple truncation; it's selective compaction that preserves the information most likely to matter for the next turn.

This enables agents to handle larger result sets, more complex queries, and longer conversations without the token cost scaling linearly with every turn. With context compaction agents, the context window remains within a limit even for chats with 30 or more turns, rather than quickly ballooning to max size.

For teams running multi-step investigations or summarizations, this is the difference between an agent that stays coherent through turn thirty and one that starts contradicting itself at turn twelve.

Monitoring: In 9.4, we also shipped monitoring for agents to track token usage. With an API available to monitor conversation turns and tool calIs. This matters because agents aren't static. Their behavior shifts based on the context they receive and the tools they call, and without visibility into those patterns, optimizing cost and performance is guesswork.

Agentic consumption model

To support these new capabilities, we're introducing an agent pricing model that directly aligns the value users gain from their agents and how they scale. Agent Builder usage will be measured by Executions. Executions are free for the first 1,000 each month in Elasticsearch and 10,000 in Elastic Security and Observability projects.

An Agent Builder execution represents a completed round of interaction with the agen. In most cases, sending a chat message and receiving a successful response from the agent counts as one execution. For messages that demand significant processing, it will be calculated as multiple executions based on the total number of input tokens required, grouped into 50,000 input token units. For example, a deep investigative task that requires 130,000 input tokens will be billed as 3 executions. This model ensures your consumption aligns with the value your agents deliver and becomes more cost-effective as your agents achieve greater context efficiency.

Where are we going with agents

Agents that can optimize context over operational data need the same kind of careful context engineering that we've spent years applying to search relevance. Getting the right information in front of the model at the right time and at the right level of detail is the new retrieval problem. These capabilities are foundational towards enabling agents that are more reliable, scalable, and cost-efficient as they scale.

Get started with an Elastic Cloud Trial, and check out the documentation here. For existing customers, Agent Builder is available in Cloud Serverless and on the Enterprise Tier in Elastic Cloud Hosted and self-managed.

From architecture to implementation

Part 3 described the control plane architecture: reverse matching as a lookup primitive, policy documents that separate match from action, and cascading transformations that compose multiple policies into a single execution plan. This post goes hands-on with the Elasticsearch feature that powers the policy lookup: the percolator query.

The percolator is a natural fit for governance because it inverts the direction of search in exactly the way a control plane needs. This post walks through the implementation step by step, starting with a clear explanation of what the percolator does and why it matters, and then moving through index design, policy storage, query-time evaluation, and multi-policy composition.

How normal search works

In an ecommerce system, you may have hundreds of thousands or millions of product documents containing fields such as title, category, and price. When a user searches for matching documents, you're asking Elasticsearch to compare the user’s search string against one or more stored fields in these product documents. Elasticsearch's default analyzer, the standard analyzer, lowercases text and splits it into tokens. A search for “oranges” matches “Oranges” because of lowercasing. With a language-aware analyzer that includes stemming, it also matches “orange” because both forms reduce to the same stem. For example, the following match query returns documents that have “orange” or “oranges” in their “title” field.

POST products/_search
{
  "query": {
    "match": {
      "title": "oranges"
    }
  }
}

So for the above query, Elasticsearch returns the product documents whose title field matches “oranges”, which could include results such as “Orange Fruit Spread”, “Orange Juice”, “Juicy oranges”, “Orange Marmalade”, and so on. The key point to remember is that Elasticsearch is commonly used to compare a search string against documents and to return the documents that match the search string.

The governance problem: Finding relevant policies before searching for products

As established in Parts 1 through 3, a governed search system does not send the user's search string directly to the product catalog. First, it checks whether any policies apply to that search string.

A merchandiser has decided that when someone searches for exactly "oranges", results should be restricted to the Oranges category, eliminating orange juice, orange marmalade, and orange soda. That business decision is stored as a policy. When a user types "oranges", the control plane needs to find that policy, read its instructions, and modify the search against the product catalog accordingly. In order to do this, the control plane needs to figure out which stored policies are relevant for this search string.

An enterprise deployment might have hundreds or thousands such policies. Checking them one by one with if/else logic is the application-layer anti-pattern described in Part 2. What we need is a way to store all of those policies in an index and instantly find the ones that match a given search string. This is where the percolator comes in.

Flipping the direction: The percolator

We previously mentioned that in a normal search, Elasticsearch is commonly used to compare a search string against documents and to return the documents that contain that search string.

The percolator inverts this. With a percolator, you have an index where each document stores a query pattern, and then an incoming search string is checked against these stored queries to determine which of these stored query patterns has triggered.

For governance, the "stored query patterns" are policies. Each policy contains a pattern that describes the kind of search string it should match. For example, does the search string exactly match “oranges”, or does the search string contain “olive oil”? The incoming string is the user's search text, which arrives at query time and needs to be checked against all stored policy patterns. This is covered in a related PRISM video at 4:09.

Step by step: How a search for "oranges" finds its policy

The policy

A merchandiser has authored a policy that matches if a user searches for exactly "oranges" without any other words. Once the percolator matches, the remainder of the document includes the rules that the control plane will use to build the Product query; in this example, one of the rules is to restrict (filter) results to the Fruits category.

{
  "percolator": {
    "match_phrase": { "query": "START oranges END" }
  },
  "rule_type": "filter",
  "rule_args": {
    "filters": [
      {
        "field": "categories",
        "values": ["Fruits"],
        "mode": "hard_filter",
        "on_conflict": "soft_boost",
        "on_conflict_boost_weight": 1.0
      }
    ]
  },
  "priority": 0,
  "enabled": true
}

The percolator field contains the pattern that defines when this policy should fire. In this case, it matches the phrase "START oranges END". The rule_type and rule_args fields define what the policy should do when it fires. The START and END tokens are boundary markers, which we will explain shortly.

You can see how a policy is authored in the PRISM Studio UI at 2:52 of the related PRISM video.

The user searches

A shopper types "oranges" into the search bar.

The control plane checks for matching policies

Before searching the product catalog, the control plane intercepts the user search string, wraps it in boundary markers, and sends it to the percolator:

POST policies/_search
{
  "query": {
    "percolate": {
      "field": "percolator",
      "document": {
        "query": "START oranges END"
      }
    }
  }
}

The string "START oranges END" is checked against all stored policy patterns. Internally, Elasticsearch runs the stored policy patterns against this string and returns the ones that match. That's the percolator. The user's search string was checked against all stored policy patterns, and the ones that matched were returned. No if/else chains. No sequential evaluation. The index handles the matching.

The control plane applies the policy

The control plane reads the matched policies’ actions. The above policy instructs the control plane to restrict results to the Fruits category. The control plane builds the final Elasticsearch query against the product catalog as follows:

POST products/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "title": "oranges" } }
      ],
      "filter": [
        { "terms": { "categories": ["Fruits"] } }
      ]
    }
  }
}

The user searched for "oranges”. The product catalog receives a query for "oranges" constrained to the Fruits category. Because of this constraint, orange juice, orange marmalade, and orange soda are excluded.

Why "orange marmalade" does NOT trigger the oranges policy

Suppose a different user searches for "orange marmalade”. The control plane wraps the string and percolates: "START orange marmalade END". The oranges policy's pattern is match_phrase: "START oranges END". The oranges policy does not match and therefore the policy isn’t applied, and the results aren’t constrained to the Fruits category.

This is the purpose of the START and END boundary markers. Without them, a policy that matches on the word "oranges" could accidentally fire on a query like "orange marmalade". By wrapping the user's search string with START and END and including those markers in the policy's pattern, we ensure that the policy only fires when "oranges" is the complete search string, without any other words. This matches both the shoppers and the merchandiser's intent.

A second policy: "olive oil" on the stemmed field

Not every policy needs an exact string match. The “olive oil” policy matches on a stemmed field, so it fires regardless of minor word-form variations:

{
  "percolator": {
    "bool": {
      "should": [
        { "match_phrase": { "query.stemmed": "START olive oil END" } }
      ]
    }
  },
  "rule_type": "filter",
  "rule_args": {
    "filters": [
      {
        "field": "categories",
        "values": ["Olive oils"],
        "mode": "hard_filter",
        "on_conflict": "soft_boost",
        "on_conflict_boost_weight": 1.0
      }
    ]
  },
  "priority": 300,
  "enabled": true
}

This policy's pattern matches against query.stemmed instead of query. When the user's search string arrives, it’s stored in both a query field (the exact text) and a query.stemmed field (analyzed with a stemming analyzer that reduces words to their stems, so "olives" and "olive" both reduce to the same stem, as do "oils" and "oil"). The policy's pattern is checked against the stemmed version of the string, so it fires regardless of minor word-form variations.

The START and END boundary markers work on the stemmed field, as well, ensuring this policy only fires when "olive oil" is the entire search string, not when it appears as part of something longer.

The rest of this post covers the implementation details that make this production-ready: the index mapping that supports both matching modes, how highlights drive phrase removal and consumed phrase tracking, and how multiple conflicting policies compose into a single execution plan.

The policy index mapping

The policy index needs a percolator field to hold stored query patterns and a text field that mirrors the structure of the incoming search string the percolator will match against. The mapping below is simplified for clarity. A production deployment is more complex, using custom analyzers to handle boundary markers, variable pattern matching (for example, recognizing that "under $4" contains a currency value), and other kinds of analysis.

PUT policies
{
  "mappings": {
    "properties": {
      "percolator": {
        "type": "percolator"
      },
      "query": {
        "type": "text",
        "fields": {
          "stemmed": {
            "type": "text",
            "analyzer": "stemming"
          }
        }
      },
      "rule_type": { "type": "keyword" },
      "rule_args": { "type": "object", "enabled": false },
      "priority": { "type": "integer" },
      "enabled": { "type": "boolean" }
    }
  }
}

The index is named policies because each document represents a complete governed policy as defined in Part 2. This includes match criteria, action, priority, and metadata. The rule_type and rule_args fields contain the action component of the policy, which contain the instructions that the control plane will use to compose the query for execution against the product catalog.

The query field is the string that the percolator matches against. It has two variants: an exact version and a stemmed version. When the user's search string arrives, it’s placed into this field in the temporary in-memory index. Policies that match on query see the exact string; policies that match on query.stemmed see the stemmed version.

Percolating with highlights, filtering, and sorting

The simple examples above showed minimal percolation requests. In practice, the control plane adds highlighting, filters disabled policies, and sorts by priority:

POST policies/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "percolate": {
            "field": "percolator",
            "document": {
              "query": "START olive oil END"
            }
          }
        },
        {
          "term": { "enabled": true }
        }
      ]
    }
  },
  "highlight": {
    "fields": {
      "query": {
        "matched_fields": ["query.stemmed"]
      }
    }
  },
  "sort": [
    { "priority": { "order": "desc" } }
  ]
}

The highlight configuration uses "query" as the field key with "query.stemmed" in matched_fields. This tells Elasticsearch's unified highlighter to return highlights on the parent query field but to also consider matches from the query.stemmed subfield when determining which tokens to highlight. This is what allows a policy that matches on the stemmed field to still produce accurate highlight spans on the original text, which the control plane needs for phrase removal and consumed phrase tracking.

The enabled: true filter ensures that disabled policies are skipped. The sort on priority ensures that higher-priority policies are returned first, so the control plane can process them in the correct order for cascading transformations. The highlight field is the most important addition; it tells us exactly which words in the user's search string triggered each match.

The response for an "olive oil" search may look as follows:

{
  "hits": {
    "hits": [
      {
        "_id": "en_2c3021c8",
        "_source": {
          "rule_type": "filter",
          "rule_args": {
            "filters": [
              {
                "field": "categories",
                "values": ["Olive oils"],
                "mode": "hard_filter",
                "on_conflict": "soft_boost",
                "on_conflict_boost_weight": 1.0
              }
            ]
          },
          "priority": 300
        },
        "highlight": {
          "query": ["START olive oil END"]
        }
      }
    ]
  }
}

Why highlights matter

Notice the highlight in the response: "<em>START olive oil END</em>". Elasticsearch is telling us exactly which words in the user's search string caused the policy to match. This isn’t cosmetic. The highlight metadata drives two critical downstream behaviors:

Phrase removal. Some policies need to remove the matched text from the search string before constructing the product catalog query. For example, a policy that matches on "cheap" removes that word and converts it into a price filter instead. The highlight identifies exactly which span of the search string the policy matched, so the system knows what to remove.

Consumed phrase tracking. As described in Part 3, when multiple policies match the same search string, a higher-priority policy might remove words that a lower-priority policy also matched on. By comparing each policy's highlight against the current (evolving) search string, the system can detect that a phrase has been consumed and skip the lower-priority policy. This prevents double-processing and ensures deterministic behavior.

You can learn more about how highlighting works in this article.

From percolation to execution plan

The percolator returns a set of matching policies. But as Part 3 described, the lookup is only half the story. The other half is composing those matches into a coherent execution plan. Here’s what that looks like for a concrete query.

Worked example: "Cheap chocolate" during a Christmas campaign

Suppose the system has two active policies: the "Cheap chocolate" policy (priority 210) and the "Christmas chocolates" policy (priority 300), both described in detail in Part 3.

Step 1: Percolate. The user searches for "cheap chocolate." The control plane wraps the search string as "START cheap chocolate END" and sends it to the percolator. Two policies match: The "Cheap chocolate" policy's pattern matches on the phrase "cheap chocolate"; and the "Christmas chocolates" policy's pattern matches on "chocolate" via the stemmed field.

Step 2: Sort by priority. The percolator returns both policies, sorted by priority in descending order. The “Christmas chocolates” policy (300) is processed first, followed by the “Cheap chocolate” policy (210).

Step 3: Apply the cascading transformation. This is the initial state → [Policy A] → state' → [Policy B] → state'' → execution plan model from Part 3.

The “Christmas chocolates” policy (priority 300) applies first:

• Adds a category hard filter: "Christmas foods and drinks," "Christmas sweets".
• Adds a price filter: less than $7.
• Adds a category soft boost: "Advent calendars" (3x).

The “Cheap chocolate” policy (priority 210) applies next against the modified state:

• Attempts to add a category hard filter: "Chocolates," "Milk chocolates"; but the Christmas policy already set this field with on_conflict: override, so the Cheap chocolate categories are dropped.
• Attempts to add a price filter: $2, the Christmas policy set on_conflict: restrict for price, and $2 is more restrictive than $7, so $2 wins.
• Removes "cheap" from the search string.

Step 4: Build the Elasticsearch query. The control plane assembles the execution plan into a single Elasticsearch query against the product catalog:

POST products/_search
{
  "query": {
    "function_score": {
      "query": {
        "bool": {
          "must": [
            { "match": { "title": "chocolate" } }
          ],
          "filter": [
            { "terms": { "categories": ["Christmas foods and drinks", "Christmas sweets"] } },
            { "range": { "price": { "lt": 2 } } }
          ]
        }
      },
      "functions": [
        {
          "weight": 1
        },
        {
          "filter": { "terms": { "categories": ["Advent calendars"] } },
          "weight": 3
        }
      ],
      "score_mode": "sum",
      "boost_mode": "multiply"
    }
  }
}

The original search string was "cheap chocolate”. The query that reaches the product catalog is a governed, intent-aware retrieval plan: The word "cheap" has been consumed and converted into a price constraint, results are restricted to Christmas seasonal categories, Advent calendar products receive a ranking boost, and the price ceiling reflects the more restrictive value from the lower-priority policy. Every transformation is deterministic, traceable, and explainable.

For a quick overview about how these multipliers interact with the base BM25 score, see 8:45 in the related PRISM video, where we briefly discuss multiplicative boosts.

Why this scales

The percolator is efficient for this use case because of the asymmetry: An enterprise ecommerce system might have millions of products but only hundreds or thousands of governance policies. The percolator is checking one incoming search string against that set of stored policy patterns, not scanning the full product catalog. The cost is proportional to the number of policies, and Elasticsearch applies internal optimizations (indexing terms from stored query patterns, short-circuiting Boolean logic) to keep matching fast.

Adding a new policy is just indexing a new document. Disabling one is a field update. No code changes, no deploys, no restarts.

From lookup to governed retrieval

The percolator provides the fast reverse-matching primitive that makes the control plane architecture from Part 3 practical at scale. Policies are data which are stored and indexed, and efficiently matched against incoming search strings. The control plane composes matching policies into a governed execution plan through the cascading transformation and per-field conflict resolution described in Part 3. And the retrieval engine executes the governed execution plan against the product catalog.

The result is a system where a merchandiser can author a new policy without touching application code, test it against representative queries, promote it to production, and immediately see the effect. The percolator makes the policy lookup fast; the control plane makes the policy composition deterministic; and the governed workflow makes the whole process safe.

What's next in this series

The next post in this series extends the governed control plane into new territory. It introduces a multi-tier search architecture, explaining how to orchestrate strict, relaxed, and semantic retrieval while maintaining stable pagination and facets.

Put governed ecommerce search into practice

The percolator-based control plane described in this post, from index mappings and boundary markers to highlight-driven phrase tracking and cascading policy composition, was built by Elastic Services Engineering as part of our repeatable ecommerce search accelerators. Every query example and policy structure shown here comes from a working system validated against enterprise-scale product catalogs.

If you want to implement a governed, policy-driven control plane on Elasticsearch, Elastic Services can get you there faster. Contact Elastic Professional Services.

Join the discussion

Have questions about search governance, retrieval strategies, or ecommerce search architecture? Join the broader Elastic community conversation.

What is recall?

Recall measures on a scale from 0 to 1 how many of the documents that your users actually want appear somewhere in your search results. If a query should surface three products and your search returns only two of them in the top 10, recall@10 = 0.67 for that query. It’s a set-based metric: It doesn’t care about the position of the relevant documents within those k results. A relevant document in position 10 counts the same as one in position 1. Having a high recall means that you’re not losing relevant results.

The diagram shows two sets: all relevant documents (left) and what BM25 actually retrieved (top 10, right). Only the intersection counts toward recall, prod_1 and prod_2 were found, while prod_3, prod_4, and prod_6 were missed entirely. Result: Recall@10 = 2/5 = 0.40.

Prerequisites

Let's get down to business to better understand how recall works. This demonstration uses Python. You can follow along with it on the companion notebook (notebook.ipynb), where every code block is a cell ready to run.

The code provided uses the following:

• Elasticsearch 9.3+
• Python 3.10+

pip install elasticsearch pandas plotly python-dotenv

• A .env file with your Elasticsearch credentials

ELASTICSEARCH_URL=https://your-cluster-url
ELASTICSEARCH_API_KEY=your-api-key

The dataset

We’ll use a product catalog of 1,000 products, spanning categories such as footwear, electronics, tools, and more.

Each document has four fields:

The dataset is loaded from dataset.csv.

The power and limits of lexical search

BM25 is the default ranking algorithm in Elasticsearch and most search engines. It scores documents by how often your query terms appear in them, adjusted for document length and the frequency of those terms across the entire index. You get analyzers on top: lowercase normalization, stemming, and stopword removal. A query for "running shoes" will match "Running Shoes" and likely "run" as well.

This works well for a large class of queries:

• "running shoes" immediately matches products with those exact tokens in the title.
• "bluetooth speaker" surfaces portable audio products because the tokens appear verbatim.

The results are deterministic and explainable: A document ranks highly because the query terms appear in it. Debugging relevance is straightforward.

Where it breaks

Now let’s try these queries against the same catalog:

• "skincare routine": The word "routine" doesn’t appear in any product title. BM25 can partially match on "skincare," but face serums, body oils, and moisturizers are described using terms like "vitamin C," "retinol," or "brightening," none of which overlap with the query. Products that form a complete skincare routine are scattered across the index with no shared token to anchor them.

ID: B06XX6DS3P, Score: 9.0552, Title: Replenix Retinol Smooth + Tighten Body Lotion - Collagen-Boosting, Regenerating Anti-Aging Body Cream, Reduces Appearance of Stretch Marks, 6.7 oz.

  ID: B08XMPKJ1L, Score: 5.2699, Title: Bio-Oil Skincare Body Oil (Natural) Serum for Scars and Stretchmarks, Face and Body Moisturizer Hydrates Skin, with Organic Jojoba Oil and Vitamin E, For All Skin Types, 6.7 oz

  ID: B01CY764KQ, Score: 5.0057, Title: Nike Up Or Down Men Deodorant - Pack of 2 | Long-Lasting Fragrance, Body Spray Combo for Men | Deodorant for Active Living | Nike Men's Deo Set | Ultimate Odor Protection | Grooming Essentials | Signature Nike Scent | High-Performance Men's Deodorant

• "pet travel accessories": This is a use-case grouping, not a product category. A dog sling carrier, a pet car seat, and a travel crate are all relevant, but their descriptions talk about portability, safety, and comfort rather than "travel accessories." BM25 matches "pet" broadly but has no signal to distinguish travel-specific products from the rest of the pet catalog.

ID: B0BVV7BKTW, Score: 7.4371, Title: Large Foldable Travel Duffel Bag with Shoes Compartment

ID: B07TNPHYNV, Score: 6.6455, Title: 40 Pieces Christmas Bronze Jingle Bells Craft Small Bells

ID: B08R8FRW53, Score: 6.6335, Title: CUBY Dog and Cat Sling Carrier
ID: B08QMCQYGM, Score: 6.5259, Title: YTFGGY Whiteboard Pinstripe Tape 6 Rolls 1/8"
ID: B0CP3LQSWM, Score: 6.2994, Title: Portable Dog Water Bottle 32 Oz

This is a recall problem. The relevant documents exist in your index. BM25 just cannot find them because the user's words and the document's words do not match closely enough.

Adding synonyms helps for known cases. But you cannot enumerate every way a user might express an intent. That is where vectors come in.

Why you should measure recall

Before fixing a problem, you need to quantify it.

Recall@k measures how many of the documents that your users actually want appear somewhere in your search results. Formally:

Recall@k = (relevant documents found in top k) / (total relevant documents)

Precision@k measures the top k results and how many are actually relevant:

Precision@k = (relevant documents in top k) / k

High precision means that the results you do return are good. In ecommerce, missing a relevant product (low recall) is often worse than showing a slightly imperfect result (lower precision), because a hidden product is a lost sale.

Elasticsearch's rank_eval API lets you measure both systematically. You provide a list of queries, each with a set of rated documents, and Elasticsearch computes the metrics for you across all queries.

Setting up the evaluation

The rank_eval API needs a ratings dataset: a mapping of queries to the documents that are relevant for each one, along with a relevance grade (0 = not relevant, 1 = relevant, 2 = highly relevant).

In the notebook, this is the judgments list:

judgments = [
    # Query 1: "running shoes" BM25 handles well (tokens appear in product titles) 
    {"query_id": "q1", "doc_id": "B09NQJFRW6", "grade": 2, "query": "running shoes"},
    {"query_id": "q1", "doc_id": "B08JMD4LMM", "grade": 2, "query": "running shoes"},
    {"query_id": "q1", "doc_id": "B08VRJ6F2Q", "grade": 2, "query": "running shoes"},
    {"query_id": "q1", "doc_id": "B07S8NRRWR", "grade": 2, "query": "running shoes"},
    {"query_id": "q1", "doc_id": "B01HD620I8", "grade": 2, "query": "running shoes"},
    {"query_id": "q1", "doc_id": "B07DX86321", "grade": 2, "query": "running shoes"},
    {"query_id": "q1", "doc_id": "B0968YVLQ8", "grade": 1, "query": "running shoes"},
    {"query_id": "q1", "doc_id": "B093QJ39ZS", "grade": 1, "query": "running shoes"},
    {"query_id": "q1", "doc_id": "B096FGSC39", "grade": 1, "query": "running shoes"},
    {"query_id": "q1", "doc_id": "B01GVQWVV2", "grade": 1, "query": "running shoes"},

    # Query 2: "skincare routine" intent-based, "routine" never appears in product titles
    {"query_id": "q2", "doc_id": "B08XMPKJ1L", "grade": 2, "query": "skincare routine"},
    {"query_id": "q2", "doc_id": "B0BN3WQB92", "grade": 2, "query": "skincare routine"},
    {"query_id": "q2", "doc_id": "B0BT7B7P5T", "grade": 2, "query": "skincare routine"},
    {"query_id": "q2", "doc_id": "B00NPA2WEY", "grade": 2, "query": "skincare routine"},
    {"query_id": "q2", "doc_id": "B06XX6DS3P", "grade": 1, "query": "skincare routine"},
    {"query_id": "q2", "doc_id": "B07PDRD1KT", "grade": 1, "query": "skincare routine"},
    {"query_id": "q2", "doc_id": "B074J7869B", "grade": 1, "query": "skincare routine"},
    {"query_id": "q2", "doc_id": "B08JV31QW4", "grade": 1, "query": "skincare routine"},
    {"query_id": "q2", "doc_id": "B00K3TVJMQ", "grade": 1, "query": "skincare routine"},

    # Query 3: "study desk setup" intent-based, products are desks/stands/organizers
    {"query_id": "q3", "doc_id": "B08CS35J2T", "grade": 2, "query": "study desk setup"},
    {"query_id": "q3", "doc_id": "B09B3LFDXJ", "grade": 2, "query": "study desk setup"},
    {"query_id": "q3", "doc_id": "B07W58LMND", "grade": 1, "query": "study desk setup"},
    {"query_id": "q3", "doc_id": "B0CHYDX91L", "grade": 1, "query": "study desk setup"},

    # Query 4: "pet travel accessories" use-case grouping, products are carriers/crates/seats
    {"query_id": "q4", "doc_id": "B08R8FRW53", "grade": 2, "query": "pet travel accessories"},
    {"query_id": "q4", "doc_id": "B01MYUYX33", "grade": 2, "query": "pet travel accessories"},
    {"query_id": "q4", "doc_id": "B003C5RKE4", "grade": 2, "query": "pet travel accessories"},
    {"query_id": "q4", "doc_id": "B09GF8GBF6", "grade": 1, "query": "pet travel accessories"},
    {"query_id": "q4", "doc_id": "B0CP3LQSWM", "grade": 1, "query": "pet travel accessories"},
]

The mix is intentional: q1 is a query that BM25 handles well (exact tokens in product titles), while q2, q3, and q4 are intent-based queries where the user's intent is expressed as a concept rather than specific product keywords.

Measuring BM25 baseline recall

First, set up the Elasticsearch client and index the raw text data:

import os
import json
import pandas as pd
import plotly.graph_objects as go
from elasticsearch import Elasticsearch, helpers
from dotenv import load_dotenv

load_dotenv()

es = Elasticsearch(
    os.getenv("ELASTICSEARCH_URL"),
    api_key=os.getenv("ELASTICSEARCH_API_KEY")
)

INDEX_NAME = "ecommerce-products"

Now build the rank_eval request for BM25. Each request in the list combines a query with its ratings:

judgments_df = pd.DataFrame(judgments)

bm25_requests = []
for query_id, query_text in (
    judgments_df[["query_id", "query"]].drop_duplicates().values
):
    relevant_docs = judgments_df[judgments_df["query_id"] == query_id]
    ratings = [
        {"_index": INDEX_NAME, "_id": row["doc_id"], "rating": row["grade"]}
        for _, row in relevant_docs.iterrows()
    ]

    bm25_requests.append({
        "id": query_id,
        "request": {
            "query": {
                "multi_match": {
                    "query": query_text,
                    "fields": ["title", "description"]
                }
            }
        },
        "ratings": ratings,
    })

bm25_eval = {
    "requests": bm25_requests,
    "metric": {"recall": {"k": 10, "relevant_rating_threshold": 1}},
}

bm25_result = es.rank_eval(index=INDEX_NAME, body=bm25_eval)
print("BM25 Recall@10:", bm25_result.body["metric_score"])

Result:

BM25 Recall@10: 0.43

0.43 means that across all four queries, BM25 finds only 43% of the documents it should find. The shortfall is concentrated in the intent-based queries: "skincare routine" misses face serums and body oils because "routine" never appears in product titles, and "pet travel accessories" retrieves off-topic pet products while missing carriers and crates described in terms of portability and safety rather than "travel accessories."

This is our baseline. Now we have a number to beat.

Adding vector search with Jina embeddings

Vector search encodes documents and queries as high-dimensional vectors, a type of vector made up of hundreds or thousands of numerical values, each encoding a specific feature of the data it represents. Documents with similar meaning end up close together in vector space, even if they share no words. "Gym equipment" and "dumbbell set" will be nearby because the concepts are related. I chose Elasticsearch as my vector database because it supports hybrid search, giving me both semantic understanding and keyword precision out of the box.

EIS includes out-of-the-box support for embedding models through its inference API.

Step 1: Using Jina embeddings v5 as an inference endpoint

INFERENCE_ENDPOINT_ID = ".jina-embeddings-v5-text-small"

If your cluster has GPU resources (available in Elastic Cloud and Elasticsearch 9.3+), the embeddings are generated on GPU, which is significantly faster than CPU inference and removes the performance trade-off that historically made vectors expensive at scale.

Why Jina embeddings specifically? jina-embeddings-v5-text is a multilingual model (119+ languages) with a 32,000-token context window and support for task-specific Low-Rank Adaptation (LoRA) adapters. It works well for short product descriptions out of the box. Read more about jina-embeddings-v5-text model here.

Step 2: Create the index with a semantic field

index_mappings = {
    "mappings": {
        "properties": {
            "title": {"type": "text", "copy_to": "semantic_field"},
            "description": {"type": "text", "copy_to": "semantic_field"},
            "brand": {"type": "keyword"},
            "category": {"type": "keyword"},
            "semantic_field": {
                "type": "semantic_text",
                "inference_id": INFERENCE_ENDPOINT_ID,
            },
        }
    }
}

if not es.indices.exists(index=INDEX_NAME):
    es.indices.create(index=INDEX_NAME, body=index_mappings)
    print(f"Created index: {INDEX_NAME}")

The semantic_text field type is the key here. It’s a higher-level abstraction over dense_vector: You point it at an inference endpoint, and Elasticsearch takes care of generating embeddings automatically.

The copy_to property on title and description means content from both fields flows into semantic_field for embedding, so a single vector captures the full product representation.

Step 3: Index the products

def bulk_index(products, index_name):
    actions = []
    for product in products:
        doc_id = product.get("_id")
        source = {k: v for k, v in product.items() if k != "_id"}
        action = {"_index": index_name, "_source": source}
        if doc_id:
            action["_id"] = doc_id
        actions.append(action)

    success, failed = helpers.bulk(es, actions, raise_on_error=False)
    if failed:
        for error in failed:
            print(f"Error: {error}")
    else:
        print(f"Successfully indexed {success} documents")

bulk_index(products, INDEX_NAME)

At index time, Elasticsearch calls the inference endpoint for each document and stores the resulting embedding in semantic_field. No extra code on your side.

Hybrid search: Combining BM25 and vectors with RRF

Adding vectors improves recall, but using vectors alone risks losing precision on exact-match queries; "running shoes" should still rank verbatim matches first. Hybrid search retains the lexical component specifically to preserve that precision.

Hybrid search with Reciprocal Rank Fusion (RRF) keeps the best of both:

• BM25 handles exact and near-exact queries with high precision.
• Semantic search handles intent-based and multilingual queries with high recall.
• RRF combines the two ranked lists into a single ranking.

The RRF formula assigns each document a score based on its rank in each result list:

score = sum(1 / (rank_constant + rank))

A document that ranks highly in both lists gets a higher combined score. The rank_constant controls how much weight lower-ranked documents receive.

hybrid_requests = []

for query_id, query_text in (
    judgments_df[["query_id", "query"]].drop_duplicates().values
):
    relevant_docs = judgments_df[judgments_df["query_id"] == query_id]
    ratings = [
        {"_index": INDEX_NAME, "_id": row["doc_id"], "rating": row["grade"]}
        for _, row in relevant_docs.iterrows()
    ]

    hybrid_requests.append({
        "id": query_id,
        "request": {
            "retriever": {
                "rrf": {
                    "retrievers": [
                        {
                            "standard": {
                                "query": {
                                    "multi_match": {
                                        "query": query_text,
                                        "fields": ["title", "description"],
                                    }
                                }
                            }
                        },
                        {
                            "standard": {
                                "query": {
                                    "match": {
                                        "semantic_field": {"query": query_text}
                                    }
                                }
                            }
                        },
                    ],
                    "rank_window_size": 50,
                    "rank_constant": 5,
                }
            }
        },
        "ratings": ratings,
    })

hybrid_eval = {
    "requests": hybrid_requests,
    "metric": {"recall": {"k": 10, "relevant_rating_threshold": 1}},
}

hybrid_result = es.rank_eval(index=INDEX_NAME, body=hybrid_eval)
print("Hybrid Recall@10:", hybrid_result.body["metric_score"])

Result:

Hybrid Recall@10: 0.75

Hybrid improves substantially over BM25 (0.43) and preserves precision for exact-match queries like "running shoes."

Results: Before and after

Here’s the full comparison across all three approaches:

methods = {
    "BM25 (Lexical)": bm25_requests,
    "Hybrid (BM25 + Vectors)": hybrid_requests,
}

recall_metric = {"recall": {"k": 10, "relevant_rating_threshold": 1}}

comparison_data = []
for method_name, requests in methods.items():
    result = es.rank_eval(
        index=INDEX_NAME,
        body={"requests": requests, "metric": recall_metric}
    )
    comparison_data.append({
        "method": method_name,
        "recall@10": result.body["metric_score"]
    })

comparison_df = pd.DataFrame(comparison_data)
print(comparison_df.to_string(index=False))

Result:

Breaking it down by query:

Conclusion

Throughout this post, we saw that BM25 lexical search is reliable when users type exact queries, but it loses recall when they search by intent rather than keywords. Using rank_eval, we established a reproducible baseline to measure that gap with real numbers. From there, we added a semantic_text field powered by Jina embeddings and ran the evaluation again. The result: Hybrid search improved recall from 0.43 to 0.75 while preserving precision on exact-match queries, though the actual margin will depend on your query mix.

The pattern scales beyond this example: Collect judgments from your users' actual queries, run rank_eval as a baseline, add semantic_text, and measure again. You'll know exactly what improved and by how much.

Next steps

• Dive deeper into recall and vector search: Recall and vector search quantization by Jeff Vestal
• Add reranking for even better precision on the top results
• Explore Elasticsearch hybrid search documentation
• Read more about the rank_eval API

Why query interpretation is often a challenge

Storing policies as code (if/else blocks in the application layer) produces tens of thousands of lines of brittle logic that lacks any indexing for efficient policy retrieval at query time. Iteration is slow (a single query behavior change may require a six-week deployment cycle), accountability is unclear (why did results change?), and business users cannot modify search behavior without engineering involvement. This is shown on the left side in the following image:

Storing policies as data in an Elasticsearch index is shown on the right side of the above image. This approach solves all of the issues associated with hard-coded query resolution logic. However, for this to work, you need a way to quickly determine which policies match the user query and how conflicts should be resolved. This is where the governed control plane comes in.

The control plane pattern

A governed control plane sits between the raw user query and an Elasticsearch retrieval. It receives user text as its input, and its output is an execution plan that includes filters, boosts, and retrieval routing decisions.

A control plane pipeline consists of:

1. User query: A user enters a string of what they’re looking for, such as “oranges” or “gift for grandpa”.
2. Policy lookup: Match the user query against the policy index.
3. Return matching policies: Policies that match the user query are returned from the policy index.
4. Policy application: The control plane analyzes these returned policies and composes matched policies into a single coherent execution plan that includes filters, boosts, overrides, and guardrails and that applies the appropriate retrieval method (for example, lexical versus semantic versus hybrid).
5. Execute: The modified intent-aware Elasticsearch query is passed to the application to be executed against a product catalog index.
6. Explain (optional): In addition to creating a query that provides business and intent-aligned results, the control plane provides an optional explainability payload to show which policies were triggered and how they were combined.

Finding which policies should be applied for a user’s search string requires a fast reverse-matching primitive, which we solve with the percolator query. After retrieving relevant policies, combining multiple matched policies into a unified execution plan requires a judgment framework: priorities, conflict strategies, consumed phrase tracking, and cascading transformations that apply policies in sequence rather than independently. Additionally, the most appropriate retrieval technology needs to be selected (for example, BM25 for “oranges” versus semantic search for “gift for grandpa”).

Policy lookup: Checking the query before searching for products

When a shopper types a query, a search system with a governed control plane doesn’t send that query directly to be executed against the product catalog. First, the query is checked against a set of stored policies and modified to reflect the intent of the query and business priorities.

Policy structure

Each policy is a simple document that defines two things:

• Match criteria: What query text should cause this policy to fire. This could be an exact phrase, a single word, a pattern, or a combination.
• Action: What to do when the policy fires. This could be applying a category filter, excluding products, extracting a price constraint, or changing the retrieval strategy.

The system finds all matching policies, composes them into an execution plan, and only then runs the product search. Taken together, policies act like a knowledgeable store associate who understands what you’re looking for and walks you to the right aisle.

The policy pattern

The first articles in this series introduced examples of policies in action: constraining "oranges" to the produce category, treating "without peanuts" as an exclusion, and routing "gift for grandpa" to semantic retrieval. The key architectural point is that in each case, the query is checked against stored policies before the product search begins. The policies determine what constraints to apply, which text to modify, and which retrieval strategy to use. The query against the product catalog comes after the policies have been applied and a new rewritten query has been created.

Why this is fast

An enterprise ecommerce system might have millions of products but only hundreds or thousands of policies. The policy lookup step is searching against a small curated index, not the full product catalog, and is therefore fast. And because policies are stored as data in their own index, a merchandiser adding a new policy doesn't touch the application code, and an engineer optimizing the product search doesn't touch the policy index. The two concerns evolve independently.

The examples above describe what happens conceptually. Under the hood, the policy lookup is implemented using the Elasticsearch percolator query type, which is purpose-built for this kind of pattern: matching incoming text against a set of stored queries. Part 4 in this series provides a hands-on deep dive into the percolator implementation, including index mappings, boundary markers, and highlight-driven phrase tracking. With the lookup mechanism covered in depth in Part 4, let's turn to what a policy document actually contains and how the control plane composes multiple policies into a single execution plan.

Example policies

Now that we've seen what policies do conceptually, let's look at what they actually contain. The two policies below have been designed to intentionally conflict, which will demonstrate the conflict resolution system described in subsequent sections.

Cheap chocolate

The policy shown below detects if a user has submitted a search containing the phrase “cheap chocolate”. If so, results are restricted to the “Chocolates” and “Milk chocolates” categories. This policy also applies a price filter of $2. Also, notice that this policy has a priority of 210; we’ll come back to this when we discuss conflict resolution in more detail.

The filter mode and conflict strategy settings shown here (hard_filter, soft_boost, restrict, override) are explained in detail in the conflict resolution section below.

When the above policy is activated, a search for “cheap chocolate” respects the price filter of $2 and restricts results to the “Chocolates” and “Milk chocolates” categories. Example results are shown below:

Christmas chocolate

The policy shown below is an example of a policy that one could imagine applying at Christmas. This example restricts results to “Christmas foods and drinks” and “Christmas sweets”, boosts any products that are also in the “Advent calendars” category, and applies a price filter of less than $7 to focus on affordable seasonal items. Additionally, notice that this policy has a priority of 300. We’ll come back to this when we discuss conflict resolution in more detail.

When the above policy is activated without any conflicting policies, a search for “chocolate” respects the price filter of $7, and restricts results to the “Christmas food and drinks” and “Christmas sweets” categories, and boosts any products tagged as “Advent calendars”. Example results are shown below:

Combining matched policies

The policy lookup described above is half the story. The other half is what happens when multiple policies match the same query.

In any nontrivial deployment, a single query will routinely trigger several policies at once. "Cheap chocolate" will match both of the policies that we demonstrated above. Each policy is correct in isolation. The challenge is composing them into a single, coherent execution plan without contradictions, without double-counting, and without one policy silently undoing the work of another.

This isn’t a lookup problem; it’s a judgment problem. The system must decide:

• Order of application: If a negation policy removes "without peanuts" from the query, does the price policy still see the original text or the modified text?
• Filter conflicts: If two policies set different price ceilings, which one wins? Is the loser silently dropped, or does it degrade gracefully into a soft boost?
• Phrase ownership: If two policies both matched on the same word and the first one already consumed it, should the second one still fire?

A naive implementation (apply all matched policies independently, merge the results) breaks as soon as policies interact. The architecture needs an explicit model for how policies compose. The next two sections describe that model: a priority and conflict resolution framework; and a cascading transformation model that makes policy interaction deterministic.

The key insight is that policy application isn’t a set of independent operations; it’s a cascading transformation. Each policy receives the rewrite state produced by all higher-priority policies and transforms it further:

initial state → [Policy A] → state' → [Policy B] → state'' → ... → execution plan

The state carries the rewritten query text, accumulated filters, current intent, and any synonym expansions. A high-priority policy can remove text from the query, and every subsequent policy sees the modified query, not the original. Context accumulates. Order matters.

Precedence and conflict resolution: Determinism matters

The specific conflict strategies are a design choice. Different organizations may resolve conflicts differently, depending on their business requirements. The following approach illustrates the kind of judgment framework a control plane needs. The important thing is not these specific strategies but that the system has explicit, deterministic strategies rather than letting conflicts resolve through unpredictable interactions.

Priority ordering

Policies are sorted by priority (highest first). When multiple policies match the same query, they’re applied in priority order. If two policies try to set the same filter field, the higher-priority policy's declared strategy for that field takes precedence. If there are multiple policies triggered that have the same priority, then the policy with the highest ID is given precedence (as if it were assigned a higher priority); this choice ensures deterministic behavior when conflicts arise.

Per-field resolution, not per policy

A critical design principle: Conflict resolution operates per field (for example, brand, category, or description), not per policy. When two policies produce filters that overlap on specific fields, only those specific fields are affected by the conflict resolution strategy, and the resolution strategy is defined by the highest-priority matching policy. Non-conflicting fields from both policies survive intact.

This matters because the alternative of a per-policy approach would force the system to either accept or reject an entire policy when only one of its fields conflicts.

Per-field resolution preserves the maximum amount of useful constraint information.

Three settings per filter field

Each filter field in a policy has three independent settings:

Filter mode: How the filter is applied when there’s no conflict.

• hard_filter (default): Applied as an Elasticsearch bool.filter clause. This is useful for excluding unrelated products entirely. For example, restricting a search for "oranges" to the produce category eliminates hits such as orange juice and orange marmalade. Non-matching documents are completely excluded from results.
• soft_boost: Applied as an Elasticsearch function_score weight with a configurable boost_weight. Documents that match get a ranking boost, but non-matching documents aren’t excluded. This is useful for something like boosting a brand, without excluding other brands.

Conflict strategy

What happens when a lower-priority policy sets the same field:

• override: This high-priority policy's value wins; the lower-priority value is dropped entirely. Valid for all field types.
• restrict: Take the more restrictive numeric value (for example, the lower ceiling for price__max, the higher floor for price__min). Valid for numeric range fields only.
• merge: Combine both values into a union. Valid for non-numeric fields only.
• soft_boost: Convert the conflicting filter to a function_score weight with a configurable boost_weight instead of a hard filter. For more details on function_score boosting, see Influencing BM25 ranking with multiplicative boosting in Elasticsearch. This is only valid for non-negation fields.

Value: The actual filter value (for example, a categories list, a price threshold).

Strategies by field type: Not all strategies make sense for all field types. For instance, an exclusion is inherently binary, so it cannot be soft-boosted. The following table shows which strategies are available for each field type:

Negation fields cannot be soft-boosted because exclusions are binary. Converting "never show canned foods" to "slightly prefer not-canned-foods" fundamentally changes the semantics; a product from "canned foods" would still appear, just ranked slightly lower, which defeats the purpose of the exclusion.

A concrete example: Searching for "cheap chocolate" during a Christmas campaign

Suppose a merchandiser has created the two policies for chocolate that we previously demonstrated, a lower priority one for cheap chocolate and another higher-priority chocolate-related policy that will be enabled during Christmas. If both of these policies are enabled, then how these are combined depends on the filter mode and conflict strategy of the higher-precedence policy. If both of the previously discussed policies are enabled, they’ll be combined as follows:

This shows two conflicts, one on categories and one on price. It’s worth noting that the query that will be executed after this transformation has the following characteristics:

• Only products from the “Christmas foods and drinks” and “Christmas sweets” categories will be shown.
• Within those categories, if the products are also tagged as being in the “Advent calendars” category, they’ll be boosted up by 3x.
• A price filter for $2 is applied, which came from the lower-priority policy (because the higher-priority policy specified to “Restrict” on conflict).
• The word “cheap” is removed, only returning products matching “chocolate”.

With both of these policies enabled, “cheap chocolate” returns results similar to the image shown below:

Relaxing constraints

Perhaps the retailer doesn’t want to exclude products in the categories of “Chocolates” and “Milk chocolates” during Christmas. The settings on the Christmas policy might have overreached and inadvertently removed categories applied by the “cheap chocolate” policy. This is an example that shows why it might be more desirable to combine lower-priority policies with conflicting higher-priority policies. For example, we could modify the Christmas chocolates promotion so that instead of “Override” on conflict, we do a soft boost. The change to that policy would be as follows:

After this modification, the query rewriter transformation pipeline execution for “cheap chocolate” looks as follows:

With the soft boost on conflict, the conflicting filters are converted into soft boosts rather than being dropped. The query that will be executed on the product catalog after this transformation has the following characteristics:

• Because “On conflict” is specified as “Soft boost” on the higher-priority policy, the conflicts will be converted to boosts as follows:
  • Products from the “Christmas foods and drinks” and “Christmas sweets” categories will have a boost of 1x applied to them.
  • Products from the “Chocolates” and “Milk chocolates” categories will have a boost of 3x applied to them.
• As in the previous example, if the products are also tagged as being in the “Advent calendars” category, they’ll be boosted up by 3x.
• As in the previous example, a price filter for $2 is applied.
• The word “cheap” is removed, only returning products matching “chocolate”.

With relaxed filtering, results look as follows:

Overriding price from a high-priority policy

Or perhaps the retailer wants to allow slightly more expensive chocolates to be shown during Christmas by increasing the price max to $7. To ensure that the max price from the Christmas chocolates policy is not overridden if someone searches for “cheap chocolates”, we can set the conflict mode on the price to “override” rather than “restrict”, as follows:

With this override, the query for “cheap chocolate” ignores maximum price that is defined in the “cheap chocolate policy” and only applies the price specified in the “Christmas chocolates” policy, as follows:

This is similar to the previous example, with the difference being that the max price is set to the $7 value from the higher-priority policy because that policy specified “Override” on conflict. With the Christmas price filter taking precedence, the results look as follows:

These three variations (override, soft_boost, and override on price) demonstrate a key property of the system: A merchandiser can change how two policies interact by modifying a setting on a single field within a single policy, without deploying any code. The conflict strategy is the lever that controls business behavior.

Consumed phrase tracking

There’s a subtler form of conflict: two policies that match on the same phrase. If a higher-priority policy removes "without peanuts" from the query, a lower-priority policy that also matched on "without" has nothing left to act on. The system detects if the matched phrase is no longer present in the rewritten query and skips the lower-priority policy.

Intent policies are exempt from consumed phrase tracking: They set the retrieval strategy based on the original query match, regardless of what text has been removed by higher-priority policies.

Priority ordering, per-field conflict resolution, and consumed phrase tracking together give the control plane a deterministic composition model. With that foundation in place, the system can make a routing decision that would be risky without it.

Governance makes retrieval strategy safe

An important insight about routing to the correct retrieval method (text, semantic, or hybrid) is that it executes after governance. If your policies have already enforced "produce category”, then semantic retrieval becomes far less risky because the candidate set is constrained. A semantic search over 500 product items is a very different proposition from a semantic search over 500,000 SKUs. Governance narrows the blast radius before retrieval begins.

For example, without governance, a semantic query for “Fruit high in vitamin C under $4”, in addition to fruits, might return vitamin bottles, carrots, and green pepper. The control plane ensures that these undesired results aren’t even considered as part of the semantic expansion.

With that constraint in place, the control plane applies pragmatic routing logic:

• Lexical for navigational and head queries where deterministic precision matters.
• Semantic for descriptive discovery queries where concept matching helps.
• Hybrid selectively, when constraints have already been enforced and the business accepts broader recall.

From architecture to implementation

The governed control plane translates business intent into deterministic, composable execution plans, without embedding that logic in application code. Policies are data: matched at query time, resolved through explicit per-field conflict strategies, and applied as cascading transformations that produce explainable results. Elastic Services Engineering has built and deployed this architecture for enterprise ecommerce teams, using repeatable patterns and accelerators that compress the path from concept to production. You can see a demo of our implementation of a control plane on YouTube at: Fixing Search Relevance in Seconds: Introducing PRISM.

What's next in this series

The next post goes hands-on with the implementation: how the Elasticsearch percolator powers the policy lookup, including index mappings, boundary markers, highlight-driven phrase tracking, and concrete query examples.

Put governed ecommerce search into practice

The control plane architecture described in this post (per-field conflict resolution, cascading policy transformations, and governance-constrained retrieval routing) was designed and built by Elastic Services Engineering. Every pattern, screenshot, and transformation pipeline shown in this series comes from a working system built by Elastic Services Engineering and validated against enterprise-scale product catalogs.

If you want to implement a governed, policy-driven control plane on Elasticsearch, Elastic Services can get you there faster.

Join the discussion

Have questions about search governance, retrieval strategies, or ecommerce search architecture? Join the broader Elastic community conversation.

Learn how to get started with practical examples you can use right away.

Elasticsearch ES|QL query builder for JavaScript and TypeScript

If you've ever built an ES|QL query in JavaScript, you've probably written something like this:

const query = `FROM logs-*
| WHERE status_code >= ${minStatus}
  AND host.name == ${hostname}
  AND @timestamp >= "${startDate}"
| STATS error_count = COUNT(*) BY status_code
| SORT error_count DESC
| LIMIT 10`

It looks fine until hostname is O'Brien's server and the whole thing blows up with a parse error. Or until a user passes "; DROP INDEX logs into a search field and you realize you've been building queries with raw string concatenation this entire time.

There's a better way. The ES|QL query builder for JavaScript and TypeScript lets you write queries like this instead:

import { ESQL, E, f } from '@elastic/elasticsearch-esql-dsl'

const query = ESQL.from('logs-*')
  .where(E('status_code').gte(minStatus))
  .where(E('host.name').eq(hostname))
  .where(E('@timestamp').gte(startDate))
  .stats({ error_count: f.count() })
  .by('status_code')
  .sort(E('error_count').desc())
  .limit(10)

Values are escaped automatically. You get autocomplete in your editor. And you can see exactly what the query does, without mentally parsing a template literal.

ES|QL query builders are already available across Elastic's language clients, including Python, Ruby, and others. This article focuses on the JavaScript and TypeScript version, walking through practical examples you can start using today.

Getting started

Install the package:

npm install @elastic/elasticsearch-esql-dsl

Here’s a minimal query:

import { ESQL, E } from '@elastic/elasticsearch-esql-dsl'

const query = ESQL.from('employees')
  .where(E('still_hired').eq(true))
  .sort(E('last_name').asc())
  .limit(10)

console.log(query.render())

This renders:

FROM employees
| WHERE still_hired == true
| SORT last_name ASC
| LIMIT 10

To run it against Elasticsearch:

import { Client } from '@elastic/elasticsearch'

const client = new Client({ node: 'http://localhost:9200' })
const response = await client.esql.query({ query: query.render() })

That’s it. No string interpolation, no manual escaping.

Building a real query, step by step

Let's walk through a realistic scenario: You're building a dashboard that analyzes web server error logs. We'll start simple and layer on features.

Step 1: Filter error logs

import { ESQL, E } from '@elastic/elasticsearch-esql-dsl'

const errors = ESQL.from('logs-*')
  .where(E('status_code').gte(400))
  .limit(100)

FROM logs-*
| WHERE status_code >= 400
| LIMIT 100

Step 2: Add a computed column

Your timestamps are in milliseconds, but you want response time in seconds:

const errors = ESQL.from('logs-*')
  .where(E('status_code').gte(400))
  .eval({ response_secs: E('response_time_ms').div(1000) })
  .limit(100)

FROM logs-*
| WHERE status_code >= 400
| EVAL response_secs = response_time_ms / 1000
| LIMIT 100

Step 3: Aggregate errors by status code

import { f } from '@elastic/elasticsearch-esql-dsl'

const errorBreakdown = ESQL.from('logs-*')
  .where(E('status_code').gte(400))
  .stats({
    error_count: f.count(),
    avg_response: f.avg('response_time_ms'),
  })
  .by('status_code')
  .sort(E('error_count').desc())

FROM logs-*
| WHERE status_code >= 400
| STATS error_count = COUNT(*), avg_response = AVG(response_time_ms) BY status_code
| SORT error_count DESC

The f namespace gives you access to 150+ ES|QL function wrappers: aggregations, string functions, date functions, math, geo, and more. They all return chainable expressions, so you can use them anywhere you'd use E().

Step 4: Use date functions for time-based analysis

const hourlyErrors = ESQL.from('logs-*')
  .where(E('status_code').gte(400))
  .eval({ hour: f.dateTrunc('@timestamp', '1 hour') })
  .stats({ error_count: f.count() })
  .by('hour')
  .sort(E('hour'))

FROM logs-*
| WHERE status_code >= 400
| EVAL hour = DATE_TRUNC(@timestamp, "1 hour")
| STATS error_count = COUNT(*) BY hour
| SORT hour

Step 5: Branch queries safely

Every method returns a new query object. The original is never mutated. This means you can build a base query and branch it for different views:

const base = ESQL.from('logs-*')
  .where(E('status_code').gte(400))
  .where(E('@timestamp').gte('2026-01-01T00:00:00Z'))

const byStatus = base
  .stats({ count: f.count() })
  .by('status_code')
  .sort(E('count').desc())

const byHost = base
  .stats({ count: f.count() })
  .by('host.name')
  .sort(E('count').desc())
  .limit(20)

const recent = base
  .sort(E('@timestamp').desc())
  .keep('@timestamp', 'status_code', 'url.path', 'message')
  .limit(50)

Three different queries, one shared base. Change the filter on base, and all three update. This is especially useful for dashboards where multiple panels query the same dataset with different aggregations.

Three ways to write expressions

The domain‑specific language (DSL) gives you flexibility in how you write conditions. Here's the same WHERE clause written three different ways:

Raw strings: When you're writing a quick one-off:

.where('status_code >= 400 AND host.name == "web-01"')

The E() expression builder: When you want type safety and autocomplete:

import { and_ } from '@elastic/elasticsearch-esql-dsl'

.where(and_(
  E('status_code').gte(400),
  E('host.name').eq('web-01')
))

The esql template tag: -When you want safe interpolation of dynamic values:

import { esql } from '@elastic/elasticsearch-esql-dsl'

const minStatus = 400
const host = 'web-01'
.where(esql`status_code >= ${minStatus} AND host.name == ${host}`)

All three produce the same ES|QL. Pick whichever fits your situation: raw strings for simple cases, E() when building expressions programmatically, and the template tag when mixing literal ES|QL with dynamic values.

Keeping queries safe

If any part of your query comes from user input, you need to think about injection. ES|QL supports parameter binding, and the DSL makes it straightforward:

function searchLogs(userQuery: string) {
  const query = ESQL.from('logs-*')
    .where(E('message').eq(E('?')))
    .limit(100)

  return client.esql.query({
    query: query.render(),
    params: [userQuery],
  })
}

The ? placeholder is replaced server-side by Elasticsearch, so the user's input never touches the query string. No escaping, no injection risk.

Beyond the basics

Once you're comfortable with the core commands, the DSL supports every advanced ES|QL feature:

Hybrid search with FORK and FUSE:

const results = ESQL.from('articles')
  .fork(
    ESQL.branch()
      .where(f.match('title', 'elasticsearch'))
      .sort(E('_score').desc())
      .limit(50),
    ESQL.branch()
      .where(f.knn('embedding', 10))
      .sort(E('_score').desc())
      .limit(50),
  )
  .fuse('RRF')
  .limit(10)

Data enrichment:

const enriched = ESQL.from('logs-*')
  .enrich('ip_lookup')
  .on('client.ip')
  .with('geo.city', 'geo.country')

Conditional aggregation:

const stats = ESQL.from('employees')
  .stats({
    eng_avg: f.avg('salary').where(E('dept').eq('Engineering')),
    sales_avg: f.avg('salary').where(E('dept').eq('Sales')),
    total: f.count(),
  })

AI/machine learning (ML) integration:

const summarized = ESQL.from('docs')
  .completion('Summarize this document')
  .with({ inferenceId: 'my-llm' })

For the full list of commands and functions, check out the ES|QL query builder documentation.

What's next

This is the initial release of @elastic/elasticsearch-esql-dsl. You can find the package on npm, explore the source on GitHub, and read the full documentation in the repository. If you run into issues or have feature requests, open an issue; we're actively developing this and want to build what JavaScript and TypeScript developers actually need.

Users familiar with cross-cluster search (CCS) know that to include a remote cluster in a search you must specifically reference it with a cluster alias prefix, such as remote:metrics* (or *:metrics* to target all remotes). With CPS, "bare" index names or patterns such as metrics* in a search index expression implicitly reference all instances of that index that can be found on the origin project and on all of its linked projects.

In this post, we pull back the curtains on how the Elasticsearch TransportSearchAction, the code that underlies the _search API (and _async_search) APIs, determines which indices, aliases, and datastreams to search on which projects when running a cross-project search.

Analyzing index expressions

One important consequence of the new CPS model is that the Elasticsearch query parameters allow_no_indices and ignore_unavailable require different handling than before. In general, these parameters control whether a search should throw an error if a concrete index name (for example, "logs") cannot be found or isn’t accessible (ignore_unavailable=false), or a wildcard pattern (for example, "logs*") doesn’t match anything or if there are no indices at all to search (allow_no_indices=false).

In CCS, those parameters are analyzed on each cluster separately: Each cluster can just consult which indices, aliases, or data streams exist locally. But in CPS, we need to account for whether each resource (index, alias, or data stream) referenced by an index expression matches on any project (origin or linked), rather than every project.

To illustrate, suppose a user has one linked project (linked1) and they issue a cross-project query with index expression logs*,metrics-1, along with allow_no_indices=false,ignore_unavailable=false. As long as we find one resource (index, alias, or data stream) that matches logs* and one that matches metrics-1 on any project, then the search can proceed. If metrics-1, for example, is found on linked1 but not the origin project, that suffices to pass the ignore_unavailable=false constraint. Only if it’s found nowhere would we throw an IndexNotFoundException.

To handle this, the _search API in cross-project search mode needs to gather information from each linked project before kicking off the actual search.

The serverless node that receives the _search REST request is considered to be the origin project, and it acts as the overall search coordinator. To fully analyze and process an index expression before kicking off the actual search, the search coordinator needs to:

1. Determine which projects are in scope for the search.
2. Determine which indices, aliases, or data streams should be searched in each individual project.
3. Do a final reconciliation step of all the information gathered: 1) Have we found all the indices required to proceed (as defined by the allow_no_indices and ignore_unavailable settings)?; and 2) Are there any projects that should be skipped since they have no matching indices?

To illustrate, we’ll follow a CPS request against the _search API from start to finish. Suppose that the incoming user request is to search metrics* and that the origin project has two indices which match the expression, namely metrics-ES and metrics-KB, and that the origin project is linked to projects P1 and P2, where P1 has index metrics-ES and P2 has no index, alias, or data stream that matches metrics*.

Determining projects in scope

The search coordinator starts by grabbing the full list of linked projects from Elasticsearch cluster state. It then determines, based on Universal Identity and Access Management (UIAM) credentials provided with the query, which of those projects the user has access to and keeps the subset of projects that the user is allowed to access.

If a project_routing parameter is present on the request, that can further limit which projects are in scope for the query. For instance, "project_routing":"_alias:_origin" would cull the list to just the origin project, while "project_routing":"_alias:P*" would include the linked projects P1 and P2 and remove the origin project from the list of projects in scope for the query.

The index expression itself can also limit which projects to access. Qualified index expressions, like "P*:metrics*", would indicate that we should search only on projects whose alias starts with "P". For our case, the "bare" or unqualified index expression "metrics*" means "search on all projects that are in scope," where "in scope" is modified by security access and project_routing.

Determining which indices each project has

Once we know which projects to search, we need to determine which matching indices each one has. To support this, the search coordinator creates a ResolvedIndexExpressions data structure that allows tracking which indices on each project should be included in the search.

For each index expression provided by the user, that data structure tracks:

• Original expression: The index expression, as provided by the user.
• Local resolution: The local expressions that will replace the original together with the resolution result.
• Remote expressions: A set of remote expressions one for each project a query can target.

On the search coordinator that received the request, we’ve so far been able to determine which projects are in scope (origin, P1, and P2) and which, if any, matching indices exist on origin. So, for our example search against metrics*, we’ll have the following structure at that point in time:

• Original expression: metrics*.
• Local resolution: SUCCESS, <metrics-ES, metrics-KB>.
• Remote expressions: <P1:metrics*, P2:metrics*>.

The remote expressions are left unresolved until we later contact the P1 and P2 projects to fill in that information.

The index resolution can be any of the following:

• SUCCESS: Local resolution completed successfully.
• NOT_VISIBLE: Indicates that a non-wildcard expression was resolved to nothing, either because the index doesn’t exist or is closed.
• UNAUTHORIZED: Indicates that the expression could be resolved to a concrete index, but the requesting user isn’t authorized to access it.
• NONE: No local resolution was attempted, typically because the expression is remote-only (for example, P1:index).

Check indices on the linked projects

In the search API, how we check for indices on the linked projects depends on whether the cross-project search is being run with minimize round trips or not. Most searches in CPS are set internally to run with minimize_roundtrips=true, so we’ll focus on that pathway.

In CCS, where index expression analysis is done locally on each cluster, we just send the same request to every cluster and, with minimize_roundtrips=true, each remote cluster sends back an entire SearchResponse that the primary search coordinator collects and eventually merges into all the other responses it receives.

By contrast, for CPS, an additional phase was introduced to search where we contact each linked project to assess which indices are present and which ones the user has access to. This is an additional round trip that uses ResolveIndexAction, the class that implements the functionality of the _resolve/index endpoint. Upon receiving the ResolveIndex responses from all the linked projects, we can fill in the ResolvedIndexExpressions data structure on the primary search coordinator. For this case, since P1 has index metrics-ES and P2 has no matching indices, the updated data structure would be:

• Original expression: metrics*.
• Local resolution: SUCCESS, <metrics-ES, metrics-KB>.
• Remote expressions: <P1: SUCCESS <metrics-ES>, P2: NOT_VISIBLE>.

CPS validator

Once we have all the linked project information, we can run the validation to honor the IndicesOptions specified by the caller.

If the user had specified a qualified index expression (for example, original=P1:metrics*), the CPS validator needs to make sure that project P1 has at least one index matching metrics*, otherwise a 404 index not found exception would be returned to the user. In other words, qualified expressions imply that an index, alias, or data stream matching that name must be present on all projects specified by the qualifier.

On the other hand, for “bare” (unqualified) CPS index expressions (for example, original=metrics*), we only need to check whether the original index expression exists anywhere.

For the example we’re following in this post, the ResolveIndexExpressions show that at least one index, alias, or data stream matching metrics* was found, so the search can proceed.

However, we can also see that the P2 cluster has no matching indices. In that case, we can remove it from the rest of the query. So now, for the actual query, we’ll:

• On the origin project, search metrics-ES, metrics-KB.
• On the P1 project, search metrics-ES.
• Skip the P2 project. Note: Since it wasn’t included in the query at all, it won’t show up on the _cluster/details of the SearchResponse.

Conclusions

We’ve described how the TransportSearchAction class in Elasticsearch supports some key new features of cross-project search. That class supports a large number of Elasticsearch REST APIs, such as, _search, _async_search, _msearch, _eql, _sql and _count, all of which use the mechanisms described in this document.

"Of course! I'd be happy to help you..."

"This should give you a good overview..."

"Feel free to let me know if you need anything else!"

This isn't just annoying; it's expensive. Every token costs money and adds latency. For production Elasticsearch queries, that overhead compounds fast. In this post, we introduce elastic-caveman and share the results of a controlled experiment across eight live Model Context Protocol (MCP) scenarios against an Elasticsearch cluster. The findings: 63.6% average token reduction, 817 tokens saved, and zero loss of technical accuracy.

Enter elastic-caveman

elastic-caveman tests a simple hypothesis: Strip AI responses to pure signal, and measure the impact. The approach:

• Normal mode: Full conversational AI with greetings, explanations, and sign-offs.
• Caveman mode: Raw data with minimal structural labels only.

We tested both modes against a live Elasticsearch instance using MCP with real support ticket and Salesforce case data across eight production scenarios.

Results: 64% token reduction, zero accuracy loss

Here's what we found across eight live MCP tool calls: The Elastic-Caveman initiative has successfully optimized AI response size without compromising quality or functionality.

Key preservations (0% loss):

• Technical accuracy
• API paths
• ES|QL syntax
• Field names

The critical finding: Every field name, case ID, ES|QL query, account name, and sentiment score was preserved exactly. Not approximately. Exactly.

Real examples: Before and after

Example 1. List indices: 87% reduction

User: Show me my indices

Normal mode (107 tokens):

Of course! I'd be happy to help you see your indices. Here's a comprehensive list of all the indices in your Elasticsearch cluster. Each entry shows the index name along with relevant metadata. This should give you a good overview of what's stored in your cluster:-- salesforce-cases-- support-ticketsThese are all the indices I found. If you'd like to dive deeper into any specific index, just let me know and I'll be glad to assist!

Caveman mode (14 tokens):

Indices: salesforce-cases, support-tickets

Saved: 93 tokens (86.9%)

Example 2. Generate ES|QL query: 75% reduction

User: Show me open critical tickets grouped by product area

Normal mode (208 tokens):

Sure thing! I'd be happy to help you write that ES|QL query. What you're looking for is a query that will filter for open tickets with critical priority and group them by product area. Here's how you can accomplish this...

[followed by the actual query, plus 150+ tokens of step-by-step explanation]

Caveman mode (52 tokens):

FROM support-tickets
WHERE status = "Open" AND priority = "Critical"
| STATS count = COUNT(*) BY product_area
| SORT count DESC
| LIMIT 100

Filters open+critical, groups by product_area, counts, sorts desc.

Saved: 156 tokens (75.0%). ES|QL syntax is character-for-character identical in both modes.
Example 3. Search recent support tickets: 35% reduction

User: Show me 5 recent support tickets

Caveman mode (143 tokens):

5 recent tickets, newest first:
Case     | Priority | Status    | Account               | Subject                                    | Days | Sentiment
CASE-0012 | Critical | Open      | Pinnacle Financial    | Regulatory reporting: incorrect tax calcs  | 2    | -0.94
CASE-0021 | Critical | Escalated | Riverstone Capital    | Security audit blocking renewal            | 2    | -0.93
CASE-0046 | Medium   | Open      | Blackrock Partners    | Pen test findings need vendor response     | 2    | -0.70
CASE-0006 | High     | Open      | United Oil & Gas      | Unauthorized access -- urgent               | 3    | -0.91
CASE-0017 | High     | Open      | Riverstone Capital    | Renewal paperwork stalled                  | 4    | -0.86

All negative sentiment. Two critical cases from April 7. All case IDs, priorities, account names, and sentiment scores preserved verbatim.

What gets removed vs. what stays

When we clean up the output, we strip out conversational filler, like “Of course! I’d be happy to help you…”, “This should give you a good overview…”, or “Would you like me to help you prioritize these?”, and we keep every piece of factual content, such as ES|QL snippets, like FROM support-tickets WHERE status = "Open"; field names like sentiment_score, product_area, and resolution_hours; and index names, like support-tickets and salesforce-cases. We also preserve concrete identifiers and business entities, such as case IDs CASE-0012 and CASE-0002; account names, like Pinnacle Financial and United Oil Gas Corp; along with all numeric values, for example, a sentiment_score of -0.94, counts like 47 duplicates, durations such as 18 days, or metrics like 27.0 average hours, so the edited text is tightly focused on query syntax, entities, and numbers while discarding only the polite scaffolding.

Results varied by operation type:

Complete evaluation breakdown

Token savings by query type across all eight scenarios against live MCP data:

Technical accuracy verification:

Zero information loss. 64% fewer tokens.

Why this matters for Elastic users

For teams building AI assistants on Elasticsearch, 64% token reduction means 64% savings on output costs at scale, faster streaming responses, and more context window space for actual data rather than fillers. When you're debugging an ES|QL query at 2 a.m., you don't need an AI telling you it's delighted to help; you just need the query response!

The bigger picture: Rethinking AI interfaces

This experiment reveals something fundamental: Conversational AI interfaces optimize for the wrong metric. They optimize for sounding human when users often just want accurate data, fast.

For technical workflows, especially data queries, there's a strong case for mode-switching:

• Conversational mode: When exploring or learning.
• Caveman mode: When you know what you want and need it now.

The Elastic MCP server makes this possible by returning structured, accurate responses that work in both modes without modification.

How elastic-caveman works

elastic-caveman is an Agent Skill, that is, a markdown file with YAML front matter that any compatible AI agent reads and follows. No runtime. No binary. No API calls. Just instructions that reshape how your agent talks when working with Elasticsearch.

Install with:

npx skills add srikolag/elastic-caveman

Supported agents: Claude Code, Cursor, Codex, Windsurf, GitHub Copilot, Gemini CLI, Roo

Trigger with:/elastic-caveman

Disable with:"normal mode" or "verbose"

Live in action

We tested elastic-caveman with the Claude model to measure its impact on token usage and cost:

• With elastic-caveman: Token usage was 368 tokens (in) and 1.6k tokens (out), resulting in a cost of $0.11.
• Without elastic-caveman: Token usage was 367 tokens (in) and 1.8k tokens (out), resulting in a cost of $0.12.

Prompt: Get me the critical support tickets from the support-tickets index in kibana for Pinnacle Financial

This test demonstrates the efficiency of elastic-caveman.

What's next

Caveman mode is just the beginning. Consider dynamic mode switching: Flip between concise and conversational mid-session. Or a hybrid approach: Lean on success, explanatory on errors. Or custom verbosity levels for teams that want something in between. The goal isn't to make AI assistants robotic; it's to give users control over the signal-to-noise ratio.

Try it yourself

Test caveman mode with your Elasticsearch data:

1. Set up the Elastic MCP server.
2. Install elastic-caveman.
3. Run queries in both normal and caveman modes.
4. Compare token counts and accuracy.

Full evaluation methodology and scripts available in the GitHub repo.

The bottom line

Across eight real scenarios with live Elasticsearch data, elastic-caveman delivered 64% average token reduction with zero accuracy loss and 100% preservation of ES|QL syntax, field names, and technical values. Sometimes the best AI response isn't the chattiest one. Sometimes you just need the data; and with elastic-caveman, you can get it 64% faster. Ready to optimize your Elasticsearch AI workflows? Check out Elasticsearch Labs for more tutorials, integrations, and research on building with Elasticsearch and AI, or start building with Elasticsearch today.

Want to optimize your Elasticsearch AI workflows? Check out Elasticsearch Labs for more tutorials, integrations, and research on building with Elasticsearch and AI. Ready to try it yourself? Start building with Elasticsearch today.

Why time series discovery matters

Elasticsearch uses time series data streams (TSDS) to efficiently store metrics. Backed by a fully columnar store, metrics stored in TSDS in Elasticsearch 9.4 require up to 17x less storage compared to using a standard index. Starting with Elasticsearch 9.2, we've also added time-series support in Elasticsearch Query Language (ES|QL) as a fully supported capability when querying data stored in TSDS.

If you operate TSDS in Elasticsearch, you already know the pattern: dimensions identify a series, metrics carry typed values like gauge or counter, and the TS source command in ES|QL enables time series aggregation functions such as RATE and AVG_OVER_TIME.

What that pipeline can't tell you (but you need to know just as often) is which metrics and time series actually exist right now, for the slice of data you care about. Field mappings enumerate every field that was ever declared; they don't show what's actively being ingested in a specific cluster, environment, or time window. That gap shows up across very different workflows:

• Dashboard building. Metric and dimension pickers should reflect what the cluster currently holds for the user's filters, not every field that has ever been mapped. Otherwise, dropdowns stay cluttered with stale options and panels render empty.
• Onboarding to an unfamiliar TSDS. A new cluster, a new integration, a customer's data. A quick list of the metrics being ingested, with their types, units, and applicable dimensions, replaces hours of mapping spelunking and ad hoc probe queries.
• Data quality investigations. Mapping drift (the same metric declared gauge in one backing index and counter in another) and dimension-cardinality explosions both surface immediately in the catalog output.
• Query validation. Before running an expensive TS ... | STATS aggregation, confirm that the metric and dimensions you're about to use really have data in your window.

Kibana already relies on this internally. The dynamic metrics catalog in the observability experience appends METRICS_INFO to the user's active TS query so the UI only offers metrics that truly exist for the current filters, rather than every field in the mapping.

The problem: Mappings are an inventory of fields, not time series

Operations teams routinely need answers to questions that mapping APIs alone cannot answer:

• Which metrics actually have data in this environment, for this cluster, in this time range?
• How are those metrics typed, and which dimensions apply when building or validating a query?
• How many distinct time series exist per metric?

Until now, answering these questions meant piecing together mapping APIs, ad hoc queries, and guesswork. METRICS_INFO and TS_INFO turn those questions into single-line ES|QL queries that fit naturally into the same pipeline you use for STATS:

TS k8s
| WHERE cluster == "prod"
| METRICS_INFO
| SORT metric_name

How these commands integrate with ES|QL pipelined queries

Both commands are processing commands. Once you run one, the table is replaced: Downstream commands, like KEEP, WHERE, or STATS, operate on metadata rows, not the original time series documents.

A few rules to keep in mind:

• They apply only after a TS source. Using them after FROM or without a preceding TS source produces an error.
• They must appear before STATS, SORT, or LIMIT run on the time series rows returned by TS. For example, TS ... | STATS ... | METRICS_INFO is invalid; TS ... | METRICS_INFO | STATS ... is valid because STATS then runs on the metadata table.
• You can filter and aggregate after METRICS_INFO or TS_INFO on the metadata columns with the usual processing commands.
• You can include filters before them, for example, narrowing by @timestamp or dimensions, so that the produced metadata reflects series that match your query context, not the entire index.

Conceptually, the pipeline looks like this:

TS + filters  →  METRICS_INFO or TS_INFO  →  KEEP / WHERE on metadata  →  STATS / SORT / LIMIT

This design means you can scope a catalog to exactly the slice of data you care about and then post-process the result with more ES|QL commands as desired.

How to use METRICS_INFO and TS_INFO in practice

METRICS_INFO retrieves information about the metrics available in your time series data streams, together with applicable dimensions and other metadata, all scoped to the current TS query. TS_INFO does the same for individual time series. Each row is one metric plus the dimension values that identify one series.

Each command offers a different view to time series metadata: METRICS_INFO collapses what you see into one row per distinct metric signature: the metric name plus how it's declared (type, unit, field type, which dimension fields apply) as observed across backing indices. TS_INFO adds one row per metric and time series, with a dimensions column that holds the concrete label set for each series, formatted as a JSON object (for instance, {"job":"elasticsearch","instance":"instance_1"}).

If the same logical metric name shows up with incompatible metadata in different places, you get multiple rows or multi-valued cells. That's a useful signal when you're tracking down mapping drift.

Both commands expose the same core columns; only TS_INFO adds dimensions.

Start with a catalog of names and types. The smallest useful query is a TS source, METRICS_INFO, and a sort so the table is easy to scan:

TS k8s
| METRICS_INFO
| SORT metric_name

You can post-process the result as usual in ES|QL. For instance, you can trim columns or filter on metadata before aggregating:

TS k8s
| WHERE cluster == "prod" AND TRANGE(1d)
| METRICS_INFO
| KEEP metric_name, metric_type
| SORT metric_name

To find how many distinct metric names match a pattern (not which series), combine METRICS_INFO with STATS:

TS k8s
| METRICS_INFO
| WHERE metric_name LIKE "network.total*"
| STATS matching_metrics = COUNT_DISTINCT(metric_name)

Document predicates before the catalog command narrow the processed time series to data samples that actually exist in your window. The metrics listed are those with matching data, not every field that has ever been mapped:

TS k8s
| WHERE cluster == "prod" AND TRANGE(1d)
| METRICS_INFO
| SORT metric_name

Run the same scoped pipeline, but swap the middle command for TS_INFO, and the question shifts from “which metrics match” to “which time series identities match”. Each row is one metric plus one combination of dimension values; sort on metric_name and dimensions so related series group together:

TS k8s
| WHERE cluster == "prod" AND TRANGE(1d)
| TS_INFO
| SORT metric_name, dimensions

That extra column can be used to deduce metric cardinality. Each TS_INFO row is one time series for a given metric, so grouping with STATS counts how many distinct time series exist per metric:

TS k8s
| TS_INFO
| STATS series_count = COUNT(*) BY metric_name
| SORT metric_name

Choosing between them: Use METRICS_INFO when you want a compact inventory of metric names and types in the filtered TS context. Use TS_INFO when you need label combinations, per-metric series counts. In practice, skim with METRICS_INFO and then switch to TS_INFO when the answer depends on which dimensions apply, not just what metrics exist.

Under the hood: How the commands are executed

Both METRICS_INFO and TS_INFO run inside the same distributed ES|QL execution that powers any TS query. In addition to standard features, like shard-level parallelism, Lucene filter pushdown, and coordinator-side merging, special care has been taken during implementation so that the cost scales with the number of matching time series, not the number of documents. Here's how each output row gets produced:

1. The TS command defines the scope. TS resolves your data stream pattern to its TSDS backing indices and turns any filters you place before the catalog command, such as a time range on @timestamp or dimension predicates in WHERE, into a Lucene query that runs on every shard that can match. Shards in backing indices outside the time window are pruned up front and never touched.

2. Each shard iterates over matching documents and tracks one per series. A TSDS index is physically sorted by _tsid first, then by @timestamp (descending). That sort matters here: All documents belonging to the same time series sit next to each other on disk, so as a shard processes documents in order, it only needs to keep the first document it sees for each new _tsid and skip the rest. The result is one representative document per time series that has at least one document matching your filters.

3. The mapping tells us what each field is. The backing index mapping is the source of truth for the metadata that describes each field:

• Fields declared with time_series_metric are metrics, and the mapping carries each metric's metric_type, field_type, and (if declared) meta.unit.

4. Synthetic source fills in the actual dimension and metric presence. For the one representative document per series, the shard reads a subset of _source containing only the dimension (and metric) paths the mapping declares. TSDS uses synthetic _source, so that subset is reconstructed primarily from doc values — no stored _source is needed. From that reconstructed sliver of JSON, the shard learns two things:

• The dimension key/value pairs for this series (the dimensions JSON for TS_INFO, and the set of dimension keys that feed dimension_fields for both commands).
• Which metric fields actually have data for this series in this backing index.

5. Partial aggregation happens inside each shard. Shards don't ship raw per-series rows upstream. They partially aggregate first, which is a big part of why catalog queries stay cheap.

6. The coordinator merges across shards and data streams. Each data node first reduces its own shards' partial results and streams them to the coordinator, which applies the same merge logic one more time.

7. The rest of the pipeline runs as usual. Everything after the catalog command (KEEP, WHERE, STATS, SORT, LIMIT) runs against this consolidated metadata table on the coordinator, exactly like any other ES|QL stage.

The net effect is that catalog queries do just enough work to identify one representative document per series, read a small reconstructed slice of that document, classify its fields against the mapping, and fold the results into a handful of metadata rows. Because the output cardinality is bounded by the number of matching series (for TS_INFO) or by the number of distinct metric signatures (for METRICS_INFO), not by the number of documents in the window, these commands stay responsive even against long retention windows and high-ingest data streams.

Running these commands against the full high cardinality TSDB benchmark corpus without a time range filter (1.84 B documents / 1.4 M time series / 2.77 TB uncompressed) on a single-node Elasticsearch (AWS c8gd.8xlarge, 24 cores, 24 GiB heap, NVMe SSD, 3 primary shards, force-merged), METRICS_INFO returns in ~4 seconds.

Beyond ad hoc queries

These commands also support product workflows inside Kibana. The UI appends METRICS_INFO to a user's TS query (when the query doesn't already include STATS) to build a metric catalog aligned with the user's filters, rather than relying solely on mappings.

These new commands are also the foundation of Prometheus-compatible metadata APIs we're adding to Elasticsearch, which Prometheus-based tools can use. Stay tuned for a dedicated blog post that goes into more detail.

Data quality

Multi-valued unit, metric_type, or field_type in the METRICS_INFO output, is a concise warning that backing indices aren't aligned on a metric's definition. TS_INFO makes it easier to see whether an explosion in cardinality comes from a handful of metrics or from dimension cardinality you should account for in alerts and aggregations. For example, ranking metrics by series count surfaces outliers at a glance:

TS k8s
| TS_INFO
| STATS series_count = COUNT(*) BY metric_name
| SORT series_count DESC

When a single metric dwarfs the rest, as network.eth0.rx does above, the explosion is concentrated in a handful of metrics, and inspecting that metric's dimensions pinpoints which label is growing. Comparable counts across metrics instead point at shared dimension cardinality, such as a newly introduced pod or instance value propagating through every time series.

Availability

METRICS_INFO and TS_INFO are generally available in Elastic Cloud Serverless and in Elasticsearch basic starting with the 9.4.0 release.

For command pages (syntax, restrictions, and examples), see METRICS_INFO and TS_INFO.

For background on TSDS and the TS command itself, start with the official documentation on time series data streams and the TS source command.

When “something is slow” finally has an answer

Query activity is already available in your Elastic Cloud Serverless project today. For Elastic Cloud Hosted and Elastic Self-Managed clusters, it ships with Kibana 9.4 and is available across all deployments and clusters on that version. Query activity is the Kibana view on top of the Tasks Management API in Elasticsearch. It’s purpose-built for search-related tasks in any query language, including Elasticsearch Query Language (ES|QL), DSL, SQL, and Event Query Language (EQL).

It always starts the same way. Someone pings you on a Friday: Discover feels stuck. The exec dashboard won’t load. Did we change something? You open your monitoring tabs, squint at CPU, maybe tail a log, and you’re still guessing. Is it one giant ES|QL pipeline? A dashboard nobody remembers? A background rule doing honest work at the worst possible time? The cluster isn’t mysterious on purpose. In-flight work is simply invisible unless you enjoy living in Dev Tools and reconstructing life stories from task IDs and snippets of JSON.

We built Query activity for everyone who has ever muttered, Just tell me what’s running. It’s a new screen in Kibana that lists active search work in ES|QL, DSL, SQL, or EQL. It shows the queries that are consuming your cluster right now, with enough context to move from panic to diagnosis without a scavenger hunt.

The play you know and the one-minute rewrite

If you’ve operated Elasticsearch for more than a week, you’ve lived the old script. In Act I, someone says the cluster feels slow. In Act II, you scatter across shards, heap, slow logs, and sticky-note task IDs. Hours pass, and you still cannot name the query. In Act III, maybe you find the culprit before dinner, or maybe next month Act I opens again with the same villain in a fake mustache.

Query activity replaces that meandering Act II with one default sequence. It’s the same story, compressed from symptom to evidence to origin to action in about a minute. Paste this into your runbook or send it to your on-call channel. It’s the whole innovation in practice.

1. Open Query activity as soon as Act I hits. On Elastic Cloud Hosted and Elastic Self-Managed clusters, go to Stack Management and then Cluster performance. On Elastic Cloud Serverless, go to Admin and Settings and then Project performance. Do this before you fork into guesswork.

2. Refresh the list of queries once so you’re looking at right now, not five minutes ago.

3. Surface the pressure. Sort by run time, or tighten the “Run time” filter, until the expensive work floats to the top.

4. Open the flyout on the worst offender. You’ll see duration, query type, index breadth, and full query text. That’s your evidence without opening Dev Tools.

5. Name the owner. Use trace.id to jump into Discover and filter on the audit or query logs for that trace, or use X-Opaque-Id to figure out which dashboard, saved search, or rule this query originates from.

6. Resolve Act III. Let the query finish, fix the upstream, or cancel when it’s appropriate and Elasticsearch says the task is cancelable.

That’s one pass through what used to be three acts. You get attribution instead of folklore and decisions instead of theater.

Query activity deep dive

The one-minute sequence above is the habit. What follows is the machinery: the concrete controls and signals in Kibana that make that rewrite possible. You get what’s executing, how long it’s been running, where it came from, and what to do next, without stitching clues across tabs.

Under the hood, this view is powered by Elasticsearch’s Tasks Management API for long-running search tasks. It’s translated into an operational UI that’s built for speed. You can find the outlier quickly, inspect rich details, and act with confidence.

Here’s how the UI backs each beat of the runbook.

The main view is a filterable list of running queries. It includes a search bar so you can match anything in the table, including task ID. You also get filters for run time, query language, and source (for example Discover, Dashboard, and similar surfaces). You stay in control of what “noisy” means.

Refresh is manual on purpose. The table does not auto-refresh. You click Refresh when you’re ready, and the UI shows when the last refresh happened. You shouldn’t have to wonder whether the list is stale.

When you click a task ID, a detail flyout opens. It shows start time, run time, how many indices the query touches, and full query text. When X-Opaque-Id is present, it can help you trace an Elasticsearch query to its origin in Kibana so you can turn “mystery load” into “that dashboard, that version.” Previous and next navigation let you walk the queue without jumping back to the list. When trace.id is available, you can open Discover with that trace prefiltered. That helps when the incident channel is already busy.

Where work is cancelable, you can request cancellation from the list or the flyout. There’s a deliberate confirmation step. After you confirm, the cancel control shows a spinner until Elasticsearch reports that the task actually stopped. The goal is oops-proof, not oops-fast.

Viewing and managing active query work requires appropriate cluster privileges. The UI states clearly when something is missing. For example, users without cluster:manage may not be able to take destructive actions. Users without cluster:monitoring may not see task details. You shouldn’t get a blank screen that feels like the stack is playing hide-and-seek.

If you’ve been following our broader story around query observability, this is the live side of the house. It’s what is happening now, in the product, with controls you can use. Over time, pair it with historical views, such as query logs and AutoOps long-running search tasks insights when you need to ask whether this has happened before. When you need to answer what’s eating your cluster in this minute, start with the new Query activity UI in Kibana.

Who this is for (and who becomes the hero)

Cluster and platform admins get the obvious win: faster incident response and less time translating APIs into narratives for stakeholders.

Centers of excellence and internal search champions get something equally valuable: a teachable moment you can screenshot. This is the query pattern that blew up shared capacity. This is what “interactive” versus “background” pressure looks like when everyone is busy.

Anyone on the hook for Service Level Agreements (SLAs) gets a cleaner bridge between user experience (“the app is slow”) and search reality (“these three requests are still running, and one of them is huge”).

You don’t have to be the person who wrote the query to be the person who explains the cluster calmly. That’s the whole point.

Not every task is cancelable, and deep tuning work still has its place. Query activity doesn’t fix your queries for you. It surfaces in seconds which ones might need attention, and it gives you faster evidence, clearer attribution, and better decisions before you reach for heavier tools.

Where to find it

You’ll find Query activity in the performance area of each deployment model. In Elastic Cloud Hosted and Elastic Self-Managed clusters, open Stack Management and then Cluster performance. In Serverless projects, open Admin and Settings and then Project performance.

Threshold hygiene: Open Stack Management and then Advanced Settings. The running_queries:minRunningTime setting defaults to 100 ms. Only tasks that have been running longer than that appear. That way you can dial through noise without drowning in instantaneous work.

What to do next

Walk through the six-step sequence once when the cluster is calm. When Act I hits, you won’t be learning a new UI under pressure. Then repeat it during the next slow moment. The gap between assuming and seeing is the whole product story.

If you aren’t on Elastic Cloud yet, you can still get hands-on with the stack at elastic.cloud/registration.

This post answers those questions. A governed control plane doesn't just improve search relevance; it changes the operating model. It moves search behavior changes from engineering deployment cycles to business-driven workflows, without sacrificing safety or accountability.

The scenario that exposes the operating model

Imagine that you’re in the weeks leading up to Christmas, and your merchandising team has identified three urgent changes that must immediately be made to search behavior:

• Campaign launch. Due to an ordering error, there’s an oversupply of in-house branded turkeys. Therefore any query for "turkey" must boost the in-house brand.
• Product recall. A supplier has recalled a product line. Queries that would surface those products shouldn’t be shown.
• Seasonal reinterpretation. Queries for "stocking" are returning women's hosiery and tights. During the holiday season, "stocking" should resolve to Christmas stockings and stocking stuffers. Once the season ends, the policy can be reverted in minutes.

Under the traditional operating model, where search logic is embedded in application code, each of these changes requires an engineering ticket, a code change, a review cycle, a staging deployment, and a production release. In organizations with conservative release processes, that's a timeline measured in weeks, not hours or minutes. The Christmas shopping window closes before engineering can ship the necessary modifications.

The bottleneck isn’t the retrieval engine; it’s the operating model. The core challenge is that business intent cannot be translated into search behavior without engineering acting as a constant intermediary, turning every strategic pivot into a technical ticket.

The anti-pattern: Search logic in application code

Part 1 described how search logic embedded in application code can turn into a "spaghetti" implementation, which creates operational friction. Here’s what that friction looks like at scale. What starts as a few targeted overrides, a filter here, a boost there, grows over time into tens of thousands of lines of if/else branching, regex patterns, and conditional query modifications. This creates problems beyond just technical debt:

This model introduces four systemic frictions that hinder both organizational speed and system scalability:

Coupling. Business strategy changes daily. Application infrastructure should remain highly stable. When both live in the same codebase, a merchandiser's request to boost a seasonal product becomes a deployment risk, and a scoring function refactor can silently break a campaign.

Latency (organizational and computational). A single query behavior change can require a six-week deployment cycle: ticket, investigation, code change, review, staging, release. Furthermore, the application layer lacks any indexing mechanism to efficiently determine which policies apply to a given query, so policy evaluation often adds meaningful latency at query time as the system walks through sequential if/else checks.

Accountability gaps. When results change unexpectedly, nobody can quickly answer why. Was it a synonym update? A scoring change? A new filter added three releases ago? When business logic is distributed across thousands of lines of application code, shipped by different teams across different releases, tracing a relevance change back to its root cause becomes an archaeology project.

Misallocated engineering. This model turns skilled software engineers into full-time relevance mechanics. Instead of building platform capabilities, they spend their cycles translating merchandising requests into code changes and debugging interactions and conflicts between hard-coded business policies.

The paradigm shift: Policies as data

The solution is to decouple business policies from application code entirely. Instead of hard-coding query modifications in middleware, store governed policies as structured documents, each one expressing a discrete business intent, and evaluate them at query time in a dedicated governed control plane layer.

A policy is a first-class data object. It has match criteria (when should this policy fire?), an action (what should it do?), a priority (how does it interact with other policies?), and metadata (a title and a description). The control plane evaluates matching policies, resolves conflicts deterministically, and produces an execution plan including constraints, boosts, and routing decisions that Elasticsearch executes against a product catalog.

For each additional search requirement, the application code doesn't change. The retrieval engine doesn't change. What changes is that business decisions are no longer encoded in code. They live in a policy index as data that can be updated without a deployment.

This changes your org chart, not just your query.

Policies vs. triggers vs. rules

A note on terminology used in this series: a policy refers to this complete governed document, including a trigger (match criteria), rule (action), priority, enabled/disabled, and metadata. A trigger refers to the matching criteria that determines when this policy fires, and a rule refers specifically to the action inside the policy, such as applying a filter or changing the retrieval strategy.

The workflow: Author → Test → Promote

Moving policies out of code and into data opens the door for business-driven search management. But enabling non-technical teams to alter search behavior requires strict operational guardrails. The goal is fast and safe iteration with governance.

To empower non-technical teams to modify search behavior with confidence, we suggest a three-stage workflow: Author, Test, and Promote. Let’s examine the components of this workflow in detail.

Author. A merchandiser creates a policy using structured fields: what the policy should match, what action it should take, and at what priority. The interface guides the business user through what’s expressible.

Elastic Services has built and deployed a governed framework for enterprise ecommerce customers, which has an admin UI that looks as follows:

Test. The policy is validated in a non-production environment where the merchandiser can run representative queries and verify that the policy produces the expected behavior, including how it interacts with other active policies. Because the control plane infrastructure is identical across environments, what works in the test environment will work in production.

Review. Before a policy is promoted to production, it passes through review. Depending on the organization's risk tolerance, this might be a peer review from another merchandiser, an approval from a search lead, or an automated validation that checks for conflicts with existing policies.

Promote. Once approved, the policy is promoted to the production policy index. It takes effect on the next query: no code deployment, no engineering release, no staging build. The entire promotion is a data operation: the same JSON document, moved to a different index.

Disable. If a production policy produces unexpected behavior, it can be disabled immediately without engineering involvement. Disabling removes the policy from query evaluation instantly, without affecting any other policy in the system.

This is the "zero-deploy" promise. It doesn't mean "no process." It means the process operates on policy data, not application code. This distinction compresses the change cycle from weeks to hours or minutes.

Why "zero-deploy" matters for revenue-critical queries

The economics of ecommerce search are asymmetric. A small number of high-volume queries ("milk," "bread," "oranges," "diapers") drive a disproportionate share of revenue. When one of these queries returns unexpected results, the cost is immediate and measurable: Conversion drops, customer complaints spike, and the merchandising team opens an urgent ticket.

Under the traditional model, the response cycle is:

1. The merchant notices the problem.
2. The merchandiser files a ticket with engineering.
3. Engineering investigates, identifies the cause, and writes a fix.
4. The fix goes through code review, staging, and release.
5. Production is updated.

Depending on the organization, steps 2 through 5 may take weeks. For a revenue-critical query during a peak sales period, that latency costs money.

Under a governed control plane, the response cycle compresses:

1. The merchant notices the problem.
2. The merchandiser drafts a policy fix (or modifies an existing policy).
3. The policy goes through review and is published.
4. The fix is live.

The difference isn't just speed. It's ownership. The person closest to the business context (the merchandiser who understands why "oranges" should resolve to produce, not beverages) is the person making the change. Engineering is freed from the daily merchandising loop to focus on the platform. This shift also unlocks something that's nearly impossible under the traditional model: attributing search performance changes to specific business decisions.

Measurability: Which policy moved conversion

When policies are discrete, versioned documents that are stored in an Elasticsearch index, each one becomes independently deployable and therefore its impact can be more easily measured. You can answer questions that are nearly impossible to answer when business logic is scattered across application code:

• Did the "cheap laptops" price threshold policy improve conversion for that query class, or did it suppress it?
• What was the click-through rate impact of the holiday campaign boost?
• When we rolled back the "oranges" category constraint last Thursday, what happened to add-to-cart rates?

This turns search governance into a data-driven discipline. Instead of vague "relevance tuning," where a release contains a dozen changes and nobody can attribute the outcome, you get measurable, attributable impact per policy. Merchandisers can iterate with evidence. Engineers can evaluate whether a policy schema change produced the expected downstream effect. Leadership can see which policies are driving revenue and which are inert.

What this means for each role

For merchandisers and business users

Search behavior becomes something you can directly influence through structured policies without understanding Elasticsearch syntax or scoring algorithms. You can see what policies are triggered for a given query to understand why it produces specific results, and make changes within hours instead of weeks. The same policy mechanism also supports sponsored product placement: A merchandiser can create a boost policy that elevates a product or brand and flags it for a 'Sponsored' indicator in the UI, without requiring engineering involvement or additional infrastructure.

For search engineers

The control plane separates two concerns that are currently entangled: retrieval optimization and business logic. Instead of maintaining tens of thousands of lines of application code that encodes business decisions, you maintain the retrieval engine and the control plane infrastructure. When a merchandiser needs a new campaign boost, they don't need engineering to write it.

This doesn't eliminate engineering involvement. Engineers design the policy schema, maintain the control plane, set guardrails on what policies can express, add new capabilities as required, and handle edge cases that fall outside the policy framework. But the day-to-day operational cadence of modifying query behavior shifts to the people who own the business context.

For site reliability engineers and platform teams

Because policies are structured documents rather than application code, they fit naturally into existing operational workflows. Policies can be stored in version control, reviewed through pull requests, and deployed through the same continuous integration and continuous deployment (CI/CD) pipelines the team already uses. Conflicts between policies are detected and resolved deterministically at query time through the control plane's priority system, not through unpredictable interactions between code branches shipped in different releases.

When something does go wrong, diagnosing the cause is straightforward: Policies are discrete, named, and individually toggleable. A problematic policy can be disabled or deleted immediately without affecting anything else in the system. Compare that to debugging a relevance regression caused by an interaction between a synonym update, a scoring function change, and a new analyzer, all shipped in the same release with no clear attribution.

Beyond manual authoring: Large language model–assisted (LLM-assisted) policy suggestions

The policies described so far are authored by humans (a merchandiser identifying a gap and drafting a fix). But the same governed workflow supports a second mode: LLM-assisted policy suggestion.

An LLM can run offline or in the background, analyzing query logs, identifying patterns where search results underperform, such as queries with high exit rates, low click-through, or frequent reformulations. An LLM can then suggest new policies that enter the same Author → Test → Promote pipeline, where a human evaluates each one before it reaches production.

Governance is the enabler, not the constraint

It might seem counterintuitive: Adding a governance layer makes the system faster to change, not slower. This is the same pattern that works in other domains. CI/CD pipelines don't slow down software delivery; they make it safe to ship frequently. Access control doesn't slow down collaboration; it makes it safe to share broadly.

A governed control plane works the same way. The reason a query behavior change takes six weeks isn't that the code change is complex; it's that nobody is confident enough to ship it faster, because the blast radius is unclear and the rollback path is uncertain.

Governance provides that confidence. When every policy is explicit, every conflict is resolved deterministically, and every change can be instantly disabled and then rolled back (because policies are structured JSON documents that can be version controlled using existing workflows), the cost of iteration drops dramatically. Business teams move at the speed of the market. Engineering focuses on the platform.

From operating model to architecture

The shift from business logic in code to business policies as data is more than a technical refactoring; it's an organizational change that puts relevance ownership with the teams closest to the business context. But it raises an architectural question: How do you evaluate policies at query time without adding latency or turning the control plane itself into a new form of spaghetti?

The next post will dig into exactly that: the design pattern that enables fast, deterministic policy evaluation at query time.

Put governed ecommerce search into practice

The workflow described here, merchandisers authoring, testing, and promoting search policies without engineering deployments, is already available. Elastic Services Engineering designed and built it, and Elastic Services has the skills to deploy it for enterprise ecommerce teams.

If your organization is ready to move from deployment-gated relevance tuning to business-editable search with governance and auditability, we can accelerate your implementation. Contact Elastic Professional Services.

Join the discussion

Have questions about search governance, retrieval strategies, or ecommerce search architecture? Join the broader Elastic community conversation.

Here, we’ll provide some intuition about the problem and how preconditioning solves it.

The problem

BBQ quantizes each dimension of a vector independently: Values above the mean become 1, values below it become 0. This works well when every dimension carries roughly the same amount of information. Transformer-based embeddings tend to have this property naturally such that their dimensions are learned representations that distribute variance evenly.

But there are lots of real-world vectors that aren’t like this. Consider a 784-dimension vector representing a grayscale image, like in the Fashion-MNIST dataset. Some pixels near the center of the image, where the clothing actually appears, vary a lot across the dataset. However, other pixels, such as those near the corners, are mostly one color and barely vary at all. When BBQ quantizes these vectors, the high-variance dimensions lose precision because a single bit can't capture their range, while the low-variance dimensions become useless. The resulting quantized vectors are poor approximations of the originals, and recall suffers.

Picture of a representation of Fashion-MNIST images. (credit: geeksforgeeks.org)

Precondition

To fix the problem, we want to spread the information more evenly across dimensions so that each bit captures roughly the same amount of information.

Preconditioning applies a linear transformation to every vector before quantization. The transformation is an orthogonal rotation that reshuffles how information is distributed across dimensions without changing the distances between vectors. If you want to dig into the math, take a look at this in-depth analysis on optimized scalar quantization (OSQ) with preconditioners.

Here’s a graphic to help illustrate how preconditioning can help when applying quantization. This simplified two-dimensional diagram illustrates the idea that the orthogonal rotation helps to increase the spread, or range, of information that was previously quite compressed. While this two-dimensional animation is not an exact representation of preconditioning, it gives a good intuition for what roughly happens in higher dimensions where buckets of dimensions are transformed independently and a random projection can greatly improve the distribution. Imagine that the y-axis represents one pixel of our Fashion-MNIST corners that are primarily one shade with very low variance and the x-axis represents a pixel of clothing at the center of the image with very high variance. Without preconditioning, quantizing vectors to a single representative point is not a particularly good discriminator.

Let’s look at the data

Today, preconditioning is supported in DiskBBQ. Here’s a benchmark showing the impact when visiting different percentages of the total vector dataset.

Fashion-MNIST Recall (784 dimensions, 60K docs, 5x oversample, k: 10)

GIST (960 dimensions, 1M docs, 5x oversample, k: 10)

SIFT (128 dimensions, 1M documents, 5x oversample, k: 10)

That’s a nice boost in recall; however, this boost comes with a cost. Applying it to all embeddings blindly is inefficient, causing ~2–4% overhead in query latencies with no improvement in recall for datasets that don’t need to be preconditioned. And upwards of 20% additional overhead at index time. For production use cases where you see initially low recall, you may want to evaluate the impact of preconditioning with your specific model and dataset.

Here’s the how

Preconditioning is available for the bbq_disk index type. Simply set precondition to true in the index_options, like so:

{
  "mappings": {
    "properties": {
      "my_vector": {
        "type": "dense_vector",
        "dims": 784,
        "index_options": {
          "type": "bbq_disk",
          "precondition": true
        }
      }
    }
  }
}

Take a look at the dense vector mapping docs for more details.

Conclusion

BBQ is highly effective for deep learning embeddings, but it can be less effective with embeddings that have uneven variance across dimensions, as can occur in feature-engineered vectors. Preconditioning redistributes that variance so quantization can be more effective. On some datasets, like Fashion-MNIST, we see as much as a 74% improvement in recall!

For now, we’ve made preconditioning optional. Hopefully, you feel more capable of knowing when it may be beneficial so you try it out yourself. In the future, we plan to iterate on performance and automatically detect when to apply preconditioning.

This blog's content was developed and verified using Elastic versions 9.2.8 and 8.19.14, along with Filestream Integration versions 2.3.0 and 1.2.0.

Important note: Depending on your environment, some steps may require specific modifications. Furthermore, be aware that dynamic templates were removed from the @package component template starting with Filestream Integration version 2.3.3.

Before starting the reindexing process, it’s important to consider the current storage allocation in your environment. The steps outlined below involve creating a copy of the existing backing index, which will temporarily reside in the hot tier.

Elasticsearch data tiers

• Hot: The hot tier is the Elasticsearch entry point for time series data, storing the most recent, frequently searched data. Hot tier nodes require fast reads and writes, necessitating more resources and faster storage (SSDs). This tier is mandatory, and new data stream indices are automatically allocated here.
• Warm: Time series data can move to the warm tier once it’s being queried less frequently than the recently indexed data in the hot tier. The warm tier typically holds data from recent weeks. Updates are still allowed but are likely infrequent. Nodes in the warm tier generally don’t need to be as fast as those in the hot tier. For resiliency, indices in the warm tier should be configured to use one or more replicas.
• Cold: Data that’s infrequently searched can move from the warm to the cold tier. The cold tier, while still searchable, prioritizes lower storage costs over search speed. Alternatively, the cold tier can store regular indices with replicas instead of searchable snapshots, allowing use of less expensive hardware for older data without reducing disk space requirements compared to the warm tier.
• Frozen: Data that’s queried infrequently or no longer queried moves from the cold to the frozen tier for its remaining lifecycle. This tier uses a snapshot repository and partially mounted indices to store and load data, reducing local storage and costs while still allowing search. Searches on the frozen tier are generally slower than on the cold tier because Elasticsearch may need to fetch frozen data from the snapshot repository. We recommend dedicated frozen tier nodes.

Prerequisites: Determine which fields have conflicts

To determine which fields have mapping conflicts, navigate to Stack Management -> Data Views -> logs-* (using the logs-* data view is the highest hierarchy of data present with the logs- prefix.) If there are any conflicts, there will be a yellow box stating that. You may either click View conflicts or, under the Field type box next to the Search box, select conflict.

Clicking the yellow Conflict button will reveal which indices are associated with which mapping types.

This situation (where the field is mapped as both a keyword and a long) typically occurs because data was ingested before a specific mapping type was defined in the component template for the relevant data stream. In such cases, Elasticsearch attempts to set the mapping based on its dynamic templates.

In order to determine which mapping is appropriate for the field, and if the field is an ECS field, verification with ECS field reference is needed. If the field in question is not an ECS field, its value must be reviewed to determine the correct mapping.

If a field, such as log.offset in this example, isn’t documented in the ECS, the next steps are to investigate the field's value, determine which conflicting mapping type has the most backing indices, and examine the component templates of the other indices.

Typically, the mapping type associated with the highest number of indices is the correct one, but we recommend you verify the value of the field in question to validate this. To confirm the validity of a mapping type (for example, long), you must also verify that the field's value is appropriate for that type. This verification can be done by using Discover to search for the field in question. Reviewing other data streams that contain the same field can provide additional confirmation also.

To review the values present for the field with the mapping issue, navigate back to the yellow Conflict button stated earlier, click the Conflict button, highlight one of the backing indices, and paste into a Discover session. Your Kibana Query Language (KQL) statement should look like the following screenshot, to include the _index: field delimiter.

Prepare the new backing index custom component template

To address the mapping conflict in the data stream, first examine the relevant @package component template. You can find this under Stack Management -> Index Management -> Component Template. Search for the data stream and select the corresponding @package link. This template contains mappings for the fields out of the box and, while it isn’t common to have a mapping mismatch, it’s possible for the more appropriate type to be overlooked.

Review the template to confirm it contains the necessary field nesting and mapping for the field in question. For example, if the template incorrectly lists log.offset as a keyword, this is the source of the issue.

Important: Because modifying @package/managed templates isn’t recommended, you must use or create an @custom component template to correct the mapping type (for example, for log.offset) for all future data.

• We don’t recommend modifying the @package/managed templates, since when you update the integration to a more recent version, any changes you make to the @package template will be overwritten. This is why we recommend using the @custom templates.
• If a data stream is experiencing mapping conflicts, you need to add any missing field (ECS and non-ECS) nestings or mappings to the data stream's @custom component template. Create this template if it doesn't exist yet, and make sure to specify the correct mapping type for the field.
• If you have multiple conflicts in your data view, apply all the necessary missing mappings for the data stream simultaneously so that the reindex is performed once versus multiple times. Having entries for proper data typing in the @custom component template will ensure any future data ingestion will follow the same mapping guideline.

To create the @custom component template (or verify it’s in use and populated), navigate to Index Templates, type in the name of the data stream in question, and click the appropriate @custom template being used by the data stream. If the template is not yet created, a yellow box will appear, allowing you to create the template through the UI.

The screenshot below shows the next page once Create component template is selected. Leave the defaults as is on the first page and click Mappings or Next until you reach the Mappings page.

To explicitly set the mapping for a new field coming in or to update a field that has a mapping conflict, when the data stream rolls over due to configuration set in the index lifecycle policy, an entry is needed for the field that the conflict exists in.

The below will set the mapping for the log.offset field in the @custom component template for the filestream data stream. Repeat the steps to add any custom fields or update necessary fields from the @package with the appropriate mappings, if needed, for this dataset. In this example, when setting offset to Long, the field type will be Numeric and the Numeric type will be Long. Click Add field and then outside of the area to continue.

Once all needed fields have been added, click through to review, and select Create component template when ready. All new data being ingested from this step forward will have log.offset set to long.

Creating the new backing index structure

The new backing index needs to have the existing mappings from the data stream’s  component template, as well as the ECS ecs@mappings component template. The ecs@mappings component template is applied after the data stream’s component as a catchall for additional mappings that potentially weren’t captured in the previous component templates.

Navigate to the browser tab for the data stream's @package mappings. (Go to Stack Management -> Index Management -> Component Template -> logs-filestream.generic@package -> Manage -> Edit.) Once there, click on the Review section, then Request, and finally the Copy button on the right. The JSON contents of the component template copied will ensure the remaining field mappings and settings are retained while we update the log.offset field mapping. The JSON will form the backing structure for the newly reindexed backing index.

Important: If the template’s JSON was not copied and work was continued on with the reindex, the log.offset conflict would be resolved but there would be new conflicts with the integration, as the integrity of the current mappings were not upheld, creating double work to resolve the original issue.

Open a second browser tab, navigate to Dev Tools, and paste the copied content. Now, to clean up what was pasted:

Modifications to the request

1. Index name: Replace _component_template/logs-filestream.generic@package with the name of the backing index you intend to reindex, appending -1 to the end. For example, use PUT <backing index to reindex>-1.

• The appended -1 signifies a reindex and won’t conflict with the default ILM rollover settings, which are based on the index's creation date.

2. Settings: Remove the line "template" (line 3), as well as the very last closing brace for the entire JSON payload; Line 3 should start with "settings": {.

• Replace the inner contents of the settings section with "index.codec": "best_compression". This action will apply Elastic's best compression to the index upon creation.
• Add in "index.lifecycle.name": "logs", as well as a line for "index.lifecycle.rollover_alias": "".
  1. The "index.lifecycle.name": "logs" entry will apply the logs ILM policy to the new backing index. Modify the ILM policy name if you aren’t using logs.
  2. The "index.lifecycle.rollover_alias": "" is blank, since this backing index won’t be rolled over, yet the setting is required to avoid ILM rollover errors into the next ILM phase after hot.

3. Structure: The request should now include both a Settings section and a Mappings section. Inside "mappings": {, you should find "dynamic_templates" and a "properties" section containing hard-coded fields and their mappings.

4. Dynamic templates modification: The current dynamic templates section contains entries for fields that may be overwritten when the ecs@mappings dynamic templates are added next, causing redundancy and extra lines that aren’t needed.

• Remove all sections in "dynamic_templates" except for the second section titled "_embedded_ecs-data_stream_to_constant": {.
• Repeat the same process as described above, gathering the dynamic mappings for the @package component template, but this time the dynamic mappings for ecs@mappings component template.
  • It may be easier to copy the entire contents of the mappings from the UI for the ecs@mappings component template, paste into the working Dev Tools dynamic_templates section, and remove duplicate and unnecessary lines where appropriate. Include these dynamic template setting contents after the"_embedded_ecs-data_stream_to_constant": { entry. The dynamic_templates section should look very similar to the below sample contents in Dev Tools.
• If dynamic_templates are not included/removed altogether, other fields (review the screenshot below) will have double mappings: text and keyword versus the appropriate mappings, if the dynamic_templates section was left included. What’s left should be the "properties" section under "mappings". This will also create issues in the data view by having the fields be double mapped (if not already mapped this way) and will cause additional mapping conflicts.

5. Metadata removal: Delete the last section labeled "_meta", as well as the section labeled "version", if present.

6. Formatting: Auto-indent the remaining sections, and adjust or remove any unnecessary curly braces that would prevent a successful execution.

7. Mapping change: Navigate to the "properties" section, find "log", and then locate "offset" nested underneath. Change the type from keyword to long, and remove the line entry (comma included) labeled "ignore_above": 1024,. If more than one entry was added to the @custom component template created earlier, include them here.

Your Dev Tools console view should now be similar to the example provided below.

PUT .ds-logs-filestream.generic-default-2026.04.14-000001-1
{
  "settings": {
    "index.codec": "best_compression",
    "index.lifecycle.name": "logs",
    "index.lifecycle.rollover_alias": ""
  },
  "mappings": {
    "dynamic_templates": [
      {
        "_embedded_ecs-data_stream_to_constant": {
          "path_match": "data_stream.*",
          "mapping": {
            "type": "constant_keyword"
          }
        }
      },
      {
        "ecs_timestamp": {
          "mapping": {
            "ignore_malformed": false,
            "type": "date"
          },
          "match": "@timestamp"
        }
      },
      {
        "ecs_message_match_only_text": {
          "path_match": [
            "message",
            "*.message"
          ],
          "mapping": {
            "type": "match_only_text"
          },
          "unmatch_mapping_type": "object"
        }
      },
      {
        "ecs_non_indexed_keyword": {
          "path_match": [
            "*event.original"
          ],
          "mapping": {
            "index": false,
            "type": "keyword",
            "doc_values": false
          }
        }
      },
      {
        "ecs_non_indexed_long": {
          "path_match": [
            "*.x509.public_key_exponent"
          ],
          "mapping": {
            "index": false,
            "type": "long",
            "doc_values": false
          }
        }
      },
      {
        "ecs_ip": {
          "path_match": [
            "ip",
            "*.ip",
            "*_ip"
          ],
          "mapping": {
            "type": "ip"
          },
          "match_mapping_type": "string"
        }
      },
      {
        "ecs_wildcard": {
          "path_match": [
            "*.io.text",
            "*.message_id",
            "*registry.data.strings",
            "*url.path"
          ],
          "mapping": {
            "type": "wildcard"
          },
          "unmatch_mapping_type": "object"
        }
      },
      {
        "ecs_path_match_wildcard_and_match_only_text": {
          "path_match": [
            "*.body.content",
            "*url.full",
            "*url.original"
          ],
          "mapping": {
            "fields": {
              "text": {
                "type": "match_only_text"
              }
            },
            "type": "wildcard"
          },
          "unmatch_mapping_type": "object"
        }
      },
      {
        "ecs_match_wildcard_and_match_only_text": {
          "mapping": {
            "fields": {
              "text": {
                "type": "match_only_text"
              }
            },
            "type": "wildcard"
          },
          "unmatch_mapping_type": "object",
          "match": [
            "*command_line",
            "*stack_trace"
          ]
        }
      },
      {
        "ecs_path_match_keyword_and_match_only_text": {
          "path_match": [
            "*.title",
            "*.executable",
            "*.name",
            "*.working_directory",
            "*.full_name",
            "*file.path",
            "*file.target_path",
            "*os.full",
            "*email.subject",
            "*vulnerability.description",
            "*user_agent.original"
          ],
          "mapping": {
            "fields": {
              "text": {
                "type": "match_only_text"
              }
            },
            "type": "keyword"
          },
          "unmatch_mapping_type": "object"
        }
      },
      {
        "ecs_date": {
          "path_match": [
            "*.timestamp",
            "*_timestamp",
            "*.not_after",
            "*.not_before",
            "*.accessed",
            "created",
            "*.created",
            "*.installed",
            "*.creation_date",
            "*.ctime",
            "*.mtime",
            "ingested",
            "*.ingested",
            "*.start",
            "*.end",
            "*.indicator.first_seen",
            "*.indicator.last_seen",
            "*.indicator.modified_at",
            "*threat.enrichments.matched.occurred"
          ],
          "mapping": {
            "type": "date"
          },
          "unmatch_mapping_type": "object"
        }
      },
      {
        "ecs_path_match_float": {
          "path_match": [
            "*.score.*",
            "*_score*"
          ],
          "mapping": {
            "type": "float"
          },
          "path_unmatch": "*.version",
          "unmatch_mapping_type": "object"
        }
      },
      {
        "ecs_usage_double_scaled_float": {
          "path_match": "*.usage",
          "mapping": {
            "scaling_factor": 1000,
            "type": "scaled_float"
          },
          "match_mapping_type": [
            "double",
            "long",
            "string"
          ]
        }
      },
      {
        "ecs_geo_point": {
          "path_match": [
            "*.geo.location"
          ],
          "mapping": {
            "type": "geo_point"
          }
        }
      },
      {
        "ecs_flattened": {
          "path_match": [
            "*structured_data",
            "*exports",
            "*imports"
          ],
          "mapping": {
            "type": "flattened"
          },
          "match_mapping_type": "object"
        }
      },
      {
        "all_strings_to_keywords": {
          "mapping": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "match_mapping_type": "string"
        }
      }
    ],
    "properties": {
      "input": {
        "properties": {
          "type": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }
      },
      "@timestamp": {
        "ignore_malformed": false,
        "type": "date"
      },
      "ecs": {
        "properties": {
          "version": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }
      },
      "log": {
        "properties": {
          "file": {
            "properties": {
              "inode": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "path": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "device_id": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "fingerprint": {
                "index": false,
                "type": "keyword"
              }
            }
          },
          "offset": {
            "type": "long"
          },
          "level": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }
      },
      "data_stream": {
        "properties": {
          "namespace": {
            "type": "constant_keyword"
          },
          "type": {
            "type": "constant_keyword"
          },
          "dataset": {
            "type": "constant_keyword"
          }
        }
      },
      "event": {
        "properties": {
          "original": {
            "index": false,
            "type": "keyword",
            "doc_values": false
          },
          "module": {
            "type": "constant_keyword",
            "value": "filestream"
          },
          "dataset": {
            "type": "constant_keyword",
            "value": "filestream.generic"
          }
        }
      },
      "message": {
        "type": "match_only_text"
      },
      "tags": {
        "ignore_above": 1024,
        "type": "keyword"
      }
    }
  }
}

After your console resembles the example (with any additional custom fields included and custom values specific to your environment), execute the command to create the shell of the new backing index, pausing to resolve any errors that arise.

Begin reindex process

With the shell of the new backing index successfully created, the next step is to reindex and resolve the mapping conflicts.

Important: If the backing index that has the mapping conflict is the most recent index and is the current write index (for example, the ending number for the backing index is -000001), the data stream needs to be rolled over. Rolling over the data stream is needed since the current write index, which is having documents fed into it, is a live backing index and cannot be modified.

With the correct field mapping now applied to the newer write index via the previously created @custom component template, all new documents will reflect this change.

This is performed by executing the following:

POST /_rollover

For example:

POST logs-filestream.generic-default/_rollover

Reindexing involves copying the data from an existing backing index to a new one within the same naming convention, typically to apply necessary changes. These modifications could include updates to a component template or the addition of a new ingest pipeline for the data to be processed through.

Next, the data will be copied from the backing index that has the incorrect mappings into a new backing index. The original backing index has been rolled over, meaning no new documents can be added to it. The new backing index will follow the same naming convention, which preserves data visibility and integrity while applying the correct ILM policy, but will include a -1 suffix to indicate that it has been reindexed.

Adjust the index names as needed and paste the following code into the console. By including wait_for_completion=false, you can track the progress of document copying, which helps estimate the remaining reindexing time. Without this setting, you cannot track the status using the GET _tasks command below and will only be able to check the document count in the newer backing index using GET <backing index name>-1/_count.

Important: If issues arise during the reindex process, don’t rerun the reindex command; doing so will restart the process and create duplicate records in the index ending with -1. If a restart is necessary, first delete the index with the trailing -1, and then execute the preceding PUT command to recreate the new backing index shell.

POST _reindex?wait_for_completion=false
{
  "source": {
    "index": ""
  },
  "dest": {
    "index": "-1"
  }
}

i.e.
POST _reindex?wait_for_completion=false
{
  "source": {
    "index": ".ds-logs-filestream.generic-default-2026.04.13-000001"
  },
  "dest": {
    "index": ".ds-logs-filestream.generic-default-2026.04.13-000001-1"
  }
}

Upon execution, the response will include a task ID. You can monitor the reindex progress using this ID with the command: GET _tasks/<task ID>.

The duration of the reindex depends on the volume of data in the original index. The completion can be tracked by looking for "completed": true when executing the GET command, which should yield a similar output.

GET _tasks/<task ID>

With the reindexing process now finished for the document count, the next step is to verify that the mappings for the new backing index and the specific field in question are correct.

GET -1/_mapping

For example:

GET .ds-logs-filestream.generic-default-2026.04.13-000001-1/_mapping

You can verify that the mapping for log.offset is as shown below. To confirm that other fields have only a single mapping entry (not both text and keyword), compare them to a field that was not part of the dynamic template section in the preceding PUT command.

If the backing index that’s being reindexed has a large number of documents, it’s helpful to check the status of those documents being copied to the new backing index; this can be done by the following two Dev Tools commands to compare the counts.

GET .ds-logs-filestream.generic-default-2026.04.14-000001/_count

GET .ds-logs-filestream.generic-default-2026.04.14-000001-1/_count

Once the counts are verified to match and the correct mappings are present, update the data stream to include the new backing index, preventing an orphaned backing index in index management, where the ILM policy will never occur on the backing index.

• The return should be an acknowledgment of true, if successful.

POST _data_stream/_modify
{
  "actions": [
    {
      "add_backing_index": {
        "data_stream": "logs-filestream.generic-default",
        "index": ".ds-logs-filestream.generic-default-2026.04.14-000001-1"
      }
    }
  ]
}

Verify the new backing index is added with the following command, making sure the ilm_policy is correct:

GET _data_stream/logs-filestream.generic-default

Check the ILM status of the backing index next with the following command:

• It’s normal to see that the index is in hot, as it was created very recently (review line 8 or 10).

GET .ds-logs-filestream.generic-default-2026.04.14-000001-1/_ilm/explain

Execute the following to transition the backing index from the hot tier to the next appropriate tier that’s after the hot phase for the ILM policy for this data stream. The specific values for phase, action, and name in the current_step below can be referenced from lines 11, 13, and 15, respectively, in the provided screenshot above.

The next_step value indicates the subsequent ILM phase or data tier to which the index will transition to.

For example:

POST _ilm/move/.ds-logs-filestream.generic-default-2026.04.14-000001-1
{
  "current_step": {
    "phase": "hot",
    "action": "rollover", 
    "name": "check-rollover-ready"
  },
  "next_step": {
    "phase": "warm" 
  }
}

• It isn’t necessary, but as a safety measure, you may execute the _ilm/explain command again to ensure the backing index has moved to the next phase and is no longer in hot.

Once the following conditions are met, you can safely delete the original backing index that had mapping conflicts:

1. A new backing index has been successfully created.
2. Documents have been moved to the new index, and the document counts match.
3. Mappings have been corrected (both data stream specific and ECS).
4. The data stream incorporates the new backing index.
5. The ILM policy has been applied and has moved the index out of the hot phase.

Important: Alternatively, before deleting the original index, you can check the Data Views page. Select logs-* and verify that the reindexed backing index (which ends in -1) now appears in the long section. The original backing index should still be present under keyword. If the reindexed backing index is not in the long section, go back and review the preceding steps and make any necessary corrections.

For example:

DELETE .ds-logs-filestream.generic-default-2026.04.14-000001

After resolving the conflicts, return to the Data Views page and select logs-*. If the conflict was solely related to log.offset, you should no longer see any conflicts listed. If there were other conflicts, the original backing index should no longer appear in the conflict list; instead, the new backing index should now be listed in the long section.

You can also verify in Discover that the log.offset field now displays the appropriate icons.

Continue this process, repeating the above steps for every backing index that has a mapping conflict until all are successfully resolved.

References:

• ECS field reference
• Reindex documents

Final thoughts

By following the steps in this blog, you will resolve mapping conflicts and ensure that all new data is correctly mapped. This is achieved by linking the necessary component templates to your data source. This workflow not only fixes the immediate issues but also establishes a secure and repeatable process for managing schema changes as your data and requirements evolve.

How we built Elasticsearch simdvec

Every vector search query in Elasticsearch, whether Hierarchical Navigable Small World (HNSW) traversal, inverted file (IVF) scan, or reranking pass, reduces to the same problem: computing distances between vectors, millions of times per query. Elasticsearch supports a wide range of data types and quantization strategies, from float32 to int8, bfloat16, binary, and Better Binary Quantization (BBQ). Each comes with different trade-offs between memory, throughput, and recall. Behind all of it is a single engine: simdvec.

We built simdvec to make every distance computation as fast as the hardware allows. In this post, we explain why we built it, what’s inside, and where it delivers the most impact.

Built like a race car

As Formula 1 enthusiasts, with one of us having previously worked with the Ferrari Formula 1 Team, we see a clear parallel. A Formula 1 car is designed with a single purpose: to achieve the best lap time. Engine power, aerodynamics, and chassis design only matter insofar as they contribute to that outcome. The same is true of a vector database, where indexing throughput, query latency, and recall define success.

While the end result is what matters, reaching the highest levels of performance requires each component to be at its best. It can’t just be good enough, it has to be the best in its category. Simdvec is built with that mindset, focusing on a critical part of the system: the engine. It’s a purpose-built, single instruction multiple data (SIMD) optimized kernel library that provides hand-tuned native C++ distance functions, called from Java via the Panama foreign function interface (FFI). It supports bulk scoring, cache line prefetching, and all vector types and layouts used in Elasticsearch.

That’s the engine behind every query.

Why we built our own

We started in 2023 with the Panama Vector API in Apache Lucene. It worked well for float32 dot products, but Elasticsearch's needs quickly outgrew what it could provide. Elasticsearch supports a wide range of quantized vector types: int8, int4, bfloat16, single-bit, and asymmetric BBQ. Each has different SIMD strategies, packing layouts, and accumulator requirements. Beyond type coverage, Elasticsearch's scoring paths demand more than single-pair throughput: HNSW needs to score several graph neighbors in one pass, IVF needs bulk scoring of thousands of candidates with prefetching, and disk-based scoring needs to work directly on mmap'd memory without copying. We looked at what was available, and nothing covered the full set.

So we built simdvec: hand-tuned native C++ kernels called from Java via FFI, with bulk scoring, prefetching, and support for every vector type Elasticsearch uses. By owning the library, we control the full stack. When we add a new quantization type like BBQ, it gets a tuned SIMD kernel wired all the way through the system. We don't wait for an upstream library to support it, and we don't compromise on performance for any type. Every vector query in Elasticsearch, whether HNSW, IVF, reranking, or hybrid, runs on this engine, built around the operations and types we actually use.

Simdvec has separate native libraries for x86 and ARM, each with multiple instruction set architecture (ISA) tiers selected at startup. The call overhead from Java via FFI is very low at single-digit nanoseconds.

The landscape

We're not the only ones building SIMD-optimized vector distance kernels. The ecosystem is rich, and we wanted to understand how simdvec performs. Not to rank projects, but to provide context and explain where Elasticsearch's engine sits. We selected three projects as reference points, each representing a different approach:

• jvector: A Java approximate nearest neighbor (ANN) library that uses the Panama Vector API for vectorized distance computation, with optional native C acceleration on x86.
• FAISS: A widely deployed open source vector search framework, with hand-tuned AVX2/AVX-512 kernels.
• NumKong (formerly SimSIMD): A comprehensive suite of over 2,000 hand-tuned SIMD kernels spanning distance functions, matrix operations, and geospatial computation.

Each project serves a different purpose and makes different trade-offs. We include reference numbers from them to give context for simdvec's performance on the specific operations that Elasticsearch needs.

How we measure

The simdvec and jvector benchmarks are written in Java with JMH, the standard JVM microbenchmark harness, with FFI overhead included. For NumKong benchmarks and FAISS benchmarks, we wrote small C/C++ harnesses using Google Benchmark, which is the standard C++ microbenchmark framework. Both frameworks report nanoseconds per operation with warmup and iteration calibration. We verified via hardware performance counters that all libraries are using SIMD on both platforms. All the benchmark code is publicly available in the linked GitHub repositories (and, in the case of simdvec, in the elasticsearch repository).

Software: JDK 25.0.2, JMH 1.37, GCC 14, Google Benchmark (latest).

One vector at a time

The most fundamental operation in vector search is computing the distance between two vectors. Every HNSW neighbor evaluation, every IVF candidate score, every reranking comparison reduces to this inner loop.

We measured single-pair throughput at 1024 dimensions on both platforms, starting with float32, the baseline type and the one where the ecosystem is most competitive. We compare simdvec against FAISS and jvector; we excluded NumKong as it uses float64 accumulators for float32, making it 3.2x-5.3x slower (depending on platform), prioritizing numerical precision over throughput. To keep the comparison like-for-like, we benchmark NumKong on int8 instead, where it uses the same accumulator strategy as simdvec.

On x86, FAISS AVX-512 is the fastest single-pair kernel at 23 ns. Simdvec AVX-512 follows at 28 ns, a gap that reflects the FFI call overhead. Both use 512-bit FMA with multi-accumulator unrolling. At the AVX2 level, the two are much closer, 36 ns and 39 ns respectively, both constrained by the 256-bit register and memory load widths. jvector lands at 44 ns using the Java Panama Vector API. Panama generates good SIMD code, but hand-tuned C++ intrinsics retain an edge.

On ARM, simdvec leads at 70 ns, well ahead of jvector at 110 ns and FAISS at 156 ns. Simdvec has hand-tuned NEON kernels for aarch64. Jvector has no native ARM code and relies on Panama. FAISS relies on compiler auto-vectorization rather than explicit NEON intrinsics, which accounts for the wider gap. This reflects a practical advantage of owning the kernel library: when Elasticsearch expanded to Graviton, we added purpose-built NEON kernels. Neither jvector nor FAISS have prioritized ARM native code to the same degree.

But Elasticsearch doesn't only score float32. Int8 quantization reduces memory by 4x, bfloat16 by 2x, and BBQ by 32x. Each type needs its own SIMD strategy, and simdvec provides hand-tuned native kernels for all of them.

Of the libraries we compared, only NumKong has comparable kernels for int8. We measured int8 dot product, squared Euclidean, and cosine at 1024 dimensions.

Int8 single-pair scoring(1024 dimensions, ns/vec op – lower is better)

On both architectures, NumKong is equal or faster at small-to-medium dimensions, where the difference is largely due to lower call overhead (direct C call vs Java FFI). At larger dimensions simdvec catches up, where the more efficient kernel implementation (which uses cascade unrolling) amortizes the call cost: As dimension increases, this gap closes and eventually reverses. Crossover is at dimensions between 768 and 1536, depending on function and architecture.

Despite the slightly higher overhead of Java FFI, simdvec is on par with highly optimized C/C++ libraries. Not only is it the only library with optimized kernels for both float32 and int8, but it also leads on ARM and only slightly behind FAISS on x86 (for float32), and very close to NumKong on both architectures (for int8). And for bfloat16, int4, binary, and BBQ, while alternatives exist, simdvec distinguishes itself through hand-tuned SIMD tailored to each type's data layout.

But a production search engine doesn’t score one vector at a time; it scores thousands per query. The next question is what happens at that scale.

Thousands at a time

Single-pair performance is only part of the picture. What matters in practice is how systems behave under load. A single HNSW query may score hundreds of graph neighbors. An IVF scan may score thousands of posting list entries. A reranking pass may score tens of thousands of candidates. Single-pair throughput matters, but what matters more is how fast you can score many vectors, and how gracefully performance degrades as the working set spills out of CPU caches.

Simdvec provides bulk scoring for every data type. These aren't just loops over single-pair kernels; they use multi-accumulator inner loops that load the query vector once per dimension stride and share it across multiple document vectors, with explicit cache-line prefetching for the next batch. Neither jvector nor FAISS offer an equivalent (at the time of writing). Jvector has no bulk API, so callers score one pair at a time in a loop. FAISS exposes fvec_inner_products_ny, which, at the time of writing, is implemented as a loop over its single-pair distance function with no query amortization or prefetching.

Float32. To measure the impact at the kernel level, we scored a single query against increasing numbers of 1024 dimension float32 document vectors using random access patterns that simulate HNSW-like scattered graph neighbor lookups. The three dataset sizes, 32, 625, and 32,500 vectors, are chosen so the working set exceeds L1, L2, and L3 cache, respectively.

When the data fits in cache, simdvec is the fastest on both platforms, but the margins are modest since kernel arithmetic dominates. The real separation appears as the working set grows beyond L3. On x86, simdvec scores at 95 ns per vector, while FAISS needs 165 ns and jvector 412 ns. On ARM, the pattern is the same: simdvec holds at 162 ns, while FAISS climbs to 347 ns, and jvector to 476 ns. The prefetching and query amortization in simdvec keeps memory latency hidden in a way that a simple loop over single-pair kernels cannot match, and the advantage widens precisely where real search workloads operate, deep in main memory.

Int8. The same pattern holds for quantized types. We measured int8 dot product bulk scoring at 1024 dimensions with dataset sizes chosen to exceed the same L1, L2, and L3 cache boundaries, comparing simdvec's bulk scoring against NumKong single-pair scoring in a loop.

On x86, simdvec is between 1.2x and 1.9x faster, driven by the combination of explicit prefetching and batch processing. On ARM, simdvec wins again (1.7x to 1.9x faster) across all dataset sizes. The advantage comes from batch processing four vectors at a time, providing memory-level parallelism via an interleaved access pattern. In both cases, the most striking result is what happens at the largest dataset size, where it matters the most.

Results for squared distance and cosine show a similar pattern, with speedups of 1.4x to 1.8x for ARM, and of 1.3x to 3.0x for x86 (details here).

When memory matters

Production vector indices typically don't fit in CPU cache. A 10M-vector int8 index at 1024 dimensions is 10GB. Scoring candidates means streaming data from DRAM, and that's where bulk scoring architecture makes the difference.

We used hardware performance counters to measure what happens inside the CPU during bulk scoring and found that hiding memory latency requires two fundamentally different strategies, one per architecture.

On x86, explicit prefetching eliminates cache misses. The bulk kernel processes vectors sequentially, one fully computed before the next, while issuing prefetch instructions for the next batch. Future data is pulled into L1 before the CPU needs it.

On ARM, the same sequential approach performed poorly, even with prefetching. Instead, the bulk kernel interleaves loads from four vectors at every stride position, giving the out-of-order engine four independent memory streams. The CPU is not fetching data any faster, but rather waiting less by always having something else to compute while memory requests are in flight. Detailed analysis can be found in this GitHub issue.

The numbers tell two different stories:

1. On x86, prefetching turns 139K cache misses into 19K, and instructions per cycle (IPC) more than doubles. The bulk advantage grows with dataset size, from 1.2x in L2 to 2.8x beyond L3, because prefetching hides progressively more expensive DRAM round trips.
2. On ARM, cache misses barely change. What changes is utilization: Backend stalls drop 40% because the interleaved access pattern keeps the pipeline fed. This advantage stays a consistent 1.8x regardless of dataset size, because memory-level parallelism applies whether data comes from cache or DRAM.

Two architectures, two strategies, one result: At production scale, simdvec keeps the CPU pipeline busy even when vectors are scattered across main memory.

What this means for Elasticsearch users

These kernel-level capabilities compound. A single vector search query may compute millions of distance operations: HNSW graph traversal, candidate scoring, reranking. Across thousands of concurrent queries, nanoseconds per operation translate directly to query latency and cluster throughput. Whether you use float32, int8, bfloat16, or BBQ, whether your index is in memory or on disk, simdvec is the engine underneath, and every one of those operations runs through the same engine, tuned down to the last nanosecond.

The key takeaway is that at production scale, vector search performance isn’t primarily determined by raw SIMD throughput. It’s dominated by how efficiently the system hides memory latency while sustaining compute across millions of small operations.

The simdvec kernels improve with almost every Elasticsearch release. When new quantization types and hardware platforms emerge, they get tuned kernels from day one. And existing types continue to get faster as we refine the implementations that are already shipping.

With this integration, Elasticsearch users gain a new deployment option that keeps data inside their security perimeter, delivers predictable infrastructure costs, and runs natively on Google Cloud. At the same time, the broader Google Cloud ecosystem gains access to Jina's purpose-built, state-of-the-art search and retrieval models.

This is the first stage of a broader rollout. Together with the models coming next, the lineup will form a complete retrieval stack: Embed your data, embed queries, retrieve and rerank candidates, and extend search to images with multimodal embeddings, all on infrastructure you control. You can start today with jina-embeddings-v3, the model already powering production search pipelines across the Elasticsearch ecosystem via Elastic Inference Service (EIS).

Every model runs on a single NVIDIA L4 (24 GB), the most cost-efficient GPU tier on Google Cloud. Most other embedding models on Google Cloud Model Garden require an A100 80 GB or H100, roughly three times the per-hour instance cost before you even start counting tokens.

No additional commercial license is required when deployed through Vertex AI.

Why Model Garden?

Why deploy through Model Garden instead of hitting an API? It comes down to three things: control, cost, and context.

Your data never leaves the house

The biggest draw for most developers is the self-deploy architecture. When you deploy a Jina model through Model Garden, the weights run on GPU instances inside your own Google Cloud project and your own VPC. This is a game-changer for anyone working in industries with data security concerns, like finance or healthcare. Because there are no external API calls, your sensitive data stays within your security perimeter.

Scaling with prediction

Instead of paying every time you embed a sentence or rerank a document, you pay a flat hourly instance cost. And because every Jina model can run on a single NVIDIA L4, the most affordable GPU tier on Google Cloud, the barrier to entry is low. Whether you process a thousand requests or a billion, your infrastructure bill stays predictable. This is a setup that actually rewards you for growing your traffic rather than taxing you for it.

Everything under one roof

If your data is already sitting in Elasticsearch on Google Cloud, BigQuery, or Cloud Storage, it makes sense to keep your inference engines nearby. By deploying through Model Garden, Jina search foundation models inherit all the enterprise features you are already using: identity and access management (IAM) for access control, unified billing on your existing Google Cloud invoice, and the ability to plug into Vertex AI Pipelines for machine learning operations (MLOps) workflows.

While the Jina AI Cloud API and Elastic Cloud offer the fastest path for bursty traffic or existing search workflows, Model Garden is ideal for enterprise applications requiring strict data security and predictable costs at scale. Elastic wants to meet you where you are.

Jina AI models

jina-embeddings-v3

Our proven multilingual embedding model with 572M parameters and 8K token context. Scores 65.5 on Massive Text Embedding Benchmark (MTEB) English. Supports five task-specific Low-Rank Adaptation (LoRA) adapters (retrieval query/passage, text-matching, classification, clustering) and Matryoshka truncation from 1024 to 64 dimensions. Already widely adopted across the Elasticsearch ecosystem via EIS.

We’re leading with v3 because many production systems already depend on it. If you’re migrating a v3-based pipeline to Google Cloud, you can now run the same model natively without changing your embedding dimensions or reindexing.

jina-embeddings-v5-text (small and nano)

Our fifth-generation text embedding models, released February 2026, achieve top-tier performance, competing with models many times their size.

v5-text-small (677M) scores 67.0 on the Multilingual MTEB (MMTEB) benchmark suite, encompassing 131 tasks of nine task types, and 71.7 on the MTEB English benchmark. It’s the strongest sub-1B multilingual embedding model on the MTEB Leaderboard.

v5-text-nano (239M) scores 65.5 on MMTEB. No other model under 500M parameters reaches this level. At less than half the size of most comparable models, it’s the natural choice for edge and latency-sensitive deployments.

Both models support:

• Four task-specific LoRA adapters: Retrieval, text-matching, classification, clustering. Selecting an appropriate adapter via task parameter at inference time.
• Matryoshka dimension truncation: Reduce embedding dimensions from 1024 (or 768 for nano) down to 32. Quality loss is minimal at moderate truncation (for example, 256 dims). Halving dimensions roughly halves storage.
• Binary quantization: Compress 1024-dim embeddings from 2KB to 128 bytes with binarization. Special training makes this compression minimal losses.
• Multilingual: 119 languages (small) and 93 (nano).

jina-reranker-v3

A 0.6B parameter multilingual listwise reranker built using a last but not late interaction architecture. The query and up to 64 candidate matches are entered into a single 131K-token context window, and the model performs cross-document comparison before scoring. Jina Reranker v3 achieves 61.94 nDCG@10 on BEIR, outperforming the model being 6× smaller in size.This is fundamentally different from pointwise rerankers that score each document in isolation, producing better results, especially for passage retrieval from single documents.

jina-clip-v2

A 0.9B multimodal, multilingual embedding model that maps text and images into a shared 1024-dimensional space. It supports:

• 89 languages for text-image retrieval.
• 512×512 image resolution.
• 8K token text input.
• Matryoshka truncation from 1024 to 64 dimensions for both modalities.

Highly competitive on image-to-text benchmarks, including multilingual tasks.

Getting started

Jina Embeddings v3 is live on Model Garden today. Here’s how to get it running.

You need a Google Cloud project with the Vertex AI API enabled and enough GPU quota for at least one g2-standard-8 instance (NVIDIA L4). If you’re new to Google Cloud, start with the setup guide.

The Model Garden page for Jina Embeddings v3 walks you through the full flow: Upload the model, create an endpoint, pick your machine type, and deploy. Open it in your own project, and follow the guided steps. A100 and H100 machines are also available where region and quota allow, but L4 is all you need to start.

From click to first embedding, the whole process takes a few minutes.

What comes next

Jina Embeddings v3 is the starting point. In the coming weeks, we’ll bring the rest of the Jina retrieval stack to Model Garden: v5 text embeddings (small and nano), jina-reranker-v3, and jina-clip-v2 for multimodal search. All will run on a single L4 GPU with the same self-deploy model.

MCP Apps change the shape of that answer. A tool can now return an interactive UI alongside its text summary, and the host (Claude Desktop, Claude.ai, VS Code Copilot, Cursor) renders it inline in the conversation. The model keeps the compact text for reasoning. The human gets a live, clickable interface right next to the chat.

Three properties make this a different kind of integration than "a webhook that returns a URL":

• Context preservation. The UI lives inside the conversation. No tab switching, no hand-offs.
• Bidirectional data flow. The UI can call tools on the MCP server for fresh data, and the host can push new results from the agent back into the UI. No separate API layer or authentication plumbing.
• Sandboxed trust boundary. MCP Apps run in a host-controlled iframe. They cannot access the parent page, read cookies, or escape their container.

Security operations run on triage, investigation graphs, and Attack Discovery, where an AI agent correlates hundreds of alerts into a handful of attack chains. Observability means distributed traces and time-series drill-downs. Building in Kibana means a dashboard grid. Flatten any of that to text and you lose the thing that makes it useful. We built MCP Apps for all three and are open-sourcing them together, so the same conversation can move from a triage queue to a dependency graph to a live dashboard without ever leaving the chat.

Each of the three reference apps is one MCP server serving many interactive views, not a bundle of separate products. The security app alone surfaces six dashboards that share the same server shell, the same tool-visibility model, and the same host bridge. The pattern is small; the surface area is where the value compounds.

Elastic Security MCP App

Why it matters for the SOC

When an agent tells a SOC analyst, "There are 47 alerts on host-314, here's a summary," it hasn't done any work. It's just pointed at where the work starts. The actual work lives in the alert list, the process tree, the investigation graph, and the case file. You can't do it from a paragraph of text.

The security MCP App returns the workflow itself. The analyst prompts the agent, and the agent returns an interactive dashboard in the chat where the analyst can drill into alerts, run threat hunts, correlate attack chains, and open cases, all without losing the thread of the conversation. And because the findings, queries, and cases all land back in Elasticsearch, the same investigation is waiting in Kibana where the analyst can pick back up after the conversation has been closed.

Six interactive dashboards

The Elastic Security MCP App ships six interactive elements, one per major SOC workflow. Each one is a React UI that renders inline when the agent calls the corresponding tool:

Each tool returns a compact text summary that the model can reason over, alongside the interactive UI the analyst acts on. The UI can also fetch fresh data behind the scenes through the MCP host bridge. The full tool model and bridge API live in the repo's architecture doc.

The app also ships with Claude Desktop skills, SKILL.md files that teach the agent when and how to use each tool. Download pre-built skill zips from the latest release.

From Alert to Case

Four skills cover the core SOC loop. Each one picks up a prompt, calls a tool, and returns an interactive dashboard alongside a text summary that the model reasons over. An analyst's day usually starts with an alert queue.

Triage alerts. Ask the agent to triage by host, rule, user, or time window. The Alert Triage skill returns a dashboard of AI verdicts above the raw alert list, with one verdict per detection rule classifying that rule's activity as benign, suspicious, or malicious, each with a confidence score and a recommended action. Click any alert to open a detailed view with a process tree, network events, related alerts, and MITRE ATT&CK tags. No need to context switch between AI conversation and your alerts dashboard inside Kibana, everything is happening in real time inside your conversation.

Hunt for threats. Ask the agent to hunt across your indices. The Threat Hunt skill returns an ES|QL workbench with the query pre-populated and auto-executed, with every entity in the results clickable for drill-down. The model writes a short read-out below the table: what's unusual, what's connected, what's worth a closer look. It then offers the next pivot: either go deeper into the threat hunt, or start a new skill within the MCP app that complements the work done so far. What ties this really well is launching an Attack Discovery to gather more context on the alerts you’ve gone deep with and the threats you have hunted so far.

Run Attack Discovery. The Attack Discovery skill triggers the Attack Discovery API and returns a ranked list of findings. Each finding is a set of related alerts stitched into one attack chain, with MITRE tactics, a risk score, a confidence label, and the impacted hosts and users surfaced up front. The agent's summary lands below the findings in the same rank order, and the conversation now holds everything needed to act: hunt queries, triage decisions, correlated chains, all staged for the next step.

Open cases without leaving the chat. Approve findings in bulk or ask the agent to open cases for specific alerts. The Case Management skill creates one case per approved finding (source alerts attached, MITRE tactics inherited from the attack chain) and renders the live case list inline. Click a case for its detail view, which includes a row of AI action buttons: Summarize case, Suggest next steps, Extract IOCs, and Generate timeline. Each one drops a structured prompt back into the chat, so the agent picks up the case context without needing a reintroduction. The agent's summary sits below the case list and covers the full IR queue, including the cases just opened and earlier findings that still need one.

Every step in this walkthrough runs the same loop: a prompt comes in, the skill picks it up, the tool returns a compact text summary for the model to reason over, alongside an interactive UI that the analyst acts on. Chain the skills together, and they compose into an end-to-end SOC flow — hunt, triage, correlate, open cases, and drive the next pivot, all with the model carrying the session context across every step. Invoke any one on its own and it's still the full dashboard, pointed at whatever slice of your data you name. Either way, the work accumulates inside the conversation; no tab switching, no copy-paste, no hand-offs.

Two more skills round out the app: a detection-rule browser for tuning noisy rules, and a sample-data generator for spinning up realistic ECS events against a fresh cluster. A follow-up post will go deep on all six: investigation graph, attack-flow canvas, and end-to-end walkthrough.

"The MCP App for Elastic Security bridges the gap between automated detection and manual hunting. By bringing our security data directly into a single interface within Claude Desktop, we surfaced 'silent' threats in under an hour, risks that didn't trigger standard alerts but required immediate action. It's a force multiplier for our analysts." Mandy Andress: Chief Information Security Officer (CISO), Elastic.

How it works

Each MCP App is a small Node.js server whose tools return both a compact text summary for the model and a React UI that the host renders inline. Because it's built on the open MCP App spec, the same server runs on any compatible host - see the repo's architecture doc for the full design.

Try it

Requires Elasticsearch 9.x with Security enabled, plus Kibana for cases, rules, and Attack Discovery. The fastest path is the one-click .mcpb bundle from the latest release - double-click it in Claude Desktop, and you'll be prompted for your Elasticsearch URL and API key. Setup guides for Cursor, VS Code, Claude Code, Claude.ai, and building from source are in the repo.

Elastic Search MCP App: Dashboards built from conversation

Every Kibana user knows the dashboard detour: leave what you're working on, open Kibana, pick an index, pick fields, pick a visualization, tweak, and save. That's five context switches before a single chart is on screen.

The new example-mcp-dashbuilder reference app collapses that into a prompt. Ask the agent to "build me a dashboard with revenue metrics, order trends, and category breakdowns" and the dashboard comes back inside the conversation without any tab-switching required.

Behind that prompt, the agent explores your Elasticsearch data via ES|QL and selects chart types to match the data: bars for comparisons, lines for trends, metric cards for KPIs, and heatmaps for two-dimensional patterns. It lays panels out on Kibana's 48-column grid using the Elastic UI Borealis theme, and the result is fully interactive: you can drag, resize, and group panels into collapsible sections right in the chat. When the dashboard looks right, a single tool call exports it to Kibana, preserving ES|QL queries and custom colors. You can also import existing Kibana dashboards back into the chat for AI-assisted editing.

The principle is the same one behind the Security app: when the artifact is the product, returning it inside the conversation closes the loop between describing what you want and seeing it.

Under the hood, it follows the same MCP App pattern. A Node.js server registers a view_dashboard model-facing tool alongside a set of app-only tools the UI calls directly (data fetching, layout persistence, time-field detection, export/import). The dashboard view itself is a single self-contained HTML file bundled with vite-plugin-singlefile and served as an MCP App resource. Builders forking the repo get the same server shell and host bridge they see in the Security app, pointed at a different job. The example-mcp-dashbuilder README has the full architecture and chart-type reference.

Elastic Observability MCP App

The third reference app, Elastic Observability MCP App, tackles the SRE version of the same shape problem. When something breaks in production, the answer the on-call engineer needs is not a chart, it is a diagnosis stitched together from K8s metrics, APM topology, ML anomalies, and risk assessment. The shape of the answer is a causal story: what failed, why, what depends on it, and what to do next.

Six tools supporting the observability investigation workflow

Cluster health rollup

Ask "what's broken?" or "give me a status report" and get a one-shot orientation: overall health badge, degraded services with reasons, top pod memory consumers, anomaly severity breakdown, and service throughput — all in one inline view. This is the starting point when something feels off, but you don't know where to look. The view adapts based on what your deployment supports. APM gives you service health. Kubernetes metrics add pod and node context. ML jobs layer in anomalies.

Service dependency graph

Ask "what calls checkout?" or "show me the topology" and get a layered dependency graph — upstream callers, downstream dependencies, protocols, call volume, and latency per edge. Let’s ask Claude to “Show me the service dependencies of the frontend”:

Zoom, pan, and hover to get all the details you need to understand the complex service relationships:

Assess risk with a blast radius

Ask "what happens if my k8s node goes down?" and get a radial impact diagram: the target node at center, full-outage deployments in red, degraded in amber, unaffected in gray. A floating summary card shows pods at risk and rescheduling feasibility. Single-replica deployments are flagged as single points of failure. 

Observe

The agent's primary access primitive for Elastic — one tool, three modes for three different needs. Say "what's CPU doing right now?" and it runs an ES|QL query once and returns a table. Say "show me frontend latency for the next 60 seconds" and it live-samples the metric, updating the chart in-place. Say "tell me when memory drops below 80MB" or "watch for anything unusual for the next 10 minutes" and it blocks until the condition fires or the window expires. The view adapts to the mode: a results table for one-shot queries, a live trend chart with current/peak/baseline stats for sampling and threshold conditions, and a severity-scored trigger card for anomaly mode.

How it works

Same MCP App pattern as the Security and Search apps: a Node.js server, six model-facing tools wired to six single-file view resources. Tools are grouped by deployment backend (Universal, APM-dependent, K8s-dependent, ML-dependent), so the agent and the user both know up front which tools apply to a given deployment instead of discovering capability gaps at call time. The MCP App also includes an example Agent Builder workflow: k8s-crashloop-investigation-otel that can trigger on a Kubernetes alert and return a structured root cause summary before you've opened a single dashboard.

The Agentic Stack, Interactive

Three properties about this pattern are worth stating directly. First, the tool result is no longer the end of the work, it is the start of it: the conversation returns an interface you can act on, not a summary you have to act from. Second, the same agent, the same model context, and the same conversation thread can now move across Security, Search, and Observability surfaces without leaving the conversation. Third, this only works because Elasticsearch and Kibana already expose the APIs. The MCP App is a thin interactive layer over the product capabilities we already ship.

Attack Discovery already powers the correlated findings view inside this app. Inside the stack, the same agentic pattern goes further: Elastic Workflows automate the deterministic steps (enrich entities, create cases, isolate hosts), while Agent Builder reasons over the data and invokes those workflows as tools. The MCP App brings that same security surface into the external conversation; Workflows and Agent Builder deepen it inside the stack. Different entry points, same Elastic APIs underneath.

Try it:

• Security: example-mcp-app-security
• Search and dashboards: example-mcp-dashbuilder
• Observability: example-mcp-observability

Don't have an Elasticsearch cluster yet? Start a free Elastic Cloud trial. For more on the building blocks behind the security app, see the related Security Labs posts on Elastic Workflows and Agent Builder, Agent Skills, and Attack Discovery.

Today, that story gets a lot simpler. Elastic Cloud API keys can now be used to authenticate directly against Elasticsearch and Kibana APIs on Elastic Cloud Serverless. You can now use a single credential to manage your organization's resources and run data operations, like Elasticsearch Query Language (ES|QL) queries, data ingestion, and alerting.

Let’s look at why we built this, how we engineered a globally distributed identity layer to make it possible, and how it lays the foundation for cross-project search.

The secret management burden

Building reliable CI/CD pipelines, GitOps workflows, or Terraform automation around data platforms comes with a hidden cost: secret sprawl.

In the previous model, developers faced a disjointed authentication story:

• Control plane (Elastic Cloud API keys): Organization-scoped keys used to create projects, invite users, and manage billing via the Elastic Cloud API.
• Data plane (Elasticsearch API keys): Project-scoped keys created inside a specific Serverless project to interact with Elasticsearch and Kibana APIs.

This meant that your deployment script had to authenticate to Elastic Cloud, provision a Serverless project, extract a newly minted Elasticsearch API key from that specific project, and then inject that second key into the downstream application or automation tool, resulting in complex pipelines, fragmented audit logs, and a higher risk of credential leaks.

Unified authentication in Elastic Cloud Serverless

With this release, the split is gone for Serverless projects. You can now create an Elastic Cloud API key that’s explicitly authorized for Cloud, Elasticsearch, and Kibana APIs.

• Before: An Elastic Cloud API key was strictly a control plane token. It could create projects, manage billing, and invite users, but it had a hard boundary; it couldn’t be used to call the Elasticsearch or Kibana APIs inside those projects. You always needed a second, project-specific key for data operations.
• Now: By opting into Cloud, Elasticsearch, and Kibana API access when creating an Elastic Cloud API key, the hard boundary is removed for Serverless. That API key becomes a truly unified credential. It retains its ability to manage your organization's infrastructure, while simultaneously gaining native access to query, ingest, and analyze data across any authorized Serverless project.

By unifying this under a single Elastic Cloud API key, you gain a single identity that can be scoped, audited, rotated, and revoked as one unit. Every API call, whether it provisions a new project or runs an ES|QL query, appears under the same credential in your audit logs, giving you a single trail to follow during incident investigations or compliance reviews. Credential rotation becomes a one-step operation instead of a coordinated update across separate control-plane and data-plane secrets. And because role assignments are per-project, a single key can span several projects, managing ingestion in your observability project and running queries in your security project, without juggling separate credentials for each.

Importantly, unified does not mean all-powerful. By using the role_assignments payload, you can scope a unified key strictly to a single project and a specific role (such as read-only), ensuring the blast radius remains completely contained if a credential is ever exposed. If a developer leaves or an application is decommissioned, you can revoke a single key from the Elastic Cloud Console, immediately terminating access across both the control plane and all associated Elasticsearch projects.

(Note: For Elastic Cloud Hosted/managed deployments, Cloud API keys still only manage the control plane. Support for extending this to hosted stack APIs is planned for a future release.)

Automating your workflows

Getting started is simple. You can configure this entirely through the Elastic Cloud console or automate it using the Elastic Cloud API.

The UI process remains the same, but now you can select Cloud, Elasticsearch, and Kibana API access under the project role assignment.

Here’s how you create a unified key programmatically using the Elastic Cloud API. Notice the application_roles array, as this is what grants the key native access to the Elasticsearch data plane:

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: ApiKey $EC_API_KEY" \
  "https://api.elastic-cloud.com/api/v1/users/auth/keys" \
  -d '{
    "description": "unified-automation-key",
    "expiration": "90d",
    "role_assignments": {
      "project": {
        "elasticsearch": [
          {
            "role_id": "elasticsearch-admin",
            "organization_id": "YOUR_ORG_ID",
            "all": false,
            "project_ids": ["YOUR_PROJECT_ID"],
            "application_roles": ["admin"]
          }
        ]
      }
    }
  }'

Once created, you simply pass this exact same key in the Authorization: ApiKey header to both api.elastic-cloud.com and your specific Serverless Elasticsearch endpoints.

Under the hood: Building a distributed identity layer

Making a Cloud API key work across both the control plane and the data plane isn't as simple as passing a token. It requires solving a fundamental distributed systems challenge.

Historically, Cloud API keys lived in a centralized global security cluster. That works fine for control plane operations where a higher latency is acceptable. However, Elasticsearch data requests require ultra-low latency. We cannot afford a round trip across the globe to a central control plane to validate every single search query or ingest request.

To solve this, we introduced a new authentication architecture backed by a globally distributed datastore. The following sequence diagram shows a client sending an Elasticsearch query using an Elastic Cloud API key, illustrating how authentication happens entirely within the local region, without a round trip to the global control plane. Elasticsearch delegates authentication to the Regional IAM Service, which validates the key and resolves its role assignments against a local replica of the globally distributed database. Once authorized, Elasticsearch executes the query and returns results to the client.

Globally distributed persistence

Instead of relying solely on a centralized security cluster, Elastic Cloud API keys and their associated role definitions are now persisted in a globally distributed, highly available database. This database synchronizes identity and access management (IAM) data across the global control plane and the regional data planes where your Serverless projects actually run.

Local validation with regional IAM

When your client sends a request to Elasticsearch using an Elastic Cloud API key, the request doesn't go back to the global control plane. Instead, it gets routed to the new regional IAM service. It validates the key against the local database replica, ensuring that authentication happens with near-zero latency and is completely insulated from global control plane outages.

Dynamic role mapping

Authentication is only half the battle; the system also needs to authorize the request. The regional IAM service instantly translates your Cloud-level role assignments for example, application_roles) into native Elasticsearch privileges. Elasticsearch can then authorize and execute the request locally, without ever needing a local .security index.

The foundation for Cross-Project Search

This distributed identity architecture is a foundational building block for the future of the Elastic platform.

Because identity and access are now unified and globally synchronized, we have the framework required to securely pass your identity between different projects. This enables the upcoming Cross-Project Search (CPS) capabilities for Serverless.

With CPS, you'll be able to query data spanning multiple remote Serverless projects, such as combining security and observability workloads, as easily as if they were a single dataset. By relying on unified API keys, the system can automatically evaluate your permissions across all projects simultaneously without requiring you to configure complex trust relationships, certificates, or duplicate credentials on every target project.

Learn more

Ready to simplify your stack?

• Read the Elastic Cloud API keys documentation to learn how to assign stack access.
• Check out the Create API key (Elastic Cloud API) reference to automate key generation.
• Review Elastic API keys for a full comparison of key types across the Elastic platform.

Start or continue building in Elastic Cloud today.

Disclaimer

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In order for you to be able to trust our estimates, we provide error estimates. Furthermore, since there are edge cases in error estimation, we certify when the estimated value and error are trustworthy. In this blog post, we’ll dive into the theory for approximating and estimating the error in such queries, as well as discuss the testing we’ve done.

Background

In order to estimate ES|QL STATS queries efficiently, we make use of a property that’s shared by many statistics: Their estimates computed from a large number of independent samples from a dataset approach their true value. In the case of an index with some field $X=\{x_i\},$ we can think of the true value of a statistic as its value computed for a random variable with uniform discrete distribution on $X$. In the following we denote this quantity $\theta$; it can be things like AVG, MEDIAN, and so on. If we make $n$ independent draws from $X$, denoted $S$, such that each value is selected with probability $\frac{1}{|X|}$, we have $n$ independent copies of this random variable. The property we rely on means that a sample statistic value $\hat{\theta}$ computed from $S$ approaches $\theta$ as $n$ becomes large. For example, if $\theta$ is the mean of some metric values then $\hat{\theta} = \frac{1}{|S|}\sum_{x\in S} x \rightarrow \frac{1}{|X|}\sum_{x\in X} x=\theta$ as $|S|=n$ becomes large. Indeed, for many statistics the limiting error distribution is known to be normal. Furthermore, it only depends on the distribution of $X$, the size of the sample $n$ and the type of the aggregation $\theta$. This means supported STATS queries can be approximated with fixed accuracy independent of the index size $|X|$.

It is easy to pick values at random from a Lucene index: create a filter that takes exponentially distributed jumps through the dataset, where the expected jump size is controlled by the desired sample probability. The AND of this filter and any other Lucene query can be performed extremely efficiently, since AND’ing filter queries is one of the things for which it is well optimized. In our other post, we discussed some real-world query examples to give a sense of the speedup we obtain for different levels of accuracy.

So far, we've only discussed obtaining an estimate of a query. While such a point estimator can be useful, without knowing anything about its error those uses are limited. We found that ES|QL has existing capabilities that make it relatively easy to incorporate cheap, flexible, and accurate error estimation at the same time. We'll discuss this next.

Error estimates

We view providing an accurate understanding of the uncertainty in our estimates as crucial for users to be able to trust the approximation. While having the option to quickly estimate an ES|QL query alone can be useful in certain situations, we wanted to provide a richer feature that allows clients to make intelligent choices. For example, if an approximate query is being used to preview a chart and the error is only a couple of pixels, there’s little point in running another expensive query to redraw it.

The way we've chosen to represent error is by a confidence interval: the $\alpha$-central confidence interval, to be precise. This can be expressed in terms of the cumulative density, $F$, of the statistic being estimated. Specifically, it's the interval which contains the true value of the statistic with probability $\alpha$ whose endpoints are $F^{-1}\left(\frac{1-\alpha}{2}\right)$ and $F^{-1}\left(\frac{1+\alpha}{2}\right)$. Confidence interval calculations are surprisingly subtle. There are also important constraints for our use case that make standard approaches undesirable. Next, we’ll take a look in more detail at the motivation and the design for the approach we’ve adopted.

A key requirement of the whole project is to dramatically accelerate expensive analytics queries. It’s therefore vital that the overhead of estimating uncertainty isn’t too large compared to estimating the query result itself. We also want the feature to be as general as possible, but “isolated” within the language. In other words, ES|QL is a flexible language, and we want estimation to work with as much of it as possible. At the same time, we don’t want to introduce a cross-cutting feature that incurs development costs on every new feature we ship.

With these considerations in mind, we chose to estimate confidence intervals by partitioning the sample set and computing the query output on each subsample. This is reminiscent of bootstrap; however, since we ensure that each partition receives a disjoint random subset of the sample data, we know that they comprise true estimates of the statistic distribution. To achieve the best possible estimate of the statistic itself, we still compute its value on the full sample. For example, to estimate the mean and its distribution the process can be expressed as follows:

FROM data | SAMPLE probability
          | EVAL bucketId = RANDOM(B) // B is the number of buckets
          | STATS avg     = AVG(x)
                  avg_0   = AVG(x) WHERE bucketId==0  
                  (...)
                  avg_B-1 = AVG(x) WHERE bucketId==B-1
            BY grouping
          | EVAL confidence_interval = CONFIDENCE_INTERVAL(avg, avg_0, ..., avg_B-1)

This introduces a complication to account for the discrepancy between the count of values used to estimate a query statistic and used to sample its distribution. This is a downside; however, there are some significant advantages.

Most of the work in analytic queries resides in computing the aggregate statistics: post-processing after a STATS reduction acts on a far smaller table, and the cost is often relatively small. In this scheme, every row in the input data to the STATS command is processed exactly twice compared to just estimating the statistic. Therefore, roughly speaking we pay a fixed overhead that's the same order of magnitude as the cost of estimating the query in order to estimate its uncertainty. Since we often achieve multiple orders of magnitude speedup on the exact query, this is acceptable.

Because this process uses a plain old table, with extra columns for the distribution samples, we can pass the whole table through any ES|QL pipeline and compute confidence intervals on the final results. For example, if we include EVAL square_avg = avg * avg in the pipeline above, we'd have exactly the same square_avg, square_avg_0, …, square_avg_B-1 extra values. At the end of the pipeline, we have samples from the distribution of the original statistics and all quantities that are computed using them. Therefore, we can apply our standard confidence interval machinery to reduce the table and convert samples into confidence intervals for derived quantities as well. This whole process is essentially transparent to the rest of the ES|QL language, and as we showed above, can be achieved by query rewriting.

The confidence interval calculation

We have independent samples of the statistic distribution $\{\hat{\theta}_i\}$. However, they're computed with fewer values than our estimate $\bar{\theta}$. We also have a relatively small number of distribution samples, to avoid the count discrepancy being too large, and so we don’t inflate the table too much. We therefore prefer a parametric approach for estimating confidence intervals.

The errors in the statistics for which we support estimation tend to normal distributions in the limit they're computed from many values. So a natural choice, the standard interval, is to estimate the mean and standard deviation from the samples and report the corresponding normal confidence intervals $\left[ m+\sigma \Phi^{-1}\left(\frac{1-\alpha}{2}\right), m+\sigma \Phi^{-1}\left(\frac{1+\alpha}{2}\right) \right]$. Here, $\Phi$ denotes the standard normal distribution function. For heavy-tailed data and statistical functions that are sensitive to outliers, such as STD_DEV, convergence to normality can be slow, resulting in poorly calibrated intervals.

Briefly, in order to assess the quality of the intervals, one can examine their calibration. Specifically, one computes a quantity called the coverage. For a central confidence interval, it should contain the true statistic value roughly $(1-\alpha)n$ times for $n$ trials. In fact, since we seek the central confidence interval, we can make the stronger statement that the true value should be above, or below, the confidence interval endpoints in roughly $\frac{1-\alpha}{2} n$ out $n$ trials. The empirical coverage is this fraction computed for a large number of trials. It allows us to compare alternative approaches by simulation. We return to this when we report our test results.

In order to obtain better confidence intervals, we tried a couple of different approaches: the Cornish-Fisher correction of quantiles and an adaptation of bias-corrected accelerated (BCa) confidence intervals. Simulation showed BCa provided more robust calibration across a range of confidences, so this is the approach we selected. The basic idea, which was introduced by Efron, is to assume that there exists a monotonic transformation of the underlying statistic $g=g(\theta)$ which, when applied to a distribution sample normalizes its distribution:

Here, $\hat{\phi}=g(\hat{\theta})$, $\phi=g(\theta)$ and $N(0,1)$ is the standard normal random variable. This is clearly a relaxation of the assumption that the statistic itself is normally distributed, which is used to derive the standard interval. In fact, this family includes many distributions, since $g$ is only constrained to be monotonic. (You can think of $1+a\phi=1+a g(\theta)$ as a first-order Taylor expansion of the case that the variance is an arbitrary function of the true parameter value. This further relaxes the assumption that the normalizing transformation also stabilizes the variance.) The nice thing about this ansatz is that $g$ never needs to be explicitly computed, and there exist standard approaches for estimating the parameters $a$ and $z_0$ from the distribution samples.

To handle $z_0$ one simply arranges for the estimate to land at the median of transformed distribution. If we assume the cumulative distribution function in theta space is $F_{\theta}$ then $z_0=\Phi^{-1}(F_{\theta}(\bar{\theta}))$, where $\bar{\theta}$ is the estimated statistic value, and as before $\Phi$ is the standard normal distribution function. Typically, $F_{\theta}$ is approximated by the empirical distribution function, computed indirectly by bootstrap. However, somewhat surprisingly, extensive simulation showed that we obtained better calibrated intervals using a normal approximation to our sample values, i.e. $F_{\theta}(\bar{\theta})=\Phi \left( \frac{\bar{\theta}-\hat{\theta}}{\hat{\sigma}}\right)$ with $\hat{\theta}$ and $\hat{\sigma}$ their empirical mean and standard deviation, respectively.

To complete the procedure, one can rearrange (1) to derive $\alpha_{\theta}$ quantiles for $\theta$ as follows:

where $z_{\alpha}$ is the standard normal z-score for quantile $\alpha$. Typically, one uses the inverse empirical cumulative density estimate of $F_{\theta}^{-1}$ to convert quantiles back to a confidence interval. However, because we have a mismatch between the count of values used to compute distribution samples and the query estimate, we need to do some sort of scaling. Exploring options by simulation, we again found it best to use a normal approximation, $F_{\theta}^{-1}(\alpha_{\theta})=\hat{m}+\frac{\hat{\sigma}}{\sqrt{s}}\Phi^{-1}(\alpha_{\theta})=\hat{m}+\frac{\hat{\sigma}}{\sqrt{s}}\left( z_0 + \frac{z_0+z_{\alpha}}{1 - a (z_0+z_{\alpha})}\right)$, where $s$ is the number of distribution samples we use. This is just applying the usual scaling of variance by $\frac{1}{\sqrt{\text{sample size}}}$.

Efron showed that in the case $\hat{\theta}$ is distributed as $f_{\theta}$, i.e. that it depends only on the true value $\theta$, then the acceleration $a$ can be estimated without any knowledge of $g$. In particular, $a=\frac{1}{6}\text{SKEW}\left(\frac{\partial f_{\theta}}{\partial \theta}\right)$. By assumption, our statistics tend to normal distributions with mean $\theta$. Since skew is translation and scale invariant, this gives that $a \approx \frac{1}{6}\text{SKEW}(\{\hat{\theta}\})$, i.e. one sixth of the skew of our distribution samples. One thing this glosses over is the dependence of skew, and therefore acceleration, on sample size. We know it tends to zero as the count increases. In fact, skew also asymptotes to zero as $\frac{1}{\sqrt{\text{sample size}}}$ and so we also adjust acceleration to be $\frac{1}{6\sqrt{s}}\text{SKEW}(\{\hat{\theta}\})$ to account for the count mismatch between the samples $\{\hat{\theta}\}$ and estimate $\bar{\theta}$.

Although we significantly improve the calibration of confidence intervals by using a better methodology, we still see issues in the case that the underlying distribution has very heavy tails for some of the supported STATS functions. Therefore, we introduce some additional guard rails we discuss next.

Guard rails

To avoid the user having to understand too much about edge cases, we provide additional safeguards that surface when we've been unable to confirm  that the distribution samples behave as we expect. This typically happens when the statistic isn’t computed from a sufficient number of values given the metric distribution. It's exacerbated by very skewed metric data and certain aggregation functions, such as the STD_DEV, which are sensitive to outliers.

We have some global constraints on the minimum count of values used to estimate a statistic for which we'll certify it. For example, if any bucket is empty, then we can’t rely on the distribution samples. This is because ES|QL allows mixing approximate statistics, which treat empty buckets differently. For example, consider the following query:

SET approximation=true;
FROM data | STATS avg = AVG(x), sum = SUM(y) | EVAL mix = avg + sum

There is no self-contained way of correctly assigning a value to mix for empty buckets, since summing requires that we treat them as zero, in which case we bias our estimate of avg. Alternatively, ignoring empty buckets introduces bias in the sum. There is also a global minimum count of values for which we’ve verified our certification method is sufficiently reliable; this is 10.

We explored a variety of additional tests to certify the results. These were based on both tests of the underlying data distribution, specifically Hill’s estimator, as well as the statistic’s distribution properties. If the true distribution of the statistic is sufficiently normal, then our estimate and confidence interval calculation behaves as we expect: The interval is well calibrated and the interval width is representative of the actual error. Therefore, in the end, we chose to use a test based on the p-value for distribution samples’ skewness and kurtosis versus a normal distribution null hypothesis. To certify a result, we require that the two tail p-values are greater than 0.05 for both tests. As we show below, we found this test was well aligned to our actual needs: to distinguish results for which the estimate and its confidence interval are more and less reliable.

There's a simple trick we can use to boost the accuracy of the accuracy of the test: Create multiple independent distribution samples and use a vote. Given a test to certify results with a failure rate $f$, the distribution of the count of $k$ failures for $t$ tests is $\frac{t!}{k!(t-k)!}f^k(1-f)^{t-k}$ for the case the null hypothesis, that the estimate is trustworthy, is true. For example, for the majority vote assuming $f=0.05$ and $t=3,$ then the significance of the test is $1-(3\times 0.95^2\times 0.05+0.95^3)=0.007$, i.e. we fail to certify fewer than 1% of trustworthy results. Note that we can compute multiple trials relatively easily using different seeds for the RANDOM bucket identifier.

This additional check allows us to certify that we trust our estimates and their errors. We surface this information in the approximate query results. When we can’t certify results, they won’t necessarily be inaccurate, but they should be treated with more caution.

Testing

The two main aims of the testing we discuss here were to understand the calibration of the confidence intervals and to see how well they characterize the statistics' estimation errors. The count function is particularly well behaved, its error distribution is binomial, so the majority of our testing focused on metric aggregations. We study smooth distributions but make sure we cover a range of tail behaviors. The presence of outliers is the key factor that reduces the accuracy of estimated statistics. For example, if an outlier isn’t sampled at all, it can significantly affect the value of some statistics.

We explored a range of light-tailed distributions, such as uniform and normal, and skewed and heavy-tailed distributions, such as exponential, log-normal, Cauchy, and Pareto. For each family of distribution, we used multiple parameterizations, focusing primarily on varying the scale parameter. In total, we had 24 distinct data distributions. Figure 1 shows some example sample distributions from this set. Note that we’ve truncated the charts to remove extreme outliers, which are present for both the Cauchy and log-normal distributions.

For each data distribution, we evaluated 14 different sample sizes, ranging from 1000 to 500000. Then, for each sample set, we evaluated AVG, COUNT, MEDIAN_ABSOLUTE_DEVIATION, MEDIAN, PERCENTILE([25, 75, 90, 95, 99]), SUM and STD_DEV at two levels of confidence, 50% and 90%. In total, we have around 7500 distinct experiments. For each experiment, we assessed the interval calibration using 100 runs and counting the number of times the true statistic lands in the confidence interval. This gives us a binomially distributed estimate for the true confidence interval coverage. The variation we expect in the estimated coverage changes slightly with the level of confidence; for example, at 50% we expect to see values mainly between 0.44 and 0.56, and for 90% we expect to see values mainly between 0.86 and 0.94 using 100 trials.

Figure 2 shows box plots for the empirical coverage for the two confidence levels computed from all experiments. In all cases, the confidence intervals are reasonably well calibrated. Extreme percentiles are biased for small sample sizes, which leads to increased outlier counts for small sample sizes. As a rule of thumb, you’d want roughly $\frac{10}{p(1-p)}$ samples to ensure that you have enough samples in the appropriate tail.

Next, we examine the degree to which the confidence intervals capture the typical size of the estimate error. To do this, we examine the distribution of the ratio of the estimated statistics' error and half the confidence interval width for all certified results. The higher the confidence, the wider the interval, so different confidence levels shift the mean of this distribution. Figure 3 shows this distribution computed for the 90% confidence interval. As expected, the distribution is roughly normal, albeit with a tail of some larger errors. We see in all cases the confidence interval width gives the order of magnitude of the estimated statistics' actual errors.

We’ve shown that certified results are nearly always reliable; however, we’d also like some insight into the proportion of results which we fail to certify that are actually reliable, to confirm that the test aligns with our objective. We use reliable here in the fairly strong sense that the confidence interval is well calibrated. Specifically, for the 50% and 90% confidence intervals, we count the proportion of uncertified results for which the confidence interval empirical calibration has an acceptable margin of error, given the number of trials used to estimate it. Using this procedure, the false positive rate across all experiments is around 1%. This agrees well with the failure rate we expect by chance, given our test parameters, and confirms the assumption underlying the test.

Finally, to better understand the difference between certified and uncertified results, Figure 4 shows the error distribution of the ratio of the estimated statistics' errors and half the 90% confidence interval for the reliable and unreliable results separately. Note that we truncated the range for uncertified intervals.

Wrapping up

In this post, we present the background behind our approach for quickly estimating ES|QL queries and providing an indication of their errors. To do this, we developed an effective confidence interval mechanism that allows us to provide error estimates. Our approach also allows us to estimate confidence intervals for quantities derived from sampled statistics via other pipeline operations. Quantifying the error comes with a relatively small overhead compared to just estimating the query. Finally, we developed a statistical test to certify results we return. Values that aren’t certified can still be accurate, but we’re less confident in them.

As well as testing the feature on a range of real-world use cases, which we discuss in our companion post, we tested the error estimation by extensive simulation across a range of data characteristics, sample sizes, aggregation functions, and confidence levels. This showed confidence intervals are well calibrated, and the interval itself provides a good approximation of the actual error we observe in the estimates. Finally, we showed that we were able to certify intervals with a low false negative rate.

We’re planning to integrate this feature into other stack capabilities in the future, so stay tuned.

One of the key requirements for a pleasant user experience is that these operations are performed quickly. Large language model–based (LLM) agents also introduce new higher bandwidth and speculative query patterns that can potentially benefit from different optimization strategies.

In this two-part blog series, we discuss an optimization approach we’re introducing to ES|QL in version 9.4 of Elasticsearch and the Elastic Stack, which exploits a relaxation of the problem. Rather than trying to get exact values for aggregates, we allow ourselves to return approximate values, together with some characterization of their error. A key benefit of approximation is that it breaks the dependency between performance and dataset size: The accuracy with which one can approximate a query doesn’t depend on the original dataset size but, principally, its data characteristics and the query itself. As we’ll see later, this allows us to achieve some dramatic performance improvements.

In our next blog post, we will discuss the theory behind our approach and the validation we’ve done of its statistical properties. Here, we introduce the syntax and give a sense of how it’s achieved using standard ES|QL and query rewriting. You can explore its performance on a subset of the popular ClickBench benchmark. Finally, we discuss some limitations and gotchas that are worth understanding when you use query approximation.

Syntax and behavior

So how do you actually use it?

SET approximation=true;
// The query you want to approximate
FROM data | commands | STATS x=agg(...) | commands

That’s it. You simply introduce the new line SET approximation=true; and write your STATS query pipeline as usual. Below, we discuss some advanced configuration options and some limitations around the agg(...) and commands. However, essentially, we choose defaults so that this will typically provide useful approximations while achieving significant speedups.

With this change, you’ll see some differences in the query results. Let’s look at a concrete example to illustrate this. Suppose the raw query is as follows:

FROM sales | WHERE @timestamp >= NOW()-1w
           | STATS count = COUNT() BY item_category
           | SORT count DESC
           | LIMIT 5

The results might look something like this:

item_category        | count
---------------------+------
Household Essentials | 5165
Kitchen              | 2132
Storage              | 1121
Home Decor           | 877
Furniture            | 357

Approximating this query introduces some extra columns for each quantity that’s estimated:

item_category | count | _approximation_confidence_interval(count) | _approximation_certified(count)
--------------+-------+-------------------------------------------+--------------------------------
Essentials    | 5150  | [5100, 5250]                              | true
Kitchen       | 2150  | [2100, 2200]                              | true
Storage       | 1120  | [1100, 1150]                              | true
Home Decor    | 880   | [860, 900]                                | true
Furniture     | 330   | [310, 350]                                | true

The count column now contains an estimate, and you’ll see it’s somewhat different from the exact values above. The _approximation_confidence_interval(count) column defaults to the central 90% confidence interval for the count estimate and the _approximation_certified(count) column indicates if we’re highly confident that the results and their confidence interval are trustworthy. In outline, the confidence interval is an interval we expect has a high probability (0.9) of containing the true value for the quantity being estimated. The certified column indicates the distribution of the approximation is behaving as we expect. When the result isn’t certified, it’s often still accurate, but our test of the properties of its distribution hasn’t been able to confirm this. These quantities are discussed in more detail in our second post.

Implementation

An approximate query is rewritten before query execution using random sampling and extrapolation. Let’s take a look at the query of the previous section. The part of the rewritten query responsible for obtaining the best estimate looks like:

FROM sales | SAMPLE probability
           | WHERE @timestamp >= NOW()-1w
           | STATS count = TO_LONG(COUNT() / probability) BY item_category
           | SORT count DESC
           | LIMIT 5

The query samples a fraction of the data, and therefore the final count has to be extrapolated by scaling up with the inverse of the sample probability. Extrapolation clearly depends on the underlying aggregation function, and we handle this appropriately for all functions we support.

To obtain the sample probability, we're setting a fixed number_of_rows to be processed by the STATS command. In this case, the probability is calculated as follows:

FROM sales | WHERE @timestamp >= NOW()-1w
           | STATS total_row_count = COUNT()
           | EVAL probability = number_of_rows / total_row_count

This query is executed before the final approximate query is executed.

As well as this best estimate, confidence intervals and a statistical test used to certify that the value distribution is behaving as we expect also need to be computed. The intervals are computed using a variant of the bias-corrected and accelerated bootstrap confidence interval (BCa) method. Therefore, the data needs to be partitioned into B buckets, which are used in turn to compute the intervals. Omitting some implementation details, this approximate query looks like:

FROM sales | SAMPLE p
           | WHERE @timestamp >= NOW()-1w
           | EVAL bucketId = RANDOM(B) // B is the number of buckets
           | STATS count     = TO_LONG(COUNT() / p) 
                   count_0   = TO_LONG(COUNT() / p) WHERE bucketId==0  
                   (...)
                   count_B-1 = TO_LONG(COUNT() / p) WHERE bucketId==B-1  
             BY item_category
           | WHERE count >= 10
           | SORT count DESC
           | LIMIT 5
           | EVAL ci = CONFIDENCE_INTERVAL(count, count_0, ..., count_B-1),
                  certified = CERTIFIED(count, count_0, ..., count_B-1)
           | DROP bucketId, count_0, ..., count_B-1

To certify the estimate and confidence interval, there should be enough data, and the distribution of the bucket values should tend to normality.

Some queries can be efficiently computed using only summary statistics maintained in the index. To handle these correctly, where sampling is both slower and inaccurate, we updated the physical query planner, since detecting this case requires information that’s only available where the data resides. When the planner detects this is possible, it simply executes the query as normal. Such queries are typically fast anyway, and there’s no real side effect, so you don’t need to worry about this when using approximation; however, you’ll see that confidence intervals for such queries always have zero length, indicating the results are exact.

Results

To explore the performance improvements, we use ClickBench. This is a benchmark for analytics workloads for database management systems (DBMS). It comprises approximately 100 million rows, with a focus on clickstream and traffic analysis, web analytics, machine-generated data, structured logs, and events data. The benchmark also defines 43 queries that are typical of ad-hoc analytics and real-time dashboards.

Some of the queries aren’t suitable for approximation. For example, we don’t support approximating the unique count of a categorical value or computing the minimum and maximum of a metric value. We also don’t care about queries targeting search alone, for which Elasticsearch has excellent performance in any case. We therefore exclude these types of query from our evaluation. Finally, we also want to test a few additional aggregation functions, such as percentiles, which are not well represented in the original query set, so add some variants of the original metric queries to this end.

Queries in the benchmark are written using standard SQL and so need porting to use ES|QL syntax. This translation is fairly straightforward. Here’s an example:

SELECT SUM(AdvEngineID), COUNT(*), AVG(ResolutionWidth) FROM hits

becomes:

FROM hits | STATS s = SUM(AdvEngineID),
                  c = COUNT(*),
                  a = AVG(ResolutionWidth)

when rewritten in ES|QL.

For running all benchmarks, we use an Elastic Cloud Hosted instance with 870GB disk, 29GB Ram, and 4 vCPUs, in effect, an Amazon Elastic Compute Cloud (EC2) i3.xlarge instance. In the following results, we simply compare ES|QL with and without query approximation. Extensive results on a range of different hardware setups and datastores can be found here. Even with significantly constrained test hardware (matching the vCPUs of the smallest setup), our approximation approach achieves competitive results against much larger systems.

We run each query and its approximation five times in a random order, clearing the query cache between each run. We report the average run time over all five runs. While clearing the cache should be sufficient to avoid most of the advantage of running second, we wanted to avoid any possible accidental prewarming effects, which is why we alternate.

The results break down into four categories:

1. Queries which are rewritten to use index summary statistics (three queries).
2. Queries that perform well (13 queries).
3. Queries with high cardinality partitioning (seven queries).
4. Queries with restrictive filters (12 queries).

Roughly speaking, for these four categories, approximate querying is: equivalent (1); faster and accurate (2); faster but unreliable (3); and slightly slower (4), compared to exact querying, respectively.

For category 1, the planner automatically detects that we’re able to perform the query using summary statistics, and we end up executing the queries in the same way. To do this, we need information that’s only available on the data nodes, so we perform the rewrite only after we've estimated the sample probability. Because we're able to do this very efficiently, the overhead is small (around 10–15%). In both cases, the results are exact.

Queries in category 2 run on average 23$\times$ faster if estimating the values and computing confidence intervals and 72$\times$ faster if just estimating the values, which you can select as follows: SET approximation={"confidence_level":null}. These headline figures hide quite some variation in the impact of approximation on performance. The table below shows some queries sampled from the range of speedups we see:

Here are the corresponding queries:

// Query 3
FROM hits | STATS s = SUM(AdvEngineID),
                  c = COUNT(*),
                  a = AVG(ResolutionWidth)

// Query 10
FROM hits | STATS s = SUM(AdvEngineID),
                  c = COUNT(*),
                  a = AVG(ResolutionWidth) BY RegionID
          | SORT c DESC
          | LIMIT 10

// Query 13
FROM hits | WHERE SearchPhrase != ""
          | STATS c = COUNT(*) BY SearchPhrase
          | SORT c DESC
          | LIMIT 10

// Query 21
FROM hits | WHERE URL != ""
          | STATS l = AVG(LENGTH(URL)), c = COUNT(*) BY CounterID
          | WHERE c > 100000
          | SORT l DESC
          | LIMIT 25

// Query 22
FROM hits | WHERE Referer != ""
          | GROK Referer """%{URIPROTO}://(?:www\.)?%{URIHOST:k}"""
          | WHERE k IS NOT NULL
          | STATS l = AVG(LENGTH(Referer)), c = COUNT(*) BY k
          | WHERE c > 100000
          | SORT l DESC
          | LIMIT 25

We'll return to the accuracy of the approximation in the next blog post, but to give a sense of this, we plot below the exact and approximate values for a sample run for query 13:

For category 3, we get an average speedup of $11\times$. However, the results of queries in this category can miss some partitions and often have large estimation errors. Approximation can still be valuable for such queries, particularly in the context of agentic workflows, but requires larger sample sizes than out default if accuracy is important. As we discuss in the next section, we provide an API to explicitly control the sample size. If the source dataset is sufficiently large, this can be increased and approximation will still yield significant performance improvements. The table below shows a couple of query examples for this category:

Here are the corresponding queries:

// Query 15
FROM hits | STATS c = COUNT(*) BY UserID, SearchPhrase
          | SORT c DESC
          | LIMIT 10

// Query 17
FROM hits | EVAL m = DATE_EXTRACT("minute_of_hour", EventTime)
          | STATS c = COUNT(*) BY UserID, m, SearchPhrase 
          | SORT c DESC
          | LIMIT 10

Finally, category 4 queries use selective filters and end up being executed exactly, but they run slightly slower because of the work done in the query rewrite stage. Typically, all these queries run fast anyway, so the absolute slowdown is small. On average, they run approximately 14% or 370ms slower than the “without” sampling for our test setup.

Limitations and best practices

It’s worth explicitly mentioning some limitations. In particular, the following queries are not currently supported:

1. Queries using the TS source command.
2. Queries using the FORK or JOIN processing command.
3. Pipelines which use two or more STATS commands.
4. The ABSENT, PRESENT, DISTINCT_COUNT, MIN, MAX, TOP, ST_CENTROID_AGG and ST_EXTENT_AGG aggregation functions.

We plan to lift some of these restrictions in future releases, such as approximating queries using TS, FORK and JOIN; however, some are intrinsic. For example, while there’s prior art for estimating the minimum and maximum of a metric dataset or the count of unique values of a categorical dataset (see, for example, this paper), they require making certain distributional assumptions, either explicitly or implicitly. In summary, we view trying to automatically provide estimates of these statistics as being too open to accidental misuse.

For the expert user, we provide another route: ES|QL supports using the SAMPLE command directly. This allows one to obtain “point estimates” of any query, albeit with no attempt to correct for the impact of sampling or quantify error. For example:

FROM data | SAMPLE 0.01 | STATS DISTINCT_COUNT(value)

computes the unique count of the value field on a sample of roughly 1/100th of the dataset. The sample probability can be adjusted to get a sense of how this is asymptoting, or more sophisticated estimation procedures can use STATS COUNT() BY value to estimate the frequency profile of the data.

There are a couple of cases that are more problematic for sampling. If a very restrictive filter is applied in the query, then sampling is of little value, since few rows match anyway. In such cases, we discover that we’d have to sample too large a proportion of the rows to estimate the query in the rewrite phase. In this case, we revert to running the query without sampling and its result is exact. However, the search procedure to determine the fraction of rows to sample comes with some overhead. One therefore pays a penalty, albeit less than the original query cost, for no benefit. If you know in advance that the query is expected to match relatively few rows, it's best to run it without approximation.

The second case only applies when computing STATS partitioned by some expression. If the cardinality of this expression is very high, then even if many rows are searched, individual statistics may be computed from a small number of rows. Some cases are more problematic than others. Sorting by ascending count, that is, finding the rarest partitions, can be impossible to estimate in a single query if heavy hitters would require us to sample most of the dataset to find them. For this particular case, heavy hitting partitions can be estimated first and sometimes efficiently excluded by updating the query. In general, infrequent partitions may be lost in the sampling process, and their statistics' estimation errors can be high. It’s worth noting that we won’t attempt to estimate any statistic for which we have fewer than 10 samples, and we simply drop them from the result set. In the case of very high cardinality BY clause, for example, a field whose value is unique for every row, this means the query can return no results. If you find approximate query results are too inaccurate, you have the option to increase the sample size, which by default is 1,000,000 for STATS, which uses grouping and 100,000 otherwise. Currently, this needs to be done manually, and we provide the following API for this:

SET approximation={"rows":12345678};
FROM data | commands | STATS x=agg(...) | commands

Occasionally, functions significantly alter the distribution characteristics of the quantities they act on. A contrived example is the following:

FROM data | STATS sl = SUM(length) | EVAL csl = COS(sl)

If the variation in the estimate sl is much larger than $2\pi,$ we expect the distribution of csl to be mainly flat in the interval $[-1,1]$ with peaks near both endpoints. In this particular case, it’s not clear that the central confidence interval is a particularly useful concept, since the modes of the distribution lie outside almost all central confidence intervals. In any case, just observing the samples of csl, our standard confidence interval machinery won’t reliably characterize this distribution and it will underestimate the variability of csl. However, our statistical test should detect this problem, and the result won’t be certified.

Finally, we note that Elasticsearch implements some query optimization strategies that ideally need to account for the fact that sampling is taking place. These rewrite the query at the Lucene level and the preprocessing involved in this rewrite can be relatively expensive. Accelerating an expensive string matching operation by first building a suitable data structure makes sense if the query needs to process every row, but if it processes only a small fraction of them, the trade-off is different. This is something we plan to enhance in future.

Conclusions

In this blog post, we introduced a new form of query optimization we’re bringing to ES|QL that enables dramatically faster querying by relaxing the constraint that the results are exact. We found on ClickBench that we were able to accurately estimate query values and their confidence intervals up to 100 times faster and values alone up to 250 times faster than we can compute them exactly. Furthermore, we expect this advantage to grow as the dataset size increases, because the approximation accuracy is independent of the dataset size. This feature works with many features of the ES|QL language and is enabled by simply prepending SET approximation=true; to the query to estimate.

As well as providing a point estimate, we also estimate confidence intervals and indicate whether we think that the underlying assumptions used to compute these are satisfied. This allows us to certify the results if the results are reliable. We explain the theory behind this feature and discuss the testing of its accuracy in our next post.

The datafeed isn’t broken. It’s doing exactly what it was built to do: reading every raw document, across every shard, every run. On a large cluster with cross-cluster search (CCS) and a broad index pattern, like logs-*, that means scanning billions of documents per bucket. There’s no hardware that makes that sustainable. The datafeed will always be chasing live data and never reaching it.

The fix is to switch from the default scroll-based datafeed configuration to an aggregation-based datafeed configuration: Let the data nodes summarize locally, and ship only compact bucket results to the ML node. Same detections, a fraction of the load. The speedup can be dramatic. More than you might expect. The numbers are in the next section. The explanation for why the gap is so large is at the end of the post, for those who want to understand the mechanics.

One catch worth knowing now: Switching requires creating a new job. The old model doesn’t transfer; weeks of learned baseline are lost. The right time to make this switch is before the job has been running for months, not after. That’s the main reason to read this before you deploy.

How much faster? Scroll vs. aggregation datafeeds for ML jobs

We ran the same job two ways on production data: first scroll-based, and then aggregation-based. The job covered 13 months of history, monitoring 836,000 log events per hour in 15-minute buckets across multiple clusters.

Training on historical data with scroll-based configuration: five days of wall-clock time, 7.9 million sequential requests, and 3.5 TB transferred; with aggregations: 2.3 minutes, 23 requests, and 34 MB (a 3,374× speedup). Think of it this way: If you start the scroll backfill at 9 a.m. Monday, it will finish Saturday morning. The aggregation version is done by 9:02 a.m.

On live data, the difference is less dramatic but still meaningful: around 20× fewer requests per tick. That adds up quickly when the datafeed runs every few minutes around the clock.

Before you start

Three things worth knowing before diving into the configuration.

This isn't wizard territory. The standard Kibana job wizards (Single Metric, Multi-Metric, Population) don't expose aggregation configuration. To create an aggregation-based job, you need either the Elasticsearch API or Kibana's Advanced Job Wizard, with JSON edited by hand. The worked example below shows the most practical path: Configure the job in the Multi-Metric Wizard, and then click Convert to advanced job before creating it. That gets you a prefilled JSON starting point instead of a blank editor.

The configuration is unforgiving and mostly silent about it. There's no schema validation that catches a misnamed aggregation key or a fixed_interval that doesn't match bucket_span. The job will run, anomalies will fire, and nothing will indicate that the results are based on the wrong data. This is why the five-step pattern exists and why the Preview tab is worth using every time: Catching a misconfiguration before the job trains is a 30-second check; catching it a week later is a much worse afternoon.

The Single Metric Viewer has a known limitation with aggregated jobs. That viewer reconstructs the "actual" data curve by re-querying the index, but it can't reproduce an arbitrary, user-defined aggregation, so the actual-value line is typically missing or approximate. The Anomaly Explorer is unaffected: Anomaly scores, swim lanes, and influencer attribution all work normally. Just don't rely on the Single Metric Viewer's chart for visual validation of what the model saw.

What we can and can’t aggregate

Almost every ML function works with aggregated datafeeds, but the right aggregation pattern depends on the function.

lat_long is the one genuine exception. The configuration is accepted, but geo_centroid computes the arithmetic mean of all coordinates in a bucket: If the same entity appears in New York and London within the same bucket, the centroid ends up in the Atlantic Ocean, which probably doesn't make sense for the use case. Keep lat_long jobs on scroll-based datafeeds.

The five-step pattern in the next section covers the standard case. We’ll walk through the remaining patterns at the end of the post.

The standard five-step pattern: Scroll-based to aggregation datafeed

Converting any scroll-based job to an aggregation-based datafeed follows the same five steps. Once you understand the pattern, applying it to any compatible job takes about 10 minutes.

Step 1: Add summary_count_field_name: "doc_count" to the analysis config. This tells the ML engine that incoming data is pre-summarized. Without it, the engine treats each aggregated bucket as a single raw document and produces wrong anomaly scores.

Step 2: Choose the bucket wrapper topology. For most functions (count, mean, sum, max, min, varp, time_of_day, time_of_week, and categorization) use a date_histogram at the top level whose fixed_interval matches your bucket_span exactly to ensure accurate analysis. For rare, freq_rare, and info_content, use a composite at the top level with a date_histogram as one of its sources. This routes the datafeed to the composite extractor, which paginates through all field-value combinations rather than truncating to a top-N.

Step 3: Add a max aggregation on @timestamp. The ML engine needs this to determine the precise end time of each bucket. In the standard topology (Step 2, date_histogram outer), it goes inside the histogram’s aggregations. In the composite topology, it sits as a sibling of the composite aggregation.

Step 4: Map each analysis field to a terms aggregation, named exactly after the corresponding field in the analysis config. One categorical field → a single nested terms. Two or more categorical fields → a composite aggregation nested inside the date_histogram, with one terms source per field. For categorization jobs, use a terms aggregation on the .keyword subfield of the categorization_field_name. The naming rule is strict: The aggregation key must exactly match the field name in the analysis config; the ML engine uses the aggregation name, not the field parameter, to look up values. A mismatch produces silently wrong results; no error, just a job that appears to run while missing everything meaningful.

Step 5: Map each detector’s metric field to its Elasticsearch aggregation equivalent:

For count, rare, freq_rare, info_content, time_of_day, time_of_week, and categorization jobs, the ML engine works from doc_count alone; no metric aggregation is needed, and this step can be skipped.

Step-by-step example: Building an aggregation-based ML job in Kibana

Let’s build this end to end using Kibana’s sample web logs. If you haven’t loaded them yet, go to the Kibana home page and click Integrations → Sample data → Sample web logs → Add data. This gives us a data view called Kibana Sample Data Logs and an index called kibana_sample_data_logs with fields including @timestamp, bytes (response size), and geo.dest (destination country).

We’ll build a job that detects unusually large response sizes: high_mean of bytes, partitioned by destination country (geo.dest), with a 1-hour bucket span.

Creating the job with the Multi-Metric Wizard

This is how most jobs get created in practice. Navigate to Machine Learning → Anomaly Detection → Manage Jobs → Create job.

Select the “Kibana Sample Data Logs” data view, and set the time range to cover the full sample dataset. On the job type screen, choose Multi-metric.

In the Multi-Metric Wizard, configure the detector:

• High mean of bytes.
• Split data by geo.dest.
• Bucket span: 1h.

Give the job an ID, and leave everything else at its defaults, but don’t click Create yet. On this last configuration step, click on Preview JSON and look at the datafeed section. What you’ll see is a plain scroll-based datafeed with no aggregations, just an index pattern and a match_all query.

This is the default every wizard produces. On a small cluster, it works fine. On a large cluster with CCS and a broad index pattern, this datafeed will scan every raw document on every run and never catch up with live data.

Instead of clicking Create, click Convert to advanced job. This keeps everything you just configured (the detector, the partition field, the bucket span) and drops you directly into the Advanced Wizard, where we can apply the five-step pattern.

Analysis configuration

The conversion prefills the detector, partition field, and bucket span. The only change needed here is Step 1 of the pattern: Open the Edit JSON view, and add summary_count_field_name to tell the ML engine that incoming data will be pre-summarized:

{
  "bucket_span": "1h",
  "summary_count_field_name": "doc_count", // Step 1
  "detectors": [
    {
      "function": "high_mean",
      "field_name": "bytes",
      "partition_field_name": "geo.dest"
    }
  ],
  "influencers": ["geo.dest"]
}

Datafeed configuration

Switch to the Datafeed tab. This is where Steps 2 through 5 of the pattern come together. Remove scroll_size if it’s present, and then enter the aggregations:

{
  "buckets": {
    "date_histogram": {               // Step 2: bucket wrapper, interval = bucket_span
      "field": "@timestamp",
      "fixed_interval": "1h"
    },
    "aggregations": {
      "@timestamp": {                 // Step 3: max timestamp anchor
        "max": { "field": "@timestamp" }
      },
      "geo.dest": {                   // Step 4: partition field, name must match exactly
        "terms": {
          "field": "geo.dest",
          "size": 1000
        },
        "aggregations": {
          "bytes": {                  // Step 5: metric field → avg aggregation
            "avg": { "field": "bytes" }
          }
        }
      }
    }
  }
}

A few notes on this config:

• Step 2: The date_histogram uses fixed_interval: "1h", matching bucket_span exactly. A mismatch produces incorrect bucket timing.
• Step 3: The max aggregation on @timestamp must be named @timestamp and placed inside the histogram’s aggregations; without it, the ML node can’t determine the precise end of each bucket.
• Step 4: The terms aggregation for the partition field must be named exactly after the partition field: geo.dest, not geo.dest_grouping or any alias. The ML engine uses the aggregation name, not the field parameter, to identify which partition value each bucket belongs to. A mismatch silently drops the partition field from results entirely.
• Step 5: The metric aggregation key bytes matches field_name in the detector exactly. Any mismatch here produces silently wrong anomaly scores.

Validate with the preview

Before we create the job, let’s use the Preview tab. This runs the aggregation against real data and shows exactly what the ML node will receive, a very useful sanity check before committing.

Three things to verify in the preview output: doc_count should be present on every bucket and greater than 1. The bytes values should look like average response sizes: numbers in the hundreds to hundreds of thousands for web traffic. And each row should correspond to a distinct (timestamp, geo.dest) pair. If anything looks off, fix it in the JSON editor and rerun the preview.

Adding influencer fields

In the example above, geo.dest is the partition field. The ML model learns a separate baseline for each destination country, and anomalies are reported per country. But you might also want machine.os to appear as an influencer in anomaly results: When the detector fires, you want to see “this looks anomalous for geo.dest: CN and machine.os: win is a contributing factor.” Influencers don’t drive anomaly detection; they provide context for the anomalies that are found.

To support an influencer alongside a partition field, the analysis config gains an influencers array:

{
  "bucket_span": "1h",
  "summary_count_field_name": "doc_count",
  "detectors": [
    {
      "function": "high_mean",
      "field_name": "bytes",
      "partition_field_name": "geo.dest"
    }
  ],
  "influencers": ["geo.dest", "machine.os"]
}

And now the datafeed needs to aggregate on both fields simultaneously. One terms nested inside another terms won’t work; a nested terms surfaces only the top-N values of the inner field per outer bucket, so you’d silently lose combinations. Instead, use a composite aggregation with one terms source per field, nested inside the date_histogram:

{
  "buckets": {
    "date_histogram": {
      "field": "@timestamp",
      "fixed_interval": "1h"
    },
    "aggregations": {
      "@timestamp": {
        "max": { "field": "@timestamp" }
      },
      "group_by_fields": {
        "composite": {
          "size": 1000,
          "sources": [
            { "geo.dest":   { "terms": { "field": "geo.dest" } } },
            { "machine.os": { "terms": { "field": "machine.os" } } }
          ]
        },
        "aggregations": {
          "bytes": {
            "avg": { "field": "bytes" }
          }
        }
      }
    }
  }
}

composite generates one bucket per unique (geo.dest, machine.os) combination. The ML node sees every pair and can correctly attribute which operating system was contributing when a country’s response sizes spiked. Use the preview to confirm distinct pairs appear. If you only see a handful of rows where you’d expect many, the size parameter on the composite may need to be raised.

Note that this composite is nested inside the date_histogram, a different structure from the top-level composite used for rare, freq_rare, and info_content below. The distinction matters: Composite nested inside date_histogram routes the datafeed to the standard extractor; composite at the top level routes it to the composite extractor, which paginates through all value combinations across time.

Categorization

Categorization works with aggregated datafeeds: summary_count_field_name and categorization_field_name can coexist in the same job. The five-step pattern applies directly. Step 2 uses the standard date_histogram topology. Step 4 has one adjustment: Instead of a partition field, we aggregate the text field itself using a terms aggregation on its .keyword subfield, named to match categorization_field_name exactly. Step 5 is skipped. The count detector works from doc_count alone.
Analysis config:

{
  "bucket_span": "1h",
  "summary_count_field_name": "doc_count",
  "categorization_field_name": "message",
  "detectors": [
    {
      "function": "count",
      "by_field_name": "mlcategory"
    }
  ],
  "influencers": ["mlcategory"]
}

Datafeed aggregations:

{
  "buckets": {
    "date_histogram": {
      "field": "@timestamp",
      "fixed_interval": "1h"
    },
    "aggregations": {
      "@timestamp": {
        "max": { "field": "@timestamp" }
      },
      "message": {
        "terms": {
          "field": "message.keyword",
          "size": 1000
        }
      }
    }
  }
}

The datafeed sends one bucket per unique message.keyword value with a doc_count for each. The ML node receives those strings, runs categorization on them, assigning an mlcategory to each, and the count detector tracks how many documents fall into each category per bucket. The naming rule from Step 4 applies: The terms aggregation must be named message, matching categorization_field_name in the analysis config exactly.

One thing to watch: Keyword fields have a default ignore_above: 256 limit. Log messages longer than 256 characters won’t be indexed as .keyword and will be silently excluded from the aggregation. If your log messages are long, check the field mapping before using this approach. You may need to raise the limit in your index template.

The minimal pattern for time_of_day and time_of_week

time_of_day and time_of_week are the easiest functions to aggregate: They only need a timestamp and a document count. The C++ process extracts the time component from the bucket timestamp and builds a cyclical model of normal activity; doc_count tells it how many events fell in each bucket. No terms sources, no metric aggregation, no composite.
Analysis config:

{
  "bucket_span": "15m",
  "summary_count_field_name": "doc_count",
  "detectors": [
    { "function": "time_of_day" }
  ]
}

Datafeed aggregations:

{
  "time": {
    "date_histogram": {
      "field": "@timestamp",
      "fixed_interval": "15m"
    },
    "aggregations": {
      "@timestamp": { "max": { "field": "@timestamp" } }
    }
  }
}

A plain date_histogram is enough; no composite needed. This makes time_of_day and time_of_week particularly CCS-friendly: one request per time chunk, minimal data over the wire. Use the same structure for time_of_week; only the function name changes.

If you want to add a partition_field_name (for example, to model time-of-day patterns per service), add a terms aggregation inside the histogram’s aggregations following the standard Step 4 pattern.

The composite pattern for rare, freq_rare, and info_content

rare, freq_rare, and info_content all need the composite extractor, the one that paginates through all unique value combinations rather than truncating to top-N. The five-step pattern applies here with a different topology in Step 2: composite goes at the top level (not date_histogram), with date_histogram as a source inside it. Step 3 places the max @timestamp aggregation as a sibling of the composite, and Step 5 is skipped since all three functions work from doc_count alone.

The datafeed structure is the same for all three functions: a composite at the top level, a date_histogram as one of its sources, and one terms source per analysis field. The only thing that varies is which fields you include as terms sources: rare needs one source for by_field_name; freq_rare needs sources for both by_field_name and over_field_name; info_content needs a source for field_name plus any by_field_name or over_field_name fields. None of the three require a metric aggregation.

{
  "buckets": {
    "composite": {
      "size": 10000,
      "sources": [
        { "@timestamp":   { "date_histogram": { "field": "@timestamp", "fixed_interval": "5m" } } },
        { "by_field":     { "terms": { "field": "by_field" } } },
        { "over_field":   { "terms": { "field": "over_field" } } }
      ]
    },
    "aggregations": {
      "@timestamp": { "max": { "field": "@timestamp" } }
    }
  }
}

A few notes:

• The composite aggregation must be the top-level aggregation, not nested inside a date_histogram. This is what routes the datafeed to the composite extractor.
• The date_histogram is a source inside the composite, not the outer wrapper. Its fixed_interval must divide evenly into bucket_span.
• The max aggregation on @timestamp sits as a sibling of the composite (inside aggregations), not nested inside it.
• composite.size controls the page size per round trip. Setting it high (10000) reduces round trips, which matters with CCS latency. With three sources and high-cardinality fields, the total combination count can be large; the extractor paginates automatically.

Why aggregation-based datafeeds outperform scroll at scale

The gap is structural, not incidental. A scroll-based datafeed reads raw documents one page at a time: Every 1,000 documents is one request, and each waits for the previous one to complete before issuing the next. The number of requests is therefore proportional to the total document count in the time range being backfilled. At 836,000 events per hour over 13 months, that's roughly 7.9 billion events, or 7.9 million sequential round trips. Each round trip crosses the CCS boundary, waits for shard responses, and transfers matching documents in full. There’s no parallelism: The datafeed holds a scroll context open on the remote cluster and processes one page at a time.

An aggregation-based datafeed works differently. The data nodes summarize data locally, grouping by time bucket and categorical fields, and ship only the bucket results to the ML node. The number of requests is proportional to field cardinalities, not document count. In our example, two influencer fields with six unique combinations produce six result rows per time bucket; the datafeed pages through those in a handful of requests regardless of how many raw events fall in each bucket. Double the ingestion rate and the scroll request count doubles; the aggregation request count stays the same. This is why the gap widens at scale: The more data you have, the worse scroll looks by comparison, and the better aggregations look.

On live data, the picture is different because each real-time tick covers only one fresh bucket: Scroll issues however many pages fit in that bucket's worth of data, while aggregations issue one request. The 20× figure for live data reflects that ratio at 836,000 events per hour with a 15-minute bucket span. The practical threshold where aggregations stop being optional is when (ingestion rate × bucket span) > scroll_size; once a single bucket contains more than one scroll page of documents, the datafeed can't keep pace with live data regardless of hardware. Below that threshold, scroll is fine and aggregations are a nice-to-have. Above it, aggregations are the only sustainable option.

Scroll-based datafeeds are the right default, and the wizards make the right call for most deployments. At scale (more shards, broader index patterns, CCS across tiers), switching to an aggregation-based datafeed is the natural next step: The data nodes summarize where the data lives, the ML node processes compact results, and the detections stay the same. The one cost to know up front is model state: Switching requires a new job, so the earlier you make the move, the less you give up.

If you hit a case not covered here, an aggregation type that doesn’t map cleanly or a composite that behaves unexpectedly, the Elastic Discuss forums are a good place to continue.

The three limits that actually matter

There’s no hard storage ceiling at the node level. Elastic has demonstrated a single node querying 1 PiB of data. In earlier versions, the per-shard overhead was high enough that the old rule of thumb was no more than 20 shards per GB of heap. Exceeding that limit meant garbage collection pressure, slow cluster state updates, and unstable nodes. Over 7.x and 8.x, a series of optimizations (more compact metadata serialization, efficient caching, off-heap data structures, and compressed cluster state) reduced per-shard overhead to the point where that rule was retired in 8.3, replaced by field-density-based sizing.

What defines your actual ceiling is workload type. Cold nodes at 20 TB with 31 GB of heap handle audit and retention workloads comfortably, because the access pattern is infrequent and aggregation-based. The same spec on a high-concurrency document search workload would struggle.

The three things to watch operationally:

• Shard size: Individual shards that are too large slow queries and recovery.
• Shards per node: Every node has a ceiling, and index lifecycle management (ILM) creates shards automatically whether you track them or not.
• Storage tier mismatch: Keeping data on expensive fast storage longer than needed.

Shard size

Target between 10 GB and 50 GB per shard. The official guidance sets the ILM rollover trigger at 50 GB per primary shard, with 10 GB as the suggested floor. Keep each shard under 200 million documents.

Shards that are too small create unnecessary overhead: more metadata for master nodes, more heap consumed, more network traffic. Shards that are too large slow query execution and make recovery after node failure slow, since Elasticsearch recovers one shard at a time.

One rule you can stop using: The "20 shards per GB of heap" guideline was deprecated in Elasticsearch 8.3. The replacement is simpler: Watch the 1,000 shard-per-node limit below, and keep shard sizes in the 10–50 GB range (or 200M documents).

How to monitor:

# size per shard
GET _cat/shards?&h=index,store&v

The shard budget

Each non-frozen data node supports up to 1,000 shards. ILM creates shards on your behalf. If your policy rolls over daily with five primary shards and one replica, that’s 10 shards per day. One node fills up in about 100 days without you changing anything.

Options when you’re approaching the ceiling:

• Wider rollover intervals: Roll weekly or monthly if shards are not reaching 50 GB before the time trigger fires.
• Fewer shards per index: For smaller daily volumes, one or two primary shards is often enough. See how to increase primary shard count if you need to rebalance an existing index.
• More nodes: If volume genuinely requires daily rollover at full shard count, distribute across more nodes.

For master nodes, plan for 1 GB of heap per 3,000 indices.

How to monitor:

# shards per node
GET _cat/allocation?h=node,shards&v

Storage

The search speed guide recommends allocating at least half of system memory to the OS filesystem cache and using directly attached storage. Remote storage generally performs worse. The indexing speed guide echoes this, recommending RAID 0 across multiple local SSDs for write-heavy workloads.

For hot data: Don’t use network-attached storage (NAS). NAS adds latency on every read, and some NAS systems don’t correctly implement POSIX filesystem semantics, which can cause data corruption. Use local SSDs.

What works at each tier:

How to monitor:

# disk usage per node and role
GET _cat/allocation?h=node,node.role,disk.used,disk.avail,disk.percent&v

On Elastic Cloud, skip this section. You select a hardware profile per tier, and Elastic handles storage provisioning.

Data tiers and ILM

Index lifecycle management moves data through tiers automatically: hot, warm, cold, frozen, delete. The further data moves from hot, the cheaper the storage.

The cold and frozen tiers use searchable snapshots:

• Cold (fully mounted): Performance comparable to a regular index, no replicas required, roughly 50% cheaper than warm.
• Frozen (partially mounted): Up to 20x storage reduction compared to warm, slower queries, requires an Enterprise license.

The cost difference at scale is significant. A Search Labs benchmark measured 90 TB of data: all-hot cost $28,222 per month. A hot+frozen architecture brought that to $3,290 per month.

A typical ILM policy for time-series data with a 14-day hot window:

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_primary_shard_size": "50gb" }
        }
      },
      "warm": {
        "min_age": "14d",
        "actions": {
          "shrink": { "number_of_shards": 1 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "my_repository"
          }
        }
      },
      "frozen": {
        "min_age": "90d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "my_repository"
          }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": { "delete": {} }
      }
    }
  }
}

Adjust min_age values to match your query patterns. Data queried weekly can move to cold sooner than data queried daily.

AutoOps

As of February 2026, AutoOps is free for all Elasticsearch users regardless of license tier. On Elastic Cloud, it’s already enabled. For Elastic Self-Managed, Elastic Cloud Enterprise (ECE), and Elastic Cloud on Kubernetes (ECK) deployments, a lightweight Elastic Agent connects your cluster in about five minutes via Cloud Connect. Internet connectivity is required; air-gapped deployments are not supported.

AutoOps samples hundreds of metrics every 10 seconds and surfaces issues with root cause analysis and remediation commands. It does not apply fixes automatically.

For large deployments it detects:

• Shards growing past the recommended size range.
• Indices without ILM policies that have grown too large.
• Shard imbalance across nodes.
• Disk watermark violations before they cause allocation failures.
• Indexing rejections and ingestion bottlenecks.
• Slow queries and circuit breaker trips from large aggregations.

It ships with 100+ customizable alerts and routes notifications to PagerDuty, Slack, Teams, or any webhook.

Conclusion

Watch shard size (10–50 GB), track your per-node shard budget as ILM rolls, put hot data on local SSDs, and use cold and frozen tiers for data that is rarely queried.

On Elastic Cloud, hardware profiles and AutoOps handle most of this for you. For self-managed deployments, this is your checklist, and AutoOps via Cloud Connect is your early warning system. If you’re unsure how much data your nodes can handle for your specific workload, use Rally to benchmark against your own data before committing to a hardware spec.

Sources

• Size your shards
• Data tiers
• Elasticsearch shards and replicas guide
• How to reduce shard count
• How to increase primary shard count
• Optimize disk space and usage
• Searchable snapshots benchmark
• AutoOps documentation
• Rally: Elastic's benchmarking framework for testing cluster sizing against your own data
• Optimizing storage efficiency in Elasticsearch webinar by Christian Dahlqvist and Alan Woodward
• Using Rally to get your cluster size right: webinar by Christian Dahlqvist and Daniel Mitterdorfer on benchmarking methodology

The Elasticsearch Java client supports ES|QL queries through the DSL, but currently it treats queries as simple strings, with no dedicated helper; and while Kibana offers an excellent UI to build ES|QL queries, we’re aware that sometimes having everything needed to write applications in the integrated development environment (IDE) offers a better experience. So, until the Java client extends its type support to ES|QL, we wrote an Intellij IDEA plugin that autocompletes, syntax checks, shows documentation, and executes ES|QL queries.

The plugin currently supports Java, Kotlin, and plain text files, in case the Java Virtual Machine (JVM) isn’t your thing.

Check it out in the JetBrains Marketplace page and in the GitHub repository, for more information.

Prerequisites

• IDE: Intellij IDEA version >= 253 (community or ultimate)

Usage

Install the plugin in Intellij IDEA like you would with every other plugin, so either from the JetBrains marketplace or by going to Settings -> Plugins -> Marketplace and searching “esql”.

The following examples are written using Java, but Kotlin is also supported and the usage is pretty much the same.

Create a text block string, write “ES|QL” in a simple comment above it, and you’re done.

// ES|QL
String query = """
""";

If you see the Elastic logo appearing on the left:

then everything is working, and you’re ready to write your queries.

Why text blocks and not simple strings? The ES|QL syntax accepts quotes in various contexts, and escaping them would trigger other errors in the syntax checker, so we decided on text blocks to keep things simple.

It’s even simpler for txt files, as you can just add the comment and start writing the query right below:

// ES|QL

Connecting to a server instance

The plugin can be connected to an Elasticsearch server instance to fetch indices and field names, which will then be added to the autocompletion options. Look for the Elastic logo on the bottom left of of the screen (or wherever you keep your tools), and configure your connection to any server instance:

Autocomplete

Start typing while in the text block to automatically open the autocompletion popup, which will return a list of acceptable commands/values to continue writing the query correctly. If you want to manually trigger autocompletion, ctlr+space is the IDE’s shortcut to use:

Syntax check

The plugin will highlight errors in queries, explaining what to fix:

Documentation

Hovering with the cursor over commands will display documentation describing what the command can be used for and its correct syntax:

Running the query

Once connected to a server instance, you can run queries by clicking on the green button beside the Elastic icon: The results will be displayed in the tool window:

Or if you’re writing an application, you can use the Java client like so:

// ES|QL
String query = """
	FROM my-index
| SORT year DESC
| LIMIT 10
""";

try (ElasticsearchClient client = ElasticsearchClient.of(e -> e
                .host(serverUrl)
                .apiKey(apiKey))) {

client.esql().query(QueryRequest.of(qr -> qr.query(query)));

}

Check our previous ES|QL Java Client article for a complete example of mapping ES|QL results to Java objects.

How does it work?

There’s no AI involved; the plugin is based on the ES|QL ANTLR grammar for autocompletion and syntax check, and it uses the Elasticsearch docs to show documentation.

Conclusion

The plugin is still experimental, so feel free to report any bug or feature request on the Github repository.

Organizations accumulate large document collections, like support tickets, legal filings, news feeds, research papers, and need to understand what's in them before they can ask the right questions. Without labels or training data, manually reviewing thousands of documents is impractical. Traditional search doesn't help when you don't know what to search for.

This post walks through an Elasticsearch-native approach to unsupervised document clustering and temporal story tracking that addresses this discovery problem. By the end, you can trace story arcs like this across days:

What you'll discover:

• Why clustering embeddings (not retrieval embeddings) matter when you want topic discovery without a query.
• How density-probed centroid classification groups documents by topic using Elasticsearch k-nearest neighbor (kNN) and batched msearch.
• How significant_text can auto-label clusters so themes are readable without training a model.
• How temporal story chains link daily clusters to show how themes evolve from day to day.

The pipeline uses ~8,500 February 2025 articles from BBC News and The Guardian as a test corpus. News is convenient because it has clear temporal behavior, but the pattern applies anywhere document discovery matters: legal review, compliance monitoring, research synthesis, customer support triage.

Stack:

• Jina v5 clustering embeddings: Task-specific Low-Rank Adaptation (LoRA) adapters for topic grouping. Jina has joined Elastic, and its models are available natively through Elastic Inference Service (EIS).
• Elasticsearch: Scalable kNN, significant_text labeling, and vector storage.
• DiskBBQ: A disk-based vector index format that combines Better Binary Quantization (BBQ) with hierarchical k-means partitioning for approximate nearest neighbors (ANN) acceleration. This index partitioning is internal to vector search and separate from the density-probed clustering algorithm used in this post. bbq_disk stores quantized vectors on disk and keeps only partition metadata in heap, dramatically reducing resource requirements, compared to bbq_hnsw, while maintaining high recall.
• Global clustering + daily temporal linking: Discovery and story evolution.

What you'll need:

• An Elasticsearch deployment (Elastic Cloud, Elasticsearch Serverless, or Elastic Self-Managed 8.18+/9.0+): bbq_disk requires 8.18 or later. The optional diversify retriever section requires 9.3+ or serverless.
• A Jina API key: The free tier includes 10 million tokens, which covers the core clustering pipeline (~4.25 million tokens). The optional retrieval-versus-clustering comparison uses a second embedding pass.
• A Guardian API key (free).

Setup

Install required packages:

pip install elasticsearch pandas numpy plotly umap-learn python-dotenv pydantic-settings datasets requests

Optional (only if you run scraping helpers from this repo):

pip install beautifulsoup4

Then configure API keys in a .env file at the project root:

ELASTIC_CLOUD_ID=your-cloud-id        # or ELASTIC_HOST=https://...
ELASTIC_API_KEY=your-api-key
JINA_API_KEY=your-jina-key
GUARDIAN_API_KEY=your-guardian-key

This notebook calls load_dotenv(override=True), so local .env values take precedence.

Connected to Elasticsearch

Part 1: Discovery clustering - Why clustering embeddings?

Most vector search uses retrieval embeddings trained to match a query to relevant documents. That's perfect for search, but not for discovery. When you want to find what topics exist in a corpus without any query at all, you need embeddings that group similar documents together.

Jina v5 solves this with task-specific Low-Rank Adaptation (LoRA) adapters. LoRA adds small low-rank updates to targeted internal layers while keeping most base-model weights frozen, so the model behavior shifts toward a specific task without full retraining. The same base model produces different embeddings depending on the task parameter:

The clustering adapter is trained to make documents about the same topic closer in embedding space and documents about different topics further apart. The visual comparison below makes the difference concrete.

Retrieval vs. clustering: A visual comparison

To see the difference, a sample of documents is embedded with both task types. Clustering is performed in the original 1024-dimensional embedding space; Uniform Manifold Approximation and Projection (UMAP) is used only to project those embeddings into 2D for visualization. UMAP preserves local neighborhood structure, making it useful for comparing cluster separation.

Below, the same 480-document sample is embedded with both task types and projected to 2D with UMAP. Look for tighter, more separated color groups in the clustering panel.

    Full dataset: 8,495 articles
    Sources: guardian: 5749, bbc: 2746
    Date range: 2025-02-01 to 2025-02-28


    Sample: 480 docs across 8 sections
    section
    Film              60
    World news        60
    Australia news    60
    Opinion           60
    Football          60
    US news           60
    Sport             60
    Business          60


    Clustering embeddings: 480
    Retrieval embeddings:  480


    UMAP projection complete

Retrieval embeddings (left) spread topics broadly; clustering embeddings (right) produce tighter, more separated groups from the same documents.

The clustering embeddings produce tighter, more visually distinct groups. The retrieval embeddings spread topics out more evenly, ideal for search (fine-grained similarity); but for discovery, tight topical clusters are what matters.

This is why task="clustering" is used for the rest of this walkthrough.

Loading the dataset

The corpus combines two news sources for February 2025:

• BBC News via the RealTimeData/bbc_news_alltime HuggingFace dataset.
• The Guardian via the Guardian Open Platform API.

Having multiple sources helps validate that clustering finds topics rather than source-specific style.

    Total articles:  8,495
    
    Source breakdown:
    source
    guardian    5749
    bbc         2746
    
    Date range: 2025-02-01 → 2025-02-28
    Days covered: 28
    
    Sample article:
      Source:  guardian
      Title:   Carbon monoxide poisoning ruled out in death of Gene Hackman and wife, police sa
      Section: Film
      Text:    Authorities have ruled out that Gene Hackman and his wife, Betsy Arakawa, died from carbon monoxide poisoning earlier this week in their home in Santa Fe, New Mexico. The Santa Fe county sheriff, Adan...

Embedding with the clustering task

The Jina v5 API is called with task="clustering" for all documents. Embeddings are cached to disk, so subsequent runs skip the API entirely.

The API call is straightforward. The task parameter is the key difference from typical embedding usage:

payload = {
    "model": "jina-embeddings-v5-text-small",
    "input": texts,
    "task": "clustering",  # ← This selects the clustering LoRA adapter
}

The timing below reflects a cache hit. First run against the API takes longer, depending on corpus size.

    Embeddings ready: 8,495 vectors of dimension 1024
    Time: 0.6s

Indexing into a single Elasticsearch index

For discovery clustering, the full month goes into one index (docs-clustering-all). Daily partitioning comes later for temporal story linking.

The index mapping uses bbq_disk for the vector field:

{
  "embedding": {
    "type": "dense_vector",
    "dims": 1024,
    "index": true,
    "similarity": "cosine",
    "index_options": {
      "type": "bbq_disk"        // hierarchical k-means partitioning for ANN index lookup; separate from this post's clustering algorithm
    }
  }
}

A 1024-dimensional float32 vector is 4 KB. bbq_disk uses hierarchical k-means to partition vectors into small clusters, binary-quantizes them, and stores the full-precision vectors on disk for rescoring. Only partition metadata lives in heap, so memory requirements stay low even for large corpora. For workloads that can afford more heap, bbq_hnsw builds a Hierarchical Navigable Small World (HNSW) graph for faster lookups at higher resource cost.

The dense_vector field type supports multiple quantization strategies: bbq_disk and bbq_hnsw are the best fits for high-dimensional embeddings like the 1024-dim vectors used here.

    Indexed 8,495 documents into docs-clustering-all
    Time: 57.5s

Clustering: Density-probed centroid classification

Traditional clustering algorithms like HDBSCAN assume you can hold the full N×d vector matrix in memory and run repeated full-pass updates. For 8,495 documents at 1024 dimensions, that's manageable (~35 MB), but the approach doesn't scale to millions of documents without additional infrastructure.

This algorithm is conceptually similar to KMeans++ initialization with Voronoi assignment and a noise floor, but it uses Elasticsearch kNN search as the compute primitive, keeping almost all work server-side:

1. Sample 5% of documents as density probes (random sample, minimum 50).
2. Probe density via batched msearch kNN. Each probe fires a kNN query and records the mean similarity of its neighbours. High mean similarity = dense region of embedding space. msearch sends multiple search requests in a single HTTP call, which is critical here: Density probing generates hundreds of kNN queries, and batching them avoids per-request overhead.
3. Select high-density seeds with diversification: Candidates above median density are sorted by density descending and greedily accepted only when their cosine similarity to every existing seed is below a separation threshold. This is the only client-side compute (~0.01s for 8k docs).
4. Classify all docs against centroids via msearch kNN: Each seed acts as a centroid; a kNN search retrieves nearby documents above a similarity threshold. Each document is assigned to whichever centroid returned it with the highest score. Small clusters are dissolved to noise.

Elasticsearch handles the heavy lifting: msearch for density probes, msearch for classification, and significant_text for labeling. For this corpus (8,495 docs), the 5% density-probe sample launches 425 kNN probe queries, which msearch batches into nine HTTP calls (at batch size 50), avoiding one-request-per-probe overhead. Combined with bbq_disk ANN lookup, this keeps the clustering stage fast and scalable. The kNN queries use a minimal num_candidates value for speed during the clustering pass; production search queries should use higher num_candidates values to improve recall at the cost of latency.

Clusters have natural sizes determined by the embedding space density around each centroid, not by a hard k cap. Dense topic regions produce larger clusters; niche topics produce smaller ones.

Why not KMeans or HDBSCAN?

KMeans assumes spherical clusters and requires the full N×d matrix in memory. For corpora that fit in memory, HDBSCAN is a strong alternative. It handles arbitrary cluster shapes and has well-understood density semantics.

The density-probed centroid approach targets a different niche: corpora where you want storage, retrieval, and clustering in one system, or where scale makes client-side matrix operations impractical. It uses Elasticsearch kNN as the compute primitive, handles arbitrary cluster sizes, and keeps nearly all computation server-side.

    Clustered global index in 31.6s
      Total clusters: 82
      Total noise:    2420 (28.5%)
      Density probes: 425 kNN queries via 9 _msearch HTTP calls

Understanding the noise rate

The ~28% noise rate is by design, not a failure mode. Documents that don't fit any dense cluster at the configured similarity_threshold are left unassigned rather than forced into a poor match. This acts as a quality gate: Opinion columns, short articles, and one-off stories naturally resist clustering because they lack the thematic density that defines a coherent group.

The threshold is tunable: Lowering similarity_threshold produces more aggressive clustering (more documents assigned, but looser clusters), while raising it tightens clusters and increases the noise fraction. For this corpus of mixed news content, ~30% noise is a reasonable operating point. Production deployments should tune the threshold against domain-specific quality criteria.

Automatic labels with significant_text

Now each cluster needs a human-readable label. Elasticsearch's significant_text aggregation finds terms that appear unusually often in a foreground set (the cluster) compared to a background set (the full corpus).

Under the hood, it uses a statistical heuristic (JLH score by default) that balances absolute and relative frequency shifts, no machine learning, no large language model (LLM) calls. A cluster about UK politics might surface terms like starmer, labour, downing because those terms are disproportionately common in that cluster compared to the overall news corpus.

For this global pass, labels are computed directly against docs-clustering-all, so both foreground and background are drawn from the full month. In Part 2, labeling uses the daily index pattern (docs-clustering-*), a wildcard that lets queries span all matching indices simultaneously, to give significant_text a broader background for better contrast.

A minimal query shape looks like this:

{
  "size": 0,
  "query": { "term": { "cluster_id": "72" } },
  "aggs": {
    "label_terms": {
      "significant_text": {
        "field": "text",
        "size": 5,
        "filter_duplicate_text": true
      }
    }
  }
}

significant_text also serves as a quality gate: Clusters that produce no significant terms have no distinguishing vocabulary. They're incoherent groupings that should be dissolved back to noise rather than given a misleading label.

A lightweight deterministic cleanup step removes noisy label terms (numeric tokens, generic words) and falls back to a representative headline when needed. This keeps labels Elasticsearch native while improving readability.

    Sample cluster labels:
      cluster   3  (200 docs)  arsenal | mikel | villa
      cluster   1  (198 docs)  volodymyr | ukrainian | kyiv
      cluster   0  (196 docs)  hostages | hamas | israeli
      cluster   4  (187 docs)  scrum | rugby | borthwick
      cluster  52  (185 docs)  fossil | renewable | renewables
      cluster  10  (156 docs)  labour | gwynne | mps
      cluster  40  (151 docs)  novel | novels | literary
      cluster  11  (149 docs)  mewis | sarina | wiegman
      cluster  44  (143 docs)  flooding | rainfall | rain
      cluster  13  (131 docs)  doge | musk | elon
      cluster  12  (128 docs)  murder | insp | knockholt
      cluster   5  (124 docs)  putin | backstop | starmer


    Reassigned 35 docs from incoherent clusters to noise
    Total docs: 8,495
    Clustered:  6,040 (71.1%)
    Noise:      2,455 (28.9%)

Visualizing the clusters

The visualizations below show what the global clustering pass discovered: a date-wise breakdown of clustered versus noise documents, a UMAP projection of the full month, and a source-mix chart confirming that clusters reflect topics rather than sources.

Daily distribution of clustered versus noise documents across February 2025.

Each colored island in the UMAP represents a cluster: a group of articles about the same topic discovered purely from embedding similarity. The gray noise points are articles that didn't fit cleanly into any cluster (often short articles, opinion pieces, or one-off stories).

The source breakdown chart confirms that clusters contain articles from both BBC News and The Guardian. The clustering is finding topics, not sources, exactly what unsupervised discovery should produce.

Exploring cluster breadth with the diversify retriever

Plain kNN returns the documents most similar to a cluster's centroid (the dense core). But real clusters cover subtopics. The diversify retriever uses Maximal Marginal Relevance (MMR) to surface documents that are relevant to the centroid but also different from each other.

The key parameter is λ (lambda):

• λ = 1.0 → pure relevance (same as plain kNN).
• λ = 0.0 → pure diversity (maximally spread results).
• λ = 0.5 → balanced: that is relevant to the topic, but covering different angles.

A minimal retriever request shape looks like this:

{
  "size": 8,
  "retriever": {
    "diversify": {
      "type": "mmr",
      "field": "embedding",
      "lambda": 0.5,
      "query_vector": "",
      "retriever": {
        "knn": {
          "field": "embedding",
          "query_vector": "",
          "k": 50,
          "num_candidates": 100
        }
      }
    }
  }
}

The type, field, and query_vector parameters are required at the diversify level: field tells MMR which dense_vector field to use for inter-result similarity, and query_vector provides the reference point for relevance scoring.

This lets you answer: "What does this cluster actually cover?" rather than just "What's at its center?"

    Exploring cluster 52 (185 docs)
    Label: fossil | renewable | renewables
    Centroid computed (dim=1024)


    ========================================================================
    Plain kNN (closest to centroid)
    ========================================================================
      1. [0.9738] Green campaigners fear ministers are poised to award billions of pounds in fresh subsidies to Drax power station, despite strong concerns...
      2. [0.9710] Thirteen more oil and gas licences could be cancelled as ministers decide new guidance for fossil fuel extraction after a landmark court...
      3. [0.9699] Experts have accused the fossil fuel industry of seeking special treatment after lobbyists argued greenhouse gas emissions from oilfields...
      4. [0.9681] Burning wood is a terrible way of producing electricity . Chopping down trees destroys habitats for wildlife, and growing new trees cannot...
      5. [0.9649] Keir Starmer will do huge damage to the global fight against climate change if he gives in to political pressure and allows the development...
      6. [0.9641] Labour will next week be confronted with stark policy choices that threaten to expose the fault lines between the Treasury and the...
      7. [0.9638] The Drax power station near Selby in north Yorkshire burns imported wood pellets  The government has agreed a new funding arrangement with...
      8. [0.9581] If you care about the world we are handing on to future generations, the news on Thursday morning was dramatic. This January was the...
    
    ========================================================================
    Diversify retriever (MMR, lambda=0.5)
    ========================================================================
      1. [0.9738] Green campaigners fear ministers are poised to award billions of pounds in fresh subsidies to Drax power station, despite strong concerns...
      2. [0.9434] Oil and gas interests have waged a coordinated campaign to kill pro-electrification policies that ban gas connections in new buildings ,...
      3. [0.9303] It was interesting to read that new licences for oil and gas production in the North Sea are being delayed by legal action ( Thirteen more...
      4. [0.9139] The US energy secretary, Chris Wright, has said he “would love to see Australia get in the game of supplying uranium and maybe going down...
      5. [0.9077] Rachel Reeves was facing criticism on Saturday night as it was confirmed that a report she cited as evidence that a third ­runway at...
      6. [0.8996] When Margaret Thatcher opened the Hadley Centre for Climate Change in 1990 journalists suggested she was attempting to appear to be doing...
      7. [0.8993] The vast majority of governments are likely to miss a looming deadline to file vital plans that will determine whether or not the world has...
      8. [0.8987] European imports of seaborne gas shipments fell by a fifth last year to their lowest level since the pandemic, according to a new report,...
    
    Overlap: 1/8 documents appear in both result sets
    
    Avg pairwise similarity (lower = more diverse):
      Plain kNN:          0.9057
      Diversify retriever: 0.6965

The plain kNN results cluster around one angle of the topic: the documents most similar to the centroid and to each other. The diversify retriever surfaces different facets of the same cluster: subtopics, different sources, and varied perspectives.

The diversity metric confirms this quantitatively: the average pairwise similarity is lower for the diversify retriever results, meaning that the returned documents cover more ground.

This is useful for:

• Understanding what a cluster actually covers, not just its center but also its edges.
• Generating summaries. Diverse representative docs give an LLM better material.
• Finding representative examples for human review or downstream labeling.
• Quality checks. If the diverse results look incoherent, the cluster may need splitting.

Part 2: Temporal story chains

Tracking stories across days

Part 1 clustered the full month globally for topic discovery. For temporal flow, the same density-probed centroid classification runs independently per day on daily indices, and then clusters are linked across adjacent days. Note that the daily clusters are independent of the global clusters from Part 1; each day produces its own cluster assignments and labels tuned to that day's content.

The linking approach: sample-and-query

For each cluster on day A:

1. Sample a few representative documents.
2. Run kNN against day B's index.
3. Count how many hits land in each day B cluster.
4. If the hit fraction exceeds a threshold (kNN fraction ≥ 0.4), record a link.

This is fast (only a few docs per cluster are queried, not all of them) and uses Elasticsearch's native kNN, no external tools needed.

Preparing daily indices for temporal linkage...


Indexed 8,495 docs into 28 daily indices


Temporal links found: 808 in 145.4s

Strongest links:
  2025.02.01 'league | arsenal | premier' -> 2025.02.02 'league | season | striker'  (100%)
  2025.02.03 'league | striker | loan' -> 2025.02.04 'league | striker | season'  (100%)
  2025.02.03 'score | operator | gedling' -> 2025.02.04 'league | striker | season'  (100%)
  2025.02.12 'playoff | leg | bayern' -> 2025.02.13 'league | players | injury'  (100%)
  2025.02.14 'league | injury | football' -> 2025.02.15 'league | premier | football'  (100%)
  2025.02.18 'russia | ukraine | talks' -> 2025.02.19 'saudi | russia | arabia'  (100%)
  2025.02.18 'football | league | bayern' -> 2025.02.19 'league | manchester | players'  (100%)
  2025.02.21 'league | premier | manchester' -> 2025.02.22 'game | players | defeat'  (100%)
  2025.02.21 'rugby | calcutta | brilliant' -> 2025.02.22 'game | players | defeat'  (100%)
  2025.02.26 'metals | kyiv | ukrainian' -> 2025.02.27 'ukraine | russia | talks'  (100%)

A kNN fraction of 100% means every sampled document from the source cluster landed in the same target cluster, the strongest possible cross-day link. Most links above are football-related, which makes sense: Premier League coverage runs daily with high topical consistency.

The score | operator | gedling → league | striker | season link is an example of a niche local football cluster (Gedling is a non-league club) being absorbed into the broader Premier League cluster on the next day, a natural effect of daily reclustering at different granularity.

Building story chains

A story chain is a sequence of linked clusters across consecutive days.

Individual pairwise links tell you that Monday's "UK politics" cluster connects to Tuesday's. Chains reveal the full arc: a story that starts Monday, evolves through the week, and fades by Friday.

Chains are built greedily from links with a kNN fraction ≥ 0.4, meaning that at least 40% of sampled documents from the source cluster landed in a single target cluster. Starting from the earliest cluster, the algorithm always follows the strongest outgoing link.

    Strong links (kNN fraction >= 0.4): 244
    Story chains spanning 3+ days: 18
      Chain 1: 'ukrainian | kyiv | eastern' (19 days: Feb 3 → Feb 21)
      Chain 2: 'playing | opposition' (19 days: Feb 10 → Feb 28)
      Chain 3: 'tadhg | maro | cadan' (10 days: Feb 1 → Feb 10)
      Chain 4: 'invade | china | putin' (8 days: Feb 21 → Feb 28)
      Chain 5: 'elected | labour | leader' (7 days: Feb 12 → Feb 18)
      Chain 6: 'film | swift | awards' (6 days: Feb 2 → Feb 7)
      Chain 7: 'amendment | termination | reporting' (6 days: Feb 12 → Feb 17)
      Chain 8: 'officers | scene | police' (5 days: Feb 1 → Feb 5)

The longest chain tracks Ukraine–Russia coverage for 19 consecutive days, unsurprising given the sustained geopolitical intensity in February 2025. The second-longest follows Premier League football across 19 days of the month. Shorter chains capture award season (film/awards, six days), Six Nations rugby (10 days), and UK political leadership coverage (seven days). Each chain represents a story arc that the algorithm discovered purely from embedding similarity across daily indices.

Sankey: Visualizing story flow

A Sankey diagram is a flow visualization where link width represents connection strength. Here, each vertical band is a day, each node is a daily cluster (sized by document count), and each colored path traces one story chain across time. Link width encodes kNN overlap strength: Thicker links mean more sampled documents landed in the target cluster. Colors are consistent per chain, so a single color path from left to right reads as one story's progression.

For example, the Ukraine–Russia chain (visible as one of the longer paths) flows continuously from early February through the third week, with consistently thick links indicating strong topical continuity across days.

Temporal story chains flowing across February 2025. Each colored path is a story persisting across days; link width indicates kNN overlap strength.

What this approach delivers

This walkthrough covered a complete unsupervised document clustering pipeline built on Elasticsearch:

1. Clustering embeddings: Jina v5's task-specific adapters produce embeddings optimized for topic grouping, not just query-document matching.
2. Global discovery clustering: Clustering the full month in one index maximizes cross-day topical discovery.
3. Density-probed centroid classification: Sample 5%, probe density via msearch kNN, select diverse high-density seeds, classify all docs against centroids. Elasticsearch handles the heavy compute; only seed selection runs client-side (~0.01s).
4. significant_text labeling: Significance testing produces meaningful cluster labels without any ML model or manual annotation. Clusters that produce no significant terms are incoherent and get demoted to noise — a built-in quality gate.
5. Temporal story linking: Daily indices and sample-and-query cross-index kNN trace how stories evolve over time.

Key takeaways:

• The embedding task type matters: Clustering embeddings produce measurably tighter topical groups.
• Elasticsearch can serve as both the storage layer and the clustering engine via kNN search.
• Density-probed centroid classification keeps nearly all compute server-side and produces clusters with natural sizes determined by embedding space density.
• significant_text is fast, interpretable, and effective for both auto-labeling and quality gating.

When this approach is useful:

• You have timestamped text and want topic discovery without labeled training data.
• You want one stack for storage, vector search, labeling, and temporal linkage.

Extensions to explore:

• Multi-period clustering (weekly, monthly rollups).
• Real-time ingestion with incremental cluster assignment.
• LLM-generated cluster summaries using the significant_text terms as seeds.
• At larger scale, sampled KMeans centroids can serve as warm-start seeds for density-based clustering, reducing the probe phase cost.

Try it yourself

Swap in your own timestamped document corpus; any collection of text with dates works with this pipeline. The full notebook and supporting code are available in the companion repository.

• Start a free Elastic Cloud trial: Spin up a managed cluster with bbq_disk support in minutes.
• Try Elasticsearch Serverless: No cluster management, scales automatically, and supports everything in this walkthrough.

Lexical retrieval (text matching), semantic retrieval (matching concepts), and hybrid retrieval (combining lexical and semantic signals) don’t solve these issues on their own. Lexical retrieval may return anything containing the word “oranges”, while pure semantic retrieval on a high-intent query like “oranges” may broaden toward related items, such as lemons or grapefruits. Hybrid retrieval blends these lexical and semantic signals, but it still doesn’t determine if this query should be treated as navigational, which constraints should be enforced, or which business policies should apply. The gap isn’t the retrieval technology itself; it’s the absence of a governance layer that understands what kind of query this is and what constraints should be enforced before retrieval begins.

In this blog, we explore ecommerce search governance, why it matters, and how a control layer ensures predictable, accurate retrieval.

What governance means in ecommerce search

Governance, in this context, means introducing a decision layer between the user's query and the retrieval engine. This layer performs the following functions:

• Classifies query intent: Is this navigation ("oranges") or discovery ("gift for grandpa")?
• Applies business constraints: What category boundaries, eligibility rules, availability constraints, or merchandising policies apply?
• Routes to the appropriate strategy: Should this use lexical retrieval, semantic retrieval, or hybrid?

A governance layer determines which retrieval approach should be used for each query, which constraints must be enforced, and which business policies should apply before retrieval begins. It’s important not to conflate governance with hybrid retrieval: hybrid is one retrieval strategy that combines lexical and semantic signals, while governance is the upstream decision layer that determines whether lexical, semantic, or hybrid should be used.

The status quo: The application layer "spaghetti" implementation

Currently, many retailers attempt to solve this by adding logic directly into the application layer. This often results in spaghetti code, that is, thousands of lines of hard-coded if-then statements, regex, and complex search templates.

This approach can provide desired search results as shown above; however, it creates significant operational friction:

• Engineering dependency: Business users and merchandisers cannot modify search behavior without engineering tickets and long deployment cycles that often span several weeks.
• Fragmentation: Search logic becomes scattered between application code and search templates, and is difficult to explain or audit, making it risky to evolve.

Even when teams recognize the need for routing, the debate often focuses on the wrong question: which retrieval method to pick.

The false choice: Lexical vs. semantic vs. hybrid

Search teams often frame the challenge as a retrieval strategy choice: lexical/BM25 versus semantic/vectors versus hybrid. That framing is understandable (retrieval methods matter), but it misses the most common failure mode in real deployments, which is that using a single retrieval approach for all queries will give suboptimal results.

Commerce search is a mix of fundamentally different intents:

• Deterministic, high-intent navigation ( "oranges", “milk”, “chocolate without peanuts”, “cheap olive oil”).
• Exploratory discovery ("jacket for hiking in the mountains", "gift for a 12-year-old who likes robotics").
• Operational constraints (availability, size, price, color).
• Merchandising and campaigns (boost, bury, seasonal campaigns).

When the system routes all of these through the same retrieval strategy, the results are often systematically wrong in predictable ways because the operating model lacks governance. When teams don't recognize this as a governance gap, they respond with the only lever they have: more tuning.

Why "relevance tuning" can become cyclical

Without a routing layer, “relevance” often turns into a never-ending backlog:

• Why is this query showing accessories above the core product?
• Why did this head query suddenly start surfacing related items?
• Why did results change after we added synonyms, adjusted analyzers, or enabled hybrid?
• Why does the business team need an engineering release to fix a single query?

Teams respond with more tuning: more synonyms, more boosts, more reranking experiments, more exceptions in application code. This can work for a while, but it often produces brittle behavior because the system still lacks an explicit decision layer for determining query type and enforcing the right constraints before retrieval.

The anatomy of ecommerce intent: Head and tail

In this section, we use “head” and “tail” as practical shorthand for common navigational and exploratory query patterns in ecommerce. In the real world, many queries contain aspects of both:

Head queries (deterministic intent)

These are direct, navigational queries where the user knows exactly what they want:

• Single-item intent ("oranges", "milk", "bread").
• Exact brands or product families ("iPhone 15 Pro", "Diet Coke").
• SKUs, model numbers, sizes ("ABC123", "air max 270").

For these queries, lexical retrieval can handle token correspondence (matching words), but the business also expects to respect constraints, return predictable rankings, and have controllable outcomes. A merchandiser needs to ensure that a query resolves within the correct category boundaries, respects eligibility, and surfaces specific business priorities.

Governance is required to enforce the intended resolution. For example, “oranges” should map to the produce category, not to orange juice, orange marmalade, or orange soda.

Tail queries (exploratory discovery)

These are descriptive, intent-rich queries where shoppers are exploring:

• "Gift for grandpa who has a sweet tooth"
• "Jacket for hiking in the mountains"
• "Shoes for standing all day"

Lexical retrieval often struggles here. Semantic retrieval excels because it can connect the query concept to the product, even when wording does not match. But semantic retrieval alone is rarely sufficient either. Real queries often require constraints to be enforced, regardless of which retrieval method is used.

Constraints are orthogonal to retrieval method

Applying constraints to semantic retrieval doesn’t mean hybrid search. These are orthogonal concepts. Constraints, such as filters and boosts in Elasticsearch, can be applied to any lexical, semantic, or hybrid retrieval. The challenge is deciding how the query should be interpreted, which constraints must be enforced, and which retrieval strategy should be used.

Below are some examples of queries that combine retrieval with hard constraints:

• Oranges: Lexical retrieval for “oranges” plus a category constraint, such as “Fruits” or “Produce”, eliminating orange marmalade, orange juice, and orange soda.
• Fruits high in vitamin C under $4: Semantic retrieval for nutritional intent plus constraints limiting results to the fruit category and products under $4.
• Comfortable shoes for work: Semantic retrieval for contextual intent plus a category constraint limiting results to shoes.

These queries can't be handled by a single approach:

• Pure lexical retrieval is often insufficient here because phrases like “high in vitamin C” or “comfortable” may not exist as clean, structured attributes. They may need to be inferred from product descriptions, reviews, or specifications.
• Pure semantic retrieval is also not always sufficient because, without explicit constraints, a query like “fruits high in vitamin C” might broaden toward vitamin supplements, fruit-flavored drinks, or high-vitamin vegetables outside the intended category and price range.

A governance layer determines whether a query needs lexical retrieval, semantic understanding, constraint enforcement, or some combination of these. Without this layer, ecommerce teams may end up:

• Over-constraining: Using lexical retrieval for semantic requests (for example, "gift for grandpa").
• Under-constraining: Using semantic queries for high-intent head queries (for example, “oranges”).

The governance challenge is to build a system that can make the right judgment call for each class of query.

What happens without governance

The most common failure mode is straightforward: Teams take the raw user query and pass it directly into a single retrieval strategy (lexical, semantic, or hybrid), without an intermediate governance layer.

Lexical retrieval misses intended resolution

When a user searches for “oranges”, a lexical retrieval strategy may return anything containing that token: orange juice, orange marmalade, or orange soda. The system matched the term correctly, but without governance it may not resolve the intended shopping context (the fruit).

Semantic retrieval broadens beyond intended constraints

When a user searches for “oranges”, a semantic system may retrieve conceptually related items across nearby product concepts. The system may correctly understand the broader domain (fruit or produce), but without explicit governance it can still over-broaden beyond the user’s intended constraint (specifically oranges).

The gap is governance

What’s required is an upstream decision layer that determines query intent and enforces the right constraints before retrieval begins. This fixes issues such as the following:

• Similar or related items appearing alongside what the user actually wanted.
• Blurred category boundaries ("beverages" versus. "produce").
• Inability to implement seasonal boosts or campaigns.
• Unpredictable and unexplainable results.

Intent understanding and routing: The necessary control plane

A governed search system introduces a lightweight control plane in front of retrieval (prior to executing a query in Elasticsearch). The control will be discussed in detail in parts 3 and 4 of this blog series; for now, we just discuss what it can do but not how it works:

A control plane can detect intent, apply business policies, and ensure the appropriate retrieval strategy as follows:

1. Detect intent signals

• Is this query likely navigation versus discovery?
• Is it a known head query (milk, bread, bananas)?
• Is there a known product, brand, or category interpretation (for example, “oranges” should resolve to produce).
• Is the query an SKU-like pattern?
• Does the query fall under an active campaign or seasonal policy (for example, during Christmas, boost turkey-related results)?
• Does the query imply constraints (category, attributes, exclusions, price/size/color)?

2. Apply governance and business policies

• Enforce deterministic constraints first (category/attribute/negation/availability).
• Apply active merchandising policies (boost/bury/pin/override).
• Resolve conflicts with precedence rules (for example, campaign overrides versus global policies).

3. Route to the appropriate retrieval strategy

• Lexical (fast, deterministic) for navigational/high-intent head queries.
• Semantic retrieval for true discovery queries.
• Hybrid where combined lexical and semantic signals add value under explicit business constraints.

In practice, the output of the control plane is not simply “use hybrid” or “use semantic.” It’s a governed retrieval plan: an interpretation of the shopper’s intent, the constraints and policies that should apply, and the retrieval strategy that should be executed. A few simple examples make this concrete:

A control plane selects the right policy and retrieval strategy for each query consistently, predictably, and at scale. This makes advanced retrieval methods more predictable in production because intent-aligned constraints are enforced first and routing decisions are explicit rather than implicit.

How this relates to other approaches

Some teams use improved embedding models to better capture product semantics, which can materially improve semantic retrieval quality. Others use reranking approaches, such as Learning To Rank (LTR), to optimize result ordering based on engagement or business signals after retrieval. Both are valuable and often complementary. Better embeddings improve similarity matching. Reranking improves ordering among retrieved candidates.

Governance addresses a different layer of the problem: It sits upstream of retrieval. It decides which retrieval strategy to use (for example, lexical, semantic, or hybrid), what deterministic constraints are required, and which queries should combine multiple business policies.

What a governed control plane enables

Once a governance layer is in place, the operating model changes fundamentally. Revenue-critical queries become predictable. Business teams can update search behavior without waiting on engineering release cycles. And advanced retrieval methods, like semantic and hybrid, can be adopted incrementally, behind routing and guardrails, instead of as a global on/off switch.

The next post in this series explores what that operating model looks like in practice and why it may matter as much as the retrieval technology underneath it.

If a merchandiser has to open a Jira ticket and wait for a deploy to fix a revenue-critical query, the bottleneck isn't the engine; it's the operating model. Modern ecommerce search needs a way to translate business intent into controlled, auditable search behavior quickly and safely, while still using advanced retrieval where it adds measurable value.

What’s next in this series

The patterns explored in this series operate upstream of retrieval: translating business intent into the right query strategy before query generation begins. In the next post, we shift from the technical problem to the operational one: what happens when business teams can change search behavior without an engineering deployment, and why governance makes that safe.

Put governed ecommerce search into practice

Engineering bottlenecks, brittle application-layer logic, and unpredictable search results are problems that Elastic Services can help you solve in enterprise ecommerce services engagements. The governed control plane architecture described in this series was built by Elastic Services Engineering.

If your team is spending engineering cycles translating merchandising requests into code changes, or if your search relevance backlog never seems to shrink, we can help you assess your current architecture and build a path to governed, business-editable search. Contact Elastic Services. 

Join the discussion

Have questions about search governance, retrieval strategies, or ecommerce search architecture? Join the broader Elastic community conversation.

We recently contributed to the mastra-ai/mastra open source project by adding support for Elasticsearch as a vector database. With this new feature, you can use Elasticsearch natively in Mastra to store embeddings. In addition to vectors, Elasticsearch provides a suite of advanced features to address all your context engineering requirements. (for example, hybrid search and reranking).

This article details the creation of an agent to implement a retrieval augmented generation (RAG) architecture using Elasticsearch. We’ll showcase a demo project where an agentic approach is used to interact with a corpus of sci-fi movie data stored within Elasticsearch. The project is available at elastic/mastra-elasticsearch-example.

Mastra

Mastra is a TypeScript framework to create agentic AI applications.

A project structure in Mastra looks as follows:

src/
├── mastra/
│   ├── agents/
│   │   └── weather-agent.ts
│   ├── tools/
│   │   └── weather-tool.ts
│   ├── workflows/
│   │   └── weather-workflow.ts
│   ├── scorers/
│   │   └── weather-scorer.ts
│   └── index.ts
├── .env.example
├── package.json
└── tsconfig.json

In Mastra, you can build agents, tools, workflows, and scores.

An agent is a class that accepts a message in input and produces a response as output. An agent can use tools, large language models (LLMs), and a memory (figure 1).

An agent's tools allow it to interact with the "external world," such as communicating with a web API or performing an internal operation, like querying Elasticsearch. The memory component is crucial for storing the history of conversations, including past inputs and outputs. This stored context enables the agent to provide more informed and relevant responses to future questions by using its past interactions.

Workflows let you define complex sequences of tasks using clear, structured steps rather than relying on the reasoning of a single agent (figure 2). They give you full control over how tasks are broken down, how data moves between them, and what gets executed when. Workflows run using the built-in execution engine by default or can be deployed to workflow runners.

In Mastra, you can also define scores, which are automated tests that evaluate agent outputs using model-graded, rule-based, and statistical methods. Scorers return scores: numerical values (typically between 0 and 1) that quantify how well an output meets your evaluation criteria. These scores enable you to objectively track performance, compare different approaches, and identify areas for improvement in your AI systems. Scorers can be customized with your own prompts and scoring functions.

Elasticsearch

For running the demo project, we need to have an Elasticsearch instance running. You can activate a free trial on Elastic Cloud or install it locally using the start-local script:

curl -fsSL https://elastic.co/start-local | sh

This will install Elasticsearch and Kibana on your computer and generate an API key to be used for configuring the Mastra integration.

The API key will be shown as output of the previous command and stored in a .env file in the elastic-start-local folder.

Install and configure the demo

We created an elastic/mastra-elasticsearch-example repository containing the source code of the demo project. The example reported in the repository illustrates how to create an agent in Mastra that implements a RAG architecture for retrieving documents from Elasticsearch.

We provided a dataset for the demo about sci-fi movies. We extracted 500 movies from the IMDb dataset on Kaggle.

The first step is to install the dependencies of the project with npm, using the following command:

npm install

Then we need to configure the .env file that will contain the settings. We can generate this file copying the structure from the .env.example file, using the following command:

cp .env.example .env

Now we can edit the .env, adding the missing information:

OPENAI_API_KEY=
ELASTICSEARCH_URL=
ELASTICSEARCH_API_KEY=
ELASTICSEARCH_INDEX_NAME=scifi-movies

The name of the Elasticsearch index is scifi-movies. If you want, you can change it using the env variable ELASTICSEARCH_INDEX_NAME.

We used OpenAI as embedding service, which means that you need to provide an API key for OpenAI in the OPENAI_API_KEY env variable.

The embedding model used in the example is openai/text-embedding-3-small, with an embedding dimension of 1536.

To generate the final answer, we used the openai/gpt-5-nano model to reduce the costs.

The RAG architecture allows you to use a less powerful (and typically less expensive) final LLM model because the heavy lifting of grounding the answer is done by the retrieval component (Elasticsearch in this case).

The smaller LLM is only responsible for two main tasks:

• Rephrasing/embedding the query: Converting the user's natural language question into a vector embedding for semantic search.
• Synthesizing the answer: Taking the highly relevant, retrieved context chunks (documents/movies) and synthesizing them into a coherent, final, human-readable answer, following the provided prompt instructions.

Since the RAG process provides the exact factual context needed for the answer, the final LLM doesn't need to be massive or highly complex and it doesn’t need to possess all the required knowledge within its own parameters (which is where large, expensive models excel). It essentially acts as a sophisticated text summarizer and formatter for the context provided by Elasticsearch, rather than as a full-fledged knowledge base itself. This enables the use of models like gpt-5-nano for cost and latency optimization.

After the configuration of the .env file, you can ingest the movies to Elasticsearch using the following command:

npx tsx src/utility/store.ts

You should see an output as follows:

🚀 Starting ingestion of 500 movies from 500_scifi_movies.jsonl...
Ingesting ░░░░░░░░░░░░░░░░░░░░░░░░ 1/500 (0%) | ok:1 | fail:0 | chunks:1 | eta:19m 33s | current:Capricorn One
Ingesting ░░░░░░░░░░░░░░░░░░░░░░░░ 2/500 (0%) | ok:2 | fail:0 | chunks:2 | eta:10m 32s | current:Doghouse
Ingesting ░░░░░░░░░░░░░░░░░░░░░░░░ 3/500 (1%) | ok:3 | fail:0 | chunks:3 | eta:7m 33s | current:Dinocroc
Ingesting ░░░░░░░░░░░░░░░░░░░░░░░░ 4/500 (1%) | ok:4 | fail:0 | chunks:7 | eta:6m 10s | current:Back to the Future           
Ingesting ░░░░░░░░░░░░░░░░░░░░░░░░ 5/500 (1%) | ok:5 | fail:0 | chunks:9 | eta:5m 14s | current:The Projected Man            
Ingesting ░░░░░░░░░░░░░░░░░░░░░░░░ 6/500 (1%) | ok:6 | fail:0 | chunks:11 | eta:4m 41s | current:I, Robot
...
✅ Ingestion complete in 1m 46s. Success: 500, Failed: 0, Chunks: 693.

The mapping of the scifi-movies index contains the following fields:

• embedding, dense_vector with 1536 dimension, cosine similarity.
• description, text containing the description of the movie.
• director, text containing the name of the director.
• title, text containing the title of the movie.

We generated the embeddings using the title + description. Since the title and the description are two separate fields, the concatenation of both ensures that the resulting embedding vector captures both the specific, unique identity (title) and the rich, descriptive context (description) of the movie, leading to more accurate and comprehensive semantic search results. This combined input gives the embedding model a better single representation of the document's content for similarity matching.

Run the demo

You can run the demo with the following command:

npm run dev

This command will start a web application at localhost:4111 to access Mastra Studio (figure 3).

Mastra Studio offers an interactive UI for building and testing your agents, along with a REST API that exposes your Mastra application as a local service. This lets you start building right away without worrying about integration.

We provided an Elasticsearch Agent that uses the createVectorQueryTool by Mastra as a tool for executing semantic search using Elasticsearch. This agent uses the RAG approach to search for relevant documents (that is, movies) to answer the user’s question.

This agent uses the following prompt:

You are a helpful assistant that answers questions based on the provided context.
Follow these steps for each response:

1. First, carefully analyze the retrieved context chunks and identify key information.
2. Break down your thinking process about how the retrieved information relates to the query.
3. Draw conclusions based only on the evidence in the retrieved context.
4. If the retrieved chunks don't contain enough information, explicitly state what's missing.

Format your response as:
THOUGHT PROCESS:
- Step 1: [Initial analysis of retrieved chunks]
- Step 2: [Reasoning based on chunks]

FINAL ANSWER:
[Your concise answer based on the retrieved context]

Important: When asked to answer a question, please base your answer only on the context provided in the tool. 
If the context doesn't contain enough information to fully answer the question, please state that explicitly and stop it.
Do not add more information than what is present in the retrieved chunks.
Remember: Explain how you're using the retrieved information to reach your conclusions.

If you click on the Mastra Studio > Agents menu and select Elasticsearch Agent, you can test the agent using a chat system. For instance, you can ask information regarding sci-fi movies with a question as follows:

Find 5 movies or TV series about UFOs.

You’ll notice that the agent will execute the vectorQueryTool. You can click on the invoked tool to have a look at the input and the output. At the end of execution, the LLM will reply to your question, given the context coming from the scifi-movies index of Elasticsearch (figure 4).

Mastra executes the following steps internally:

1. Vector conversion: The user's question, Find 5 movies or TV series about UFOs, is converted into a vector embedding using OpenAI's openai/text-embedding-3-small model.
2. Vector search: This embedding is then used to query Elasticsearch via a vector search.
3. Result retrieval: Elasticsearch returns a set of 10 movies highly relevant to the query (that is, those with vectors closest to the user's query vector).
4. Answer generation: The retrieved movies and the original user question are sent to the LLM, specifically openai/gpt-5-nano. The LLM processes this information and generates a final answer, ensuring that the user's request for five results is met.

The Elasticsearch Agent

Here we reported the source code of Elasticsearch Agent.

import { Agent } from "@mastra/core/agent";
import { ElasticSearchVector } from '@mastra/elasticsearch';
import { createVectorQueryTool } from '@mastra/rag';
import { ModelRouterEmbeddingModel } from "@mastra/core/llm";
import { Memory } from "@mastra/memory";

const es_url = process.env.ELASTICSEARCH_URL;
const es_apikey = process.env.ELASTICSEARCH_API_KEY;
const es_index_name = process.env.ELASTICSEARCH_INDEX_NAME;
const prompt = 'insert here the previous prompt';

const esVector = new ElasticSearchVector({
  id: 'elasticsearch-vector',
  url: es_url,
  auth: {
    apiKey : es_apikey
  }
});

const vectorQueryTool = createVectorQueryTool({
  vectorStore: esVector,
  indexName: es_index_name,
  model: new ModelRouterEmbeddingModel("openai/text-embedding-3-small")
});

export const elasticsearchAgent = new Agent({
  id: "elasticsearch-agent",
  name: "Elasticsearch Agent",
  instructions: prompt,
  model: 'openai/gpt-5-nano',
  tools: { vectorQueryTool },
  memory: new Memory(),
});

The vectorQueryTool is the tool that’s invoked to implement the retrieval part of the RAG example. It uses the ElasticSearchVector implementation that Elastic contributed to Mastra.

The agent is an object of the agent class that consumes the vectorQueryTool, the prompt, and a memory. As you can see, the code that we need to put in place for connecting Elasticsearch to an agent is very minimal.

Conclusion

This article demonstrated the simplicity and power of integrating Elasticsearch with the Mastra framework to build sophisticated agentic AI applications. Specifically, we walked through creating a RAG agent capable of performing semantic search over a corpus of sci-fi movie data indexed in Elasticsearch.

A key takeaway is the direct contribution by Elastic to the Mastra open source project, providing native support for Elasticsearch as a vector store. This integration significantly lowers the barrier to entry, as evidenced by the Elasticsearch Agent source code. Using the ElasticSearchVector and createVectorQueryTool, the complete setup for connecting Elasticsearch to your agent requires only a minimal number of lines of configuration code.

Elasticsearch provides several advanced features to enhance result relevance. For example, hybrid search significantly boosts accuracy by combining lexical search with vector search. Another interesting feature is reranking using the latest Jina models that can be applied at the end of hybrid search. To learn more about these techniques, consult the following articles from Elasticsearch Labs:

• Elasticsearch hybrid search by Valentin Crettaz
• An introduction to Jina models, their functionality, and uses in Elasticsearch by Scott Martens

We also encourage you to explore the provided example and begin building your own data-powered agents with Mastra and Elasticsearch. For more information about Mastra, you can have a look at the official documentation here.

Why this matters

• Security and support: The last Elasticsearch 6.x patch release was on January 13, 2022. ECK lets you upgrade at your own pace, with a supported operator from the creators of Elasticsearch. Remaining on an old Elasticsearch version exposes you to supportability risks or well-known security issues.
• Features you’ve been missing: Autoscaling, data tiers, machine learning (ML) jobs, searchable snapshots. None of these are available in the legacy operator.

Future-proof operations: ECK ships day-and-date with every new Elastic release, so you’re never stuck waiting again.

High-level plan

Feel free to bookmark this list. Each milestone is small, reversible, and validated before you move on.

0. Preflight checks

A. Health first: Run /_cat/health and make sure you’re green.

B. Disk watermarks: Keep at least 20% free before starting a migration.

C. Final snapshot: S3, GCS, NFS: It doesn’t matter, as long as you can mount the same repo in the new cluster.

1. If you don’t have object storage handy in your environment, you can use this solution-post by Red Hat to snapshot your data to local storage on the OpenShift cluster.

D. Review the documentation: Elastic provides thorough documentation for migrating data between Elasticsearch clusters.

1. Installing ECK 2.16.1 (your “bridge” operator)

ECK 2.16.1 is the last release that still accepts spec.version: 6.8.x, which makes it the ideal bridge between past and future Elasticsearch versions.

helm repo add elastic https://helm.elastic.co
helm repo update
oc create namespace elastic-system 

helm install elastic-operator elastic/eck-operator --version=2.16.1 -n elastic-system --create-namespace

You can keep the Red Hat operator in place; the two watch different Custom Resource Definitions (CRDs), so they won’t step on each other’s toes.

Keep in mind that with OpenShift, ECK might display some Transport Layer Security (TLS) errors in its logs as OpenShift tries to connect to its healthcheck webhook endpoint via HTTP, but ECK allows TLS communication only. This is a well-known issue and shouldn’t pose a problem.

You can refer further to Elastic documentation, in case you need to make a local namespaced installation.

2. Launching a 6.x cluster under ECK

Below is a starter Kubernetes manifest that balances resiliency (separate masters) with cost (three hot-tier data nodes). Swap storage class names, resources, and snapshot credentials to match your environment.

Note: The syntax used below is a bit different than what it would be for newer Elasticsearch versions on ECK.

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: es-logs
  namespace: elastic # Create this namespace prior, or use another namespace
spec:
  version: 6.8.23
  nodeSets:
    - name: hot
      count: 3
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            storageClassName: gp3-csi   # adjust if needed
            resources:
              requests:
                storage: 100Gi # Storage may vary depending on  
      config:
        node.master: true
        node.data: true
        node.ingest: true
        node.attr.data: hot
        cluster.routing.allocation.awareness.attributes: data
      podTemplate:
        spec:
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 16Gi
                  cpu: 2
                limits:
                  memory: 16Gi
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: elastic
spec:
  version: 6.8.23
  count: 1
  elasticsearchRef:
    name: es-logs
  podTemplate:
    spec:
      containers:
        - name: kibana
          resources:
            requests:
              memory: 1Gi
              cpu: 0.5
            limits:
              memory: 4Gi

Deploy it, watch pods come up, and you’re ready for data.

3. Moving the data

To move data from one Elasticsearch cluster to another, you can also further consult this guide in the Elastic documentation. For the purpose of this post, we’re assuming that snapshot and restore are used.

Snapshot and restore are quickest:

# on the old cluster, take a snapshot
PUT _snapshot/log-backups
{
  "type": "s3",
  "settings": { ... }
}

PUT _snapshot/log-backups/final-snap-2025-08-07

# on the new cluster (readonly!)
PUT _snapshot/log-backups
{
  "type": "s3",
  "settings": {
    "readonly": true,
    ...
  }
}

# Perform the restore operation
POST _snapshot/log-backups/final-snap-2025-08-07/_restore

Can’t share an object store? Use remote re-index (slower, but works everywhere; has drawbacks in terms of not migrating index templates, component templates, and more) or pump logs through a one-off Logstash job.

4. Configuring ClusterLogging operator

First, we’ll need to decommission our Red Hat operator–managed Elasticsearch cluster. We’ll modify our ClusterLogging like so:

oc edit clusterlogging instance -n openshift-logging 
---------
 logStore:
    elasticsearch:
      nodeCount: 0 # scale down node count, previously > 0 
      redundancyPolicy: ZeroRedundancy
    type: elasticsearch
  managementState: Managed # this needs to be kept, as it will manage the fluentd instance for us.
  visualization:
    kibana:
      replicas: 0 # scale down kibana as well 
    type: kibana

Then we’ll define a ClusterLogForwarder to direct the logs from fluentd to our newly built Elasticsearch 6.x cluster managed by ECK. We’ll need to create a secret with the Elasticsearch credentials:

oc create secret generic eck-es-credentials \
  -n openshift-logging \
  --from-literal=username=elastic \
  --from-literal=password=$(oc get secret es-logs-es-elastic-user -n elastic -o jsonpath='{.data.elastic}' | base64 -d)

For configuring TLS (as recommended), you’ll need to create a ConfigMap for ClusterLogForwarder to trust the ECK ca certificates. Further guidance can be found here. We’ll run the command: 

oc -n elastic get secret es-logs-es-http-certs-public \
-o go-template='{{index .data "tls.crt" | base64decode}}' > ca.crt

oc -n openshift-logging create configmap eck-es-ca \
--from-file=ca-bundle.crt=ca.crt

To create the certificate secret, and then we’ll reference it in the ClusterLogging CRD:

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
    - name: eck-es
      type: elasticsearch
      url: https://es-logs-es-http.elastic.svc:9200
      secret:
        name: eck-es-credentials # this secret needs to be created first
      tls:
        # insecureSkipVerify: true # can be used for lab testing purposes
        ca:
          name: eck-es-ca
  pipelines:
    - name: send-to-eck
      inputRefs:
        - application
        - infrastructure
        - audit
      outputRefs:
        - eck-es

⚠️ If you’re troubleshooting connectivity issues, you can temporarily set tls.insecureSkipVerify: true, but this shouldn’t be used long term.

Because we’re restoring legacy indices into a fresh ECK-managed cluster, OpenShift Logging will not automatically recreate the legacy index layout or aliases. You must ensure that write aliases exist and point to writable indices. In my case, I needed to verify that I have proper aliases, set up as:

• app-write
• infra-write
• audit-write

Pointing to indices with dynamic mappings (not recommended) for minimizing errors and troubleshooting steps. 

# Forward ES port to local machine
oc -n elastic port-forward svc/es-logs-es-http 9200:9200

PASS="$(oc -n elastic get secret es-logs-es-elastic-user -o jsonpath='{.data.elastic}' | base64 -d)"

# Make sure the write alias points to the correct backing index
curl -s -k -u "elastic:${PASS}" -XPOST "https://localhost:9200/_aliases" \
  -H 'Content-Type: application/json' \
  -d '{
    "actions": [
      { "add": { "index": "infra-000002", "alias": "infra-write", "is_write_index": true } }
    ]
  }'

Repeat for app-write and audit-write with their respective backing indices.We should see data start flowing now toward our new ECK managed cluster.

5. Rolling upgrade to 7.17.29, and verify

Now you can finally leave 6.x behind.

A. Check _xpack/migration/deprecations?pretty using curl against Elasticsearch, to tackle deprecations. This API will return either warnings or critical things to attend to before you upgrade.

B. Patch the CRD to upgrade it to the latest 7.x version. I’m using 7.17.29.

oc -n elastic patch elasticsearch es-logs --type=merge -p '{"spec":{"version":"7.17.29"}}'

C. ECK restarts nodes one at a time. Your cluster should be online throughout.

D. Give cluster tasks and shard recoveries time to settle before pressing on.

E. Don’t forget to upgrade Kibana in the same way.

oc -n elastic patch kibana kibana --type=merge -p '{"spec":{"version":"7.17.29"}}'

Once complete, check your Elasticsearch version and Kibana version, as well as the health state:

oc -n elastic get elasticsearch es-logs
oc -n elastic get kibana kibana

6. Operator upgrade: ECK 2.16.1 → 3.3.1

ECK upgrades are pleasantly boring:

helm upgrade elastic-operator elastic/eck-operator -n elastic-system --version 3.3.1

Watch the operator pod roll. Your Elasticsearch cluster keeps running; only the controller restarts.

Verify that the upgrade is successful by looking at the operator logs and ensuring that no major errors appear:

oc logs -n elastic-system sts/elastic-operator

And then verifying the new version of the operator (will now be 3.3.1):

helm -n elastic-system list

7. Your roadmap to 8.x and 9.x (when you’re ready)

You’re now on:

• ECK Operator: 3.3.1
• Elastic Stack: 7.17.29

That pair is fully supported and serves as the official launchpad for 8.x. It’s important to first go through the Elastic upgrade documentation.

We’ll again go through the procedure of checking for any hard-breaking changes between our 7.17.29 and the latest 8 version (8.19.9):

GET _migration/deprecations?pretty

It's important to look through the result of this query carefully and to go through necessary steps, like re-indexing indices and changing mappings, among others.

Once you’ve addressed all required changes from 7.17.29 to 8.x:

oc -n elastic patch elasticsearch es-logs --type=merge -p '{"spec":{"version":"8.19.9"}}'
oc -n elastic patch kibana kibana --type=merge -p '{"spec":{"version":"8.19.9"}}'

ECK will handle the rest. Just remember to upgrade Beats, Logstash pipelines, and client libraries in lockstep to avoid wire-protocol surprises.

Repeat the process again to migrate to the latest 9.x version.

8. Cleanup

• Remove the Red Hat Elasticsearch operator.

Now that you’re no longer using the Red Hat Elasticsearch operator, you can remove it from your cluster. You can do that via the following steps:

A. In the OpenShift Console, go to Operators and then to Installed Operators.

B. In the Filter By Name field, enter “Elasticsearch” to find the installed Red Hat Elasticsearch operator.

C. On the Operator Details page, select Uninstall Operator from the Actions list.

D. On the Uninstall Operator? dialog box, select Uninstall. This removes the operator, the operator deployments, and the pods. After this step, the operator stops running and will no longer receive updates.

All of these steps can be found in this link from Red Hat OpenShift documentation.

Wrapping up

By installing ECK 2.16.1 as a bridge, snapshot-restoring into a new cluster, and stepping cleanly through 7.x before landing on ECK 3.3, you’ve transformed an aging, unsupported logging back end into a modern, secure, first-class Elastic deployment, without surprises or downtime.

EIS provides managed, GPU-accelerated inference tightly integrated with Elasticsearch. With EIS, you don’t need to host, scale, or maintain infrastructure for embedding models.

Semantic search retrieves results based on meaning. Text is converted into vector embeddings so queries can match related concepts, even when the exact words differ.

The semantic_text field type simplifies this entire workflow, with automatic chunking, embedding generation at index time, and seamless querying via the semantic query, without building custom pipelines or managing separate model inference.

The jina-embeddings-v5-text model family just launched on EIS, giving developers powerful multilingual embeddings accessible as part of the core semantic_text workflow. So now your semantic search works across languages out of the box, and global datasets, such as support articles, product descriptions, user reviews, and multilingual websites, work without extra configuration.

This default opens up broad, globe-spanning semantic retrieval with no operational overhead.

jina-embeddings-v5-text

The jina-embeddings-v5-text models represent the latest generation of compact, high-performance multilingual embedding models on EIS.

• State-of-the-art multilingual performance: Top scores on MMTEB benchmarks across hundreds of languages. jina-embeddings-v5-text-nano leads models under 500M parameters, and jina-embeddings-v5-text-small outperforms significantly larger alternatives.
• Multiple task capabilities: Spanning across retrieval, semantic matching, clustering, and classification.
• Flexible choices to fit your use case: Two model sizes (small, nano) let you balance speed, cost, and quality.
• Long-context support: Embed long texts efficiently, ideal for document collections with extended context.

Get started

1. Create index

Define a semantic_text field with no additional configuration. Embeddings will be generated automatically at index time using the default model. For production workloads, explicitly specify the model to ensure consistent behavior and results.

PUT /multilingual-reviews
{
  "mappings": {
    "properties": {
      "product": { "type": "keyword" },
      "review": { "type": "semantic_text" },
      "language": { "type": "keyword" }
    }
  }
}

2. Index multilingual documents

Add product reviews in six different languages. Each document’s review field is automatically embedded at ingest time, with no separate pipeline or preprocessing needed.

POST /multilingual-reviews/_bulk?refresh=wait_for
{ "index": { "_id": "1" } }
{ "product": "wireless-headphones", "review": "Amazing noise cancellation and the battery lasts all day. Perfect for long flights.", "language": "en" }
{ "index": { "_id": "2" } }
{ "product": "wireless-headphones", "review": "La cancelación de ruido es impresionante. Muy cómodos incluso después de horas de uso.", "language": "es" }
{ "index": { "_id": "3" } }
{ "product": "wireless-headphones", "review": "ノイズキャンセリングが素晴らしく、長時間つけていても耳が痛くなりません。", "language": "ja" }
{ "index": { "_id": "4" } }
{ "product": "wireless-headphones", "review": "Réduction de bruit excellente et très confortable pour les longs trajets en avion.", "language": "fr" }
{ "index": { "_id": "5" } }
{ "product": "wireless-headphones", "review": "Hervorragende Geräuschunterdrückung. Ideal für Pendler und Vielflieger.", "language": "de" }
{ "index": { "_id": "6" } }
{ "product": "wireless-headphones", "review": "O cancelamento de ruído é excelente e a bateria dura o dia todo.", "language": "pt" }

3. Search across languages with a query in English

GET /multilingual-reviews/_search
{
  "query": {
    "match": {
      "review": "comfortable for long flights"
    }
  }
}

The results show all six reviews ranked by semantic relevance to the English query:

{
  "took": 83,
  "timed_out": false,
  "_shards": {
    "total": 6,
    "successful": 6,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 6,
      "relation": "eq"
    },
    "max_score": 0.8275735,
    "hits": [
      {
        "_index": "multilingual-reviews",
        "_id": "4",
        "_score": 0.8275735,
        "_source": {
          "product": "wireless-headphones",
          "review": "Réduction de bruit excellente et très confortable pour les longs trajets en avion.",
          "language": "fr"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "1",
        "_score": 0.7616198,
        "_source": {
          "product": "wireless-headphones",
          "review": "Amazing noise cancellation and the battery lasts all day. Perfect for long flights.",
          "language": "en"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "5",
        "_score": 0.72122526,
        "_source": {
          "product": "wireless-headphones",
          "review": "Hervorragende Geräuschunterdrückung. Ideal für Pendler und Vielflieger.",
          "language": "de"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "2",
        "_score": 0.6867013,
        "_source": {
          "product": "wireless-headphones",
          "review": "La cancelación de ruido es impresionante. Muy cómodos incluso después de horas de uso.",
          "language": "es"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "3",
        "_score": 0.66513836,
        "_source": {
          "product": "wireless-headphones",
          "review": "ノイズキャンセリングが素晴らしく、長時間つけていても耳が痛くなりません。",
          "language": "ja"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "6",
        "_score": 0.61658823,
        "_source": {
          "product": "wireless-headphones",
          "review": "O cancelamento de ruído é excelente e a bateria dura o dia todo.",
          "language": "pt"
        }
      }
    ]
  }
}

Notice that the French review ranks first, even above the English one. That's because "très confortable pour les longs trajets en avion" ("very comfortable for long trips by plane") is a closer semantic match to the query than the English review, which splits its focus across noise cancellation, battery life, and flights. This demonstrates the jina-embeddings-v5-text-small ability to rank by meaning, not language.

4. Search across languages with a Japanese query

GET /multilingual-reviews/_search
{
  "query": {
    "match": {
      "review": "長時間のフライトに最適"
    }
  }
}

The results show all six reviews ranked by semantic relevance to the Japanese query (“Ideal for long-haul flights”):

{
  "took": 89,
  "timed_out": false,
  "_shards": {
    "total": 6,
    "successful": 6,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 6,
      "relation": "eq"
    },
    "max_score": 0.7556782,
    "hits": [
      {
        "_index": "multilingual-reviews",
        "_id": "4",
        "_score": 0.7556782,
        "_source": {
          "product": "wireless-headphones",
          "review": "Réduction de bruit excellente et très confortable pour les longs trajets en avion.",
          "language": "fr"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "1",
        "_score": 0.7395687,
        "_source": {
          "product": "wireless-headphones",
          "review": "Amazing noise cancellation and the battery lasts all day. Perfect for long flights.",
          "language": "en"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "5",
        "_score": 0.68835545,
        "_source": {
          "product": "wireless-headphones",
          "review": "Hervorragende Geräuschunterdrückung. Ideal für Pendler und Vielflieger.",
          "language": "de"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "3",
        "_score": 0.6487931,
        "_source": {
          "product": "wireless-headphones",
          "review": "ノイズキャンセリングが素晴らしく、長時間つけていても耳が痛くなりません。",
          "language": "ja"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "6",
        "_score": 0.6241487,
        "_source": {
          "product": "wireless-headphones",
          "review": "O cancelamento de ruído é excelente e a bateria dura o dia todo.",
          "language": "pt"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "2",
        "_score": 0.6183049,
        "_source": {
          "product": "wireless-headphones",
          "review": "La cancelación de ruido es impresionante. Muy cómodos incluso después de horas de uso.",
          "language": "es"
        }
      }
    ]
  }
}

The ranking is nearly identical to the English query: French and English still lead because they're the most semantically relevant to "perfect for long flights," regardless of query language. The Japanese review didn't get artificially boosted just because the query was in Japanese. It ranks fourth because it discusses wearing comfort, not flights. Semantic relevance takes priority over language matching.

Note: For English-only use cases

If you prefer a sparse representation or would like to continue to use Elastic Learned Sparse EncodeR (ELSER) for English workloads, ELSER remains available and fully supported as an option for semantic_text.

You can explicitly choose ELSER by specifying inference_id: ".elser-2-elastic in your mappings when creating an index.

Conclusion: Semantic search without borders

With semantic_text now defaulting to the jina-embeddings-v5-text family on Elastic Inference Service, multilingual semantic search becomes the standard developer experience in Elasticsearch. This means developers can build search, retrieval augmented generation (RAG), and AI applications that work across global datasets without stitching pipelines together.

Create a semantic_text field, index your data, and start searching. All Elastic Cloud trials have access to Elastic Inference Service. Try it now on Elastic Cloud Serverless or Elastic Cloud Hosted, or use EIS via Cloud Connect with your self-managed cluster.

The problem

Imagine you have two indices, index-a (source) and index-b (target), and you want to find all documents that exist in index-a but are missing from index-b.

A naive approach, querying both indices and comparing results in memory, won't scale. Elasticsearch is designed to handle millions of documents, and loading them all at once isn’t practical.

There are two scenarios:

1. IDs are stable: Both indices use the same _id for the same document (for example, emp_no as the document ID). This is the easy case.
2. IDs are generated: Documents were ingested through different pipelines that assigned random or sequential IDs. You can't compare by _id; you need to match on content.

Let's walk through both.

Step 0 — A lighter CLI for Elasticsearch

All the examples in this post use escli, a small Rust command line interface (CLI) that wraps the Elasticsearch REST API. It reads your cluster URL and credentials from environment variables, so you don’t have to repeat authentication headers on every command.

To see why that matters, here's a typical _search call with raw curl:

curl -X GET \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":{"term":{"user.id":"kimchy"}}}' \
  "$ELASTICSEARCH_URL/my-index-000001/_search"

With escli, the same request becomes:

./escli search --index my-index-000001 <<< '{"query":{"term":{"user.id":"kimchy"}}}'

The credentials live in a .env file that escli sources automatically — no -H "Authorization: ..." on every call, no risk of leaking secrets in shell history. The request body is passed via stdin (<<<), which makes it easy to pipe in multi-line JSON built dynamically with jq.

Step 1 — Count documents in both indices

Before doing a full scan, get a quick count of each index. If the counts match, the indices are likely in sync, and there’s no need to scan at all.

./escli count --index index-a
./escli count --index index-b

The _count API returns:

{ "count": 1000000 }

If the counts differ, proceed to the full comparison.

Step 2 — When IDs mean something: Use op_type=create

If both indices use the same _id for the same document, for example, because you indexed documents using a functional business key like emp_no rather than a generated UUID, you can find and fix missing documents in a single _reindex call.

Why functional IDs matter

Using a meaningful field as _id (instead of a random UUID) is a best practice when the data has a natural key. It means:

• The same document always gets the same _id, regardless of which pipeline ingested it.
• You can easily update or delete documents by ID.
• You can use op_type=create to skip documents that already exist in the target.
• No client-side scanning or comparison is needed.

The op_type=create trick

_reindex with op_type=create tries to create each document from the source in the target. If a document with the same _id already exists, Elasticsearch reports it as a version_conflict and moves on. It doesn’t overwrite the existing document. Setting conflicts=proceed tells the API to continue instead of aborting on the first conflict.

./escli reindex <<< '{
  "source": { "index": "index-a" },
  "dest":   { "index": "index-b", "op_type": "create" },
  "conflicts": "proceed"
}'

The response tells you exactly what happened:

{
  "total": 1000000,
  "created": 49594,
  "version_conflicts": 950406,
  "failures": []
}

• created: Documents that were missing from index-b and have now been added.
• version_conflicts: Documents that already existed in index-b and were left untouched.

No scanning, no client-side comparison, no intermediate file. Everything happens server-side in about six seconds on a 1M-document dataset.

Step 3 — When IDs are not stable: Business-key comparison

Sometimes you can't rely on _id. A document pipeline that generates IDs at ingestion time will assign a different _id each time the same record is processed. If index-a and index-b were populated by two such pipelines, the same employee record might have _id: "abc123" in one index and _id: "xyz789" in the other, even though the underlying data is identical.

In this case, you need to match documents by content rather than by ID. The key is to identify a set of fields that together form a unique business key.

For an employee dataset, a reasonable business key is (first_name, last_name, birth_date). A document in index-a is "missing" from index-b if no document in index-b has the same combination of those three fields.

3a — Scan the source with PIT + search_after

Open a point in time (PIT) on the source index to get a consistent snapshot, and then paginate through it, fetching only the business-key fields:

./escli open_point_in_time index-a 5m
# → { "id": "46ToAwMDaWR..." }

./escli search <<< '{
  "size": 10000,
  "_source": ["first_name", "last_name", "birth_date"],
  "pit": { "id": "46ToAwMDaWR...", "keep_alive": "5m" },
  "sort": [{ "_shard_doc": "asc" }]
}'

The sort key _shard_doc is the most efficient sort for full-index pagination: it uses the internal Lucene document order with no overhead. Repeat with search_after until the response contains zero hits. Always close the PIT when done:

./escli close_point_in_time <<< '{"id": "46ToAwMDaWR..."}'

3b — Check each page against the target via _msearch

For each page of source documents, build one _msearch request with one subquery per document. Each subquery uses a bool/must on the three business-key fields and requests size: 0; we only need to know whether a match exists, we don’t need to retrieve the document itself.

./escli msearch << 'EOF'
{"index": "index-b"}
{"size":0,"query":{"bool":{"must":[{"term":{"first_name.keyword":"Alice1"}},{"term":{"last_name.keyword":"Smith"}},{"term":{"birth_date":"1985-03-12"}}]}}}
{"index": "index-b"}
{"size":0,"query":{"bool":{"must":[{"term":{"first_name.keyword":"Bob2"}},{"term":{"last_name.keyword":"Jones"}},{"term":{"birth_date":"1990-07-24"}}]}}}
EOF

The response contains one entry per subquery, in the same order:

{
  "responses": [
    { "hits": { "total": { "value": 1 } } },
    { "hits": { "total": { "value": 0 } } }
  ]
}

total.value == 0 means no document in index-b matches that business key; the document is missing. Collect the corresponding _id from the source page.

Note on .keyword subfields: term queries require exact (keyword) matching. The first_name and last_name fields must have a .keyword subfield in the index mapping. The demo's mapping.json includes this.

3c — Speed it up with split-by-date

If the business key includes a date field, you can partition the source into date slices and run each slice as an independent job. Each slice opens its own PIT with a range filter on birth_date, runs its own msearch loop, and writes its results to a separate file. The parent script launches all slices in parallel and aggregates the results when they’re all done.

But depending on your use case, you might want to partition by a different field; for example, if you have a team field, you could run one slice per team. The key is to find a field that allows you to split the data into reasonably even chunks that can be processed in parallel.

[compare] Launching 5 slices in parallel...

  → Slice 1: 1960-01-01 → 1969-12-31 ✅ — 244408 checked, 12207 missing
  → Slice 2: 1970-01-01 → 1979-12-31 ✅ — 243624 checked, 12212 missing
  → Slice 3: 1980-01-01 → 1989-12-31 ✅ — 243551 checked, 11921 missing
  → Slice 4: 1990-01-01 → 1999-12-31 ✅ — 243895 checked, 11991 missing
  → Slice 5: 2000-01-01 → 2009-12-31 ✅ — 24522 checked, 1263 missing

Performance on a 1M dataset

To validate the approaches, the demo generates 1,000,000 documents in index-a and deliberately skips ~5% in index-b (49,594 missing documents), and then runs the full compare → reindex cycle.

Results on a MacBook M3 Pro:

Comparison (compare-indices.sh):

The op_type=create approach is fastest because everything is server-side and requires no client-side scanning. The split-by-date strategy cuts the business-key duration from 1m 38s down to 36s through parallelism: not bad for a comparison across two 1M-document indices.

Decision tree

Are _id values stable between both indices?
├── Yes → _reindex with op_type=create          (6s, server-side)
└── No  → Do you have a reliable business key?
          ├── Yes, simple scan is fast enough → business-key   (1m 42s)
          └── Yes, and you need more speed    → split-by-date  (36s, parallel)

Conclusion

Elasticsearch doesn't offer a native index diff command, but the right strategy depends on your data model:

• Use functional _ids (a natural business key like emp_no) whenever possible. It unlocks the simplest and fastest approach: _reindex with op_type=create finds and fills gaps in one server-side call.
• When IDs are unstable, match by business key using PIT + _msearch. Partition by a field and run slices in parallel to recover most of the performance. If you find yourself doing this regularly, consider computing a hash of your business key fields and using it as _id at ingestion time. You get the best of both worlds: stable IDs and efficient lookups.

The complete demo, including dataset generation, comparison scripts, and reindex scripts, is available at https://github.com/dadoonet/blog-compare-indices/.

Elastic Workflows is a built-in automation engine inside Kibana that lets you define multistep processes using a simple YAML configuration. Each workflow can be triggered on a schedule or event or as a tool in Elastic Agent Builder, and each step can call Kibana APIs, query Elasticsearch, or transform data.

We’ll use dashboard view counts as a concrete example, but the same pattern applies to any metric exposed through the Kibana saved objects API.

Prerequisites

• Elastic Cloud or self-managed cluster running 9.3
• Workflows enabled (Advanced settings)

Step 1: Explore the raw data in Dev Tools

Before building anything, let's understand what data we have. Kibana stores most of its configuration and metadata as saved objects in a dedicated internal index. One of the things Kibana tracks this way is dashboard view counts, using a special saved object type called usage counters. You can query them directly from Dev Tools:

GET kbn:/api/saved_objects/_find?type=usage-counter&filter=usage-counter.attributes.domainId:"dashboard"%20and%20usage-counter.attributes.counterType:"viewed"&per_page=10000

The response looks like this:

{
  "page": 1,
  "per_page": 10000,
  "total": 1,
  "saved_objects": [
    {
      "type": "usage-counter",
      "id": "dashboard:346f3c64-ebca-484d-9d57-ec600067d596:viewed:server:20260310",
      "attributes": {
        "domainId": "dashboard",
        "counterName": "346f3c64-ebca-484d-9d57-ec600067d596",
        "counterType": "viewed",
        "source": "server",
        "count": 1
      },
      ...
    }
  ]

The counterName field is the dashboard ID, and count is the cumulative view count for that dashboard on that specific day. Kibana creates one counter object per dashboard per day; you can see the date suffix in the object ID (...viewed:server:20260310). The count grows throughout the day as users open the dashboard.

Rather than replicating this daily-document model in our index, we’ll create one document per workflow execution. Each document records how many views that dashboard had accumulated for the day at the moment of capture.

Step 2: Create the destination index

We need an index to store our dashboard view snapshots. The following command creates it with explicit mappings so we can aggregate and visualize later. Run this in Dev Tools:

PUT dashboard-views
{
  "mappings": {
    "properties": {
      "captured_at": {
        "type": "date"
      },
      "dashboard_id": {
        "type": "keyword"
      },
      "dashboard_name": {
        "type": "keyword"
      },
      "view_count": {
        "type": "integer"
      }
    }
  }
}

Using keyword mappings for IDs and names allows aggregations. Using integer for view_count is a safe default, since Kibana resets the counter daily, reaching the 32-bit limit (more than 2 billion views in a single day) isn’t a realistic concern. It still supports numeric operations, like max, avg, and min among others.

Step 3: Create the workflow

Go to Stack Management > Workflows > New Workflow, and paste the following workflow YAML configuration:

name: dashboard-views-ingestion
triggers:
  - type: scheduled
    with:
      every: 30m

steps:
  - name: fetch_dashboard_views
    type: kibana.request
    with:
      method: GET
      path: >-
        /api/saved_objects/_find?type=usage-counter&per_page=10000&filter=usage-counter.attributes.domainId:"dashboard"%20and%20usage-counter.attributes.counterType:"viewed"

  - name: index_each_dashboard
    type: foreach
    foreach: "{{ steps.fetch_dashboard_views.output.saved_objects }}"
    steps:
      - name: fetch_dashboard_name
        type: kibana.request
        with:
          method: GET
          path: /api/saved_objects/dashboard/{{ foreach.item.attributes.counterName }}
        on-failure:
          continue: true

      - name: index_doc
        type: elasticsearch.request
        with:
          method: POST
          path: /dashboard-views/_doc
          body:
            dashboard_id: "{{ foreach.item.attributes.counterName }}"
            dashboard_name: "{{ steps.fetch_dashboard_name.output.attributes.title }}"
            view_count: "${{ foreach.item.attributes.count | plus: 0 }}"
            captured_at: "{{ execution.startedAt | date: '%Y-%m-%dT%H:%M:%SZ' }}"

In the next section, let's break down the workflow step by step.

How the workflow works

Triggers

The workflow runs on a scheduled trigger every 30 minutes. This gives us time-series data without hammering the API.

fetch_dashboard_views

Uses kibana.request to call the Kibana saved objects API. No authentication setup is needed: The workflow engine automatically attaches the correct headers based on the execution context.

index_each_dashboard (foreach)

Iterates over the saved_objects array returned by the previous step. The current item in each iteration is available as foreach.item. Inside the loop, we run two nested steps for each dashboard.

1. fetch_dashboard_name:

Resolves the human-readable dashboard title by calling GET /api/saved_objects/dashboard/{id}. We add on-failure: continue: true so that if a dashboard was deleted but still has view counters, the loop continues instead of failing the whole execution.

2. index_doc:

Indexes each document using POST /dashboard-views/_doc (without an explicit ID), which lets Elasticsearch auto-generate IDs. This creates a new document on every run, building a history of view counts over time rather than overwriting the previous snapshot.

Two things worth noting:

• The captured_at field uses the date filter to format the timestamp as ISO 8601. Without it, the value comes out as a JavaScript date string, like Tue Mar 10 2026 05:03:47 GMT+0000, which Elasticsearch won't map as a date.
• The view_count uses ${{ }} syntax with | plus: 0 to preserve the numeric type. Using {{ }} would render it as a string, which would prevent math operations in the dashboard.

The UI allows you to nicely debug each of the workflow steps.

Step 4: Build the stats dashboard

Once the workflow has run a few times and data is collected, create a new dashboard in Kibana using the dashboard-views data view.

Some panels to start with:

• Top dashboards by views: Use a Bar chart with dashboard_name on the X axis and last_value(view_count) on the Y axis. This shows the current daily view count per dashboard.
• Views over time: Use a Line chart with captured_at on the X axis and last_value(view_count) on the Y axis, broken down by dashboard_name. Since each run appends a new document, use last value to get the peak count per time bucket rather than summing duplicates.
• Current snapshot: Use a Data table with the latest captured_at to show the most recent view counts across all dashboards.

Since each workflow creates a new document, you can filter by time range to analyze activity in specific periods, compare week over week, or build alerts when a dashboard drops below a view threshold.

Conclusion

Elastic Workflows is a good fit for this kind of periodic data collection because both the source (Kibana API) and the destination (Elasticsearch) are native, which means zero credential management. The workflow engine handles authentication automatically for kibana.request and elasticsearch.request steps, so the only thing you write is the logic.

Resources

• Elastic Workflows
• Kibana API

Elasticsearch was rejecting late-arriving metrics. Those rejections caused the pipeline to fall behind, resulting in more late data, which triggered even more rejections. Eventually, the pipeline stalled completely.

We had to restore from snapshot, reindex the data, and redesign the ingestion pipeline to recover.

The root cause wasn't index lifecycle management (ILM) itself. It was time series data streams (TSDS) and how they enforce time‑bound backing indices.

TSDS can reduce storage requirements for metrics by 40–70%, but the architectural changes that make TSDS efficient also alter how indices behave over time. Those changes matter when designing ILM policies or when your ingestion pipelines may produce late‑arriving data.

TL;DR

When using TSDS:

• Backing indices only accept documents within a specific time window.
• If late data arrives after an index moves to cold or frozen, Elasticsearch rejects those documents or routes them to the failure store, if configured.

Design rule:

warm_min_age > rollover_max_age + maximum_expected_lateness

What is a time series data stream?

A time series data stream (TSDS) is a specialized data stream optimized for metrics data. Data is routed so that related documents are located within the same shards, optimizing them for query and retrieval. Here’s how Elasticsearch does it:

Each document contains:

• A timestamp.
• Dimension fields identifying the time series.
• Metric fields representing measured values.

Examples include:

• CPU usage per host.
• Request latency per service.
• Temperature readings per sensor.

Dimensions identify what we want to measure, while metrics represent values that change over time.

Dimensions

Dimensions describe the measured entity.

Examples:

host.name
service.name
container.id

We define them in mappings with:

time_series_dimension: true

Metrics

Metrics represent numeric values and are defined using:

time_series_metric

Common metric types:

• Gauge: Values that rise and fall.
• Counter: Values that increase until reset.

Elastic Agent primarily collects metrics and logs data, so even if you haven’t enabled any TSDS indices by hand, you may still have them in your cluster.

The _tsid field

Elasticsearch internally generates a _tsid value from dimension fields. This allows documents with identical dimensions to be routed to the same shard, improving:

• Compression.
• Query locality.
• Aggregation performance.

The key difference: Time‑bound backing indices

Traditional data streams always write to the most recent backing index, called the write index, but TSDS behaves differently.

Each TSDS backing index has a defined time window and only accepts documents with @timestamp values that fall in that window:

GET _data_stream/my-metrics-data-stream
{
  "index_mode": "time_series",
  "time_series": {
    "temporal_ranges": [
      {
        "start": "2026-01-15T14:35:50.000Z",
        "end": "2026-03-16T11:34:40.000Z"
      }
    ]
  }
}

When a document is indexed, Elasticsearch routes it to the backing index responsible for that timestamp, meaning that, unlike traditional indices, a TSDS may write to multiple backing indices simultaneously.

For example:

• Real‑time data → newest index.
• Late data → earlier index covering that time range.

Designing for late‑arriving data

Real ingestion pipelines rarely deliver metrics perfectly on time. Metrics can be delayed by network outages, backlogs along the way, batch ingestion, and loss of edge devices, which reconnect and start to catch up.

Traditional indices quietly absorb those delays. TSDS does not.

If a document's timestamp falls outside the range of writable backing indices, Elasticsearch rejects it, meaning your ILM policy must account for late data.

The critical constraint

Backing indices must remain writable long enough to accept delayed data.

In practical terms:

time_until_readonly > maximum_expected_lateness

Because ILM measures ages from rollover, the operational rule becomes:

warm_or_cold_min_age > rollover_max_age + maximum_expected_lateness

For example, if metrics may arrive up to six hours late, indices must remain writable at least six hours after rollover.

Failing to account for this constraint was exactly what caused the ingestion failure described earlier. Late-arriving data was directed to an earlier index, which was already in the cold tier and therefore write-blocked.

Handling rejected documents

When TSDS rejects a document, Elasticsearch returns an error, indicating that the timestamp doesn’t fall within the range of writable indices. How your ingestion pipeline handles that error determines whether you lose data or stall ingestion.

The primary mechanism for handling rejected documents is the failure store.

Failure store (recommended in Elasticsearch 9.1+)

Elasticsearch 9.1 introduced the failure store, which automatically captures rejected documents. Instead of returning errors to clients, Elasticsearch writes failed documents to a dedicated failure index inside the data stream.

You can inspect failures using:

GET metrics-myapp::failures/_search

Using the failure store prevents ingestion pipelines from choking on rejection errors while preserving failed data for analysis or reindexing.

Monitoring for rejection issues

Late‑arrival problems usually appear first as ingestion anomalies. You may notice them first as:

• Sudden drops in indexing rate.
• Spikes in rejected documents.
• A growing number of failure store entries.
• Mismatches between pipeline input and output counts.

Alerting on these signals allows operators to detect issues before pipelines stall. Workflows, machine learning jobs, and other mechanisms can be used to automate detection and notification.

Migration checklist for TSDS + ILM

If you're migrating a metrics cluster to TSDS, introducing ILM tiering, or upgrading to an Elasticsearch version where metrics are TSDS by default, review these items first.

1. Measure ingestion latency

Before changing ILM policies, determine:

• Normal ingestion delay.
• Worst-case delay during incidents.
• Delays caused by batch pipelines.

Your ILM design must accommodate the maximum realistic delay.

2. Verify index time windows

Inspect your TSDS backing indices:

GET _data_stream/

Look for:

• time_series.start_time
• time_series.end_time

These bounds determine which indices can accept documents. Understanding these windows can help you determine how late data can be before it’s rejected.

3. Size the hot tier for late arrivals

Ensure backing indices remain writable long enough for delayed data.

Operational rule:

• warm_min_age > rollover_max_age + maximum_expected_lateness

Remember, indices must remain writable for at least six hours if metrics may arrive six hours late.

4. Decide how to handle rejected documents

Choose a strategy before enabling TSDS:

• Failure store (recommended in Elasticsearch 9.1+).
• Logstash dead letter queue.
• Fallback index for late arrivals.
• Accepting limited data loss.

5. Monitor ingestion health

Add alerts for:

• Indexing rate drops.
• Rejected documents.
• Failure store growth.
• Pipeline input/output mismatches.

Late data issues often appear first as ingestion anomalies.

Summary

Time series data streams provide major storage and performance improvements for metrics workloads, but they introduce an important architectural change: Backing indices are time‑bound, which affects how ILM behaves.

When using TSDS:

• Indices must remain writable long enough to accept delayed data.
• Ingestion pipelines should handle rejected documents safely.

The key rule to remember is:

warm_min_age > rollover_max_age + maximum_expected_lateness

If you design ILM policies around that constraint, TSDS works extremely well for metrics workloads.

Ignore it, though, and your ingestion pipeline may discover those time boundaries the hard way.

Your first query

Start by defining a plain old CLR object (POCO) that maps to your Elasticsearch index. Property names are resolved to ES|QL column names through standard System.Text.Json attributes, like [JsonPropertyName], or through a configured JsonNamingPolicy. The same source serialization rules that apply across the rest of the client apply here as well.

using System.Text.Json.Serialization;

public class Product
{
    [JsonPropertyName("product_id")]
    public string Id { get; set; }

    public string Name { get; set; }

    public string Brand { get; set; }

    [JsonPropertyName("price_usd")]
    public double Price { get; set; }

    [JsonPropertyName("in_stock")]
    public bool InStock { get; set; }
}

With the type in place, a query looks like this:

var minPrice = 100.0;
var brand = "TechCorp";

await foreach (var product in client.Esql.QueryAsync(q => q
    .From("products")
    .Where(p => p.InStock && p.Price >= minPrice && p.Brand == brand)
    .OrderByDescending(p => p.Price)
    .Take(10)))
{
    Console.WriteLine($"{product.Name}: ${product.Price}");
}

The provider translates this into the following ES|QL:

FROM products
| WHERE (in_stock == true AND price_usd >= ?minPrice AND brand == ?brand)
| SORT price_usd DESC
| LIMIT 10

A few details to note:

• Property name resolution: p.Price becomes price_usd because of the [JsonPropertyName] attribute, and p.Brand becomes brand following the default camelCase naming policy.
• Parameter capturing: The C# variables minPrice and brand are captured as named parameters (?minPrice, ?brand). They’re sent separately from the query string in the JSON payload, which prevents injection and enables server-side query plan caching.
• Streaming: QueryAsync<T> returns IAsyncEnumerable<T>. Rows are materialized one at a time as they arrive from Elasticsearch.

You can also inspect the generated query and its parameters without executing it:

var query = client.Esql.CreateQuery()
    .Where(p => p.InStock && p.Price >= minPrice && p.Brand == brand)
    .OrderByDescending(p => p.Price)
    .Take(10);

Console.WriteLine(query.ToEsqlString());
// FROM products | WHERE (in_stock == true AND price_usd >= 100) | SORT price_usd DESC | LIMIT 10

Console.WriteLine(query.ToEsqlString(inlineParameters: false));
// FROM products | WHERE (in_stock == true AND price_usd >= ?minPrice AND brand == ?brand) | SORT price_usd DESC | LIMIT 10

var parameters = query.GetParameters();
// { "minPrice": 100.0, "brand": "TechCorp" }

How does this work? A quick LINQ refresher

The mechanism that makes LINQ providers possible is the distinction between IEnumerable<T> and IQueryable<T>.

When you call .Where(p => p.Price > 100) on an IEnumerable<T>, the lambda compiles to a Func<Product, bool>, a regular delegate that the runtime executes in-process. This is LINQ-to-Objects.

When you call the same method on an IQueryable<T>, the C# compiler wraps the lambda in an Expression<Func<Product, bool>> instead. This is a data structure that represents the structure of the code rather than its executable form. The expression tree can be inspected, analyzed, and translated into another language at runtime.

// IEnumerable: the lambda is a compiled delegate
IEnumerable local = products.Where(p => p.Price > 100);

// IQueryable: the lambda is an expression tree, a data structure
IQueryable remote = queryable.Where(p => p.Price > 100);

The IQueryProvider interface is the extension point. Any provider can implement CreateQuery<T> and Execute<T> to translate these expression trees into a target language. Entity Framework uses this to emit SQL. The LINQ to ES|QL provider uses it to emit ES|QL.

The expression tree for the query above looks like this:

Expression tree for the example query.

The tree is nested inside out: Take wraps OrderByDescending, which wraps Where, which wraps From, which wraps the root EsqlQueryable<Product> constant. The Where predicate is itself a subtree of BinaryExpression nodes for the &&, >=, and == operators, with MemberExpression leaves for property accesses and closure captures for the minPrice and brand variables. This is the data structure that the provider walks to produce the final ES|QL.

Under the hood: The translation pipeline

The path from a LINQ expression to query results follows a six-stage pipeline:

Translation pipeline overview.

1. Expression tree capture

When you chain .Where(), .OrderBy(), .Take() and other operators on an IQueryable<T>, the standard LINQ infrastructure builds an expression tree. EsqlQueryable<T> implements IQueryable<T> and delegates to EsqlQueryProvider.

2. Translation

When the query is executed (by enumerating, calling ToList(), or using await foreach), the EsqlExpressionVisitor walks the expression tree inside out. It dispatches each LINQ method call to a specialized visitor:

During translation, C# variables referenced in expressions are captured as named parameters.

3. Query model

The visitors don’t produce strings directly. Instead, they produce QueryCommand objects, an immutable intermediate representation. A FromCommand, a WhereCommand, a SortCommand, and a LimitCommand, each representing one ES|QL processing command. These are collected into an EsqlQuery model.

Query model and command pattern.

This intermediate model is decoupled from both the expression tree and the output format. It can be inspected, intercepted (via IEsqlQueryInterceptor), or modified before formatting.

4. Formatting

EsqlFormatter visits each QueryCommand in order and produces the final ES|QL string. Each command becomes one line, separated by the pipe (|) operator that ES|QL uses to chain processing commands. Identifiers containing special characters are automatically escaped with backticks.

5. Execution

The formatted ES|QL string and captured parameters are sent to Elasticsearch’s /_query endpoint as a JSON payload. The IEsqlQueryExecutor interface abstracts the transport layer, which is where the layered package architecture comes into play.

6. Materialization

EsqlResponseReader streams the JSON response without buffering the entire result set into memory. A ColumnLayout tree, precomputed once per query, maps flat ES|QL column names (like address.street, address.city) to nested POCO properties. Each row is assembled into a T instance and yielded one at a time via IEnumerable<T> or IAsyncEnumerable<T>.

The layered architecture

The LINQ to ES|QL functionality is split across three packages:

Package architecture.
Elastic.Esql is the pure translation engine. It has zero HTTP dependencies and contains the expression visitors, query model, formatter, and response reader. You can use it stand alone to build and inspect ES|QL queries without an Elasticsearch connection, which is useful for testing, query logging, or building your own execution layer.

// Translation-only: no Elasticsearch connection needed
var provider = new EsqlQueryProvider();
var query = new EsqlQueryable(provider)
    .From("products")
    .Where(p => p.InStock)
    .OrderByDescending(p => p.Price);

Console.WriteLine(query.ToEsqlString());
// FROM products | WHERE in_stock == true | SORT price_usd DESC

Elastic.Clients.Esql is a lightweight stand-alone ES|QL client. It adds HTTP execution on top of Elastic.Esql via Elastic.Transport. If your application only needs ES|QL and none of the other Elasticsearch APIs, this is the minimal dependency option.

Elastic.Clients.Elasticsearch is the full Elasticsearch .NET client. It also builds on Elastic.Esql and exposes the LINQ provider through the client.Esql namespace. This is the recommended entry point for most applications.

Both execution-layer packages provide their own implementation of IEsqlQueryExecutor, the strategy interface that bridges translation and transport.

All three packages are compatible with Native AOT when used with a source-generated JsonSerializerContext. For the full client, see the Native AOT documentation.

Beyond the basics

The example above covered filtering, sorting, and pagination. The provider supports a broader set of operations.

Aggregations

GroupBy, combined with aggregate functions in Select, translates to ES|QL STATS ... BY:

var stats = client.Esql.Query(q => q
    .GroupBy(p => p.Brand)
    .Select(g => new
    {
        Brand = g.Key,
        Count = g.Count(),
        AvgPrice = g.Average(p => p.Price),
        MaxPrice = g.Max(p => p.Price)
    }));

// -> FROM products | STATS COUNT(*), AVG(price_usd), MAX(price_usd) BY brand

Projections

Select, with anonymous types generates EVAL, KEEP, and RENAME commands:

var query = client.Esql.CreateQuery()
    .Select(p => new { ProductName = p.Name, p.Price, p.InStock });

// -> FROM products | KEEP name, price_usd, in_stock | RENAME name AS ProductName

Rich function library

Over 80 ES|QL functions are available through the EsqlFunctions class, covering date/time, string, math, IP, pattern matching, and scoring. Standard Math.* and string.* methods are also translated:

.Where(p => p.Name.Contains("Pro"))       // -> WHERE name LIKE "*Pro*"
.Where(p => EsqlFunctions.CidrMatch(      // -> WHERE CIDR_MATCH(ip, "10.0.0.0/8")
    p.IpAddress, "10.0.0.0/8"))

LOOKUP JOIN

Cross-index lookups translate to ES|QL LOOKUP JOIN:

var enriched = client.Esql.Query(q => q
    .LookupJoin(
        "category-lookup-index",
        product => product.Id,
        category => category.CategoryId,
        (product, category) => new { product.Name, category!.CategoryLabel }));

Raw ES|QL escape hatch

For ES|QL features not yet covered by the LINQ provider, you can append raw fragments:

var results = client.Esql.Query(q => q
    .Where(p => p.InStock)
    .RawEsql("| EVAL discounted = price_usd * 0.9"));

Server-side async queries

For long-running queries, submit them for background processing on the server:

await using var asyncQuery = await client.Esql.SubmitAsyncQueryAsync(
    q => q.Where(p => p.InStock),
    asyncQueryOptions: new EsqlAsyncQueryOptions
    {
        WaitForCompletionTimeout = TimeSpan.FromSeconds(5),
        KeepAlive = TimeSpan.FromMinutes(10)
    });

await asyncQuery.WaitForCompletionAsync();
await foreach (var product in asyncQuery.AsAsyncEnumerable())
    Console.WriteLine(product.Name);

Server-side async queries are especially useful for long-running analytical queries / large dataset processing that might exceed typical timeout thresholds, or in timeout-sensitive environments with load balancers, API gateways, or proxies that enforce strict HTTP timeouts. Async queries avoid connection drops by decoupling submission from result retrieval.

Getting started

LINQ to ES|QL is available starting from:

• Elastic.Clients.Elasticsearch v9.3.4 (9.x branch)
• Elastic.Clients.Elasticsearch v8.19.18 (8.x branch)

Install from NuGet:

dotnet add package Elastic.Clients.Elasticsearch

The entry points are on client.Esql:

For the full feature reference, including query options, multifield access, nested objects, and multivalue field handling, see the LINQ to ES|QL documentation.

Conclusion

LINQ to ES|QL brings the full expressiveness of C# LINQ to Elasticsearch's ES|QL query language, letting you write strongly typed, composable queries without handcrafting query strings. With automatic parameter capturing, streaming materialization, and a layered package architecture that scales from stand-alone translation to the full Elasticsearch client, it fits naturally into .NET applications of any size. Install the latest client, point your LINQ expressions at an index, and let the provider handle the rest.

If judgment lists answer the question, “How good is my ranking?,” Learning To Rank (LTR) answers, “How do I systematically make it better?”

In this article, we take the next step: using those judgment lists to train an LTR model using XGBoost, Eland, and Elasticsearch. We’ll focus on understanding the process rather than on implementation details. For the complete code, refer to the companion notebook.

What is LTR?

LTR uses machine learning (ML) to build a ranking function for your search engine. Instead of manually tuning query weights, you provide examples of proper rankings (your judgment list) and let the model learn what makes documents relevant. In Elasticsearch, LTR works as a second-stage reranker following retrieval of documents from Elasticsearch:

• First stage: A standard query (BM25, vector, or hybrid) retrieves candidate documents quickly.
• Second stage: The LTR model reranks the top results using multiple signals it learned to combine.

For a deeper introduction, see Introducing Learning To Rank (LTR) in Elasticsearch.

The journey from judgment list to model

A judgment list tells us which documents should rank highly for a given query. But the model cannot learn directly from document IDs. It needs numerical signals that explain why certain documents are potentially relevant.

The process works like this:

1. Start with judgments. Query-document pairs with relevance grades, so you define that doc1 is a good match for “DiCaprio performance” search terms.
2. Extract features. For each query-document pair, compute numerical signals, some about the document alone (for example, popularity), and others about how the query and document interact (for example, BM25 score).
3. Train the model. The model learns which feature patterns predict high grades.
4. Deploy. Deploy the trained model to your Elasticsearch cluster.
5. Query. Use the model to rerank search results.

The key insight is that features must capture what your judgments are measuring. If your judgment list rewards popular thriller movies but your features only include text-matching scores, the model has no way to learn what makes those documents relevant.

What are features?

Features are numerical values that describe a query-document pair. In Elasticsearch, we define features using queries that return scores. There are three types:

• Query-document features measure how well a query matches a document. Eland provides the QueryFeatureExtractor utility to define these features, which computes the BM25 relevance score for each query-document pair:

QueryFeatureExtractor(
    feature_name="title_bm25",
    query={"match": {"title": "{{query}}"}}
)

This extracts the BM25 score from the title field for each document relative to the query.

• Document features are properties of the document that don’t depend on the query. You can extract these using script_score or function_score:

QueryFeatureExtractor(
    feature_name="popularity",
    query={
        "script_score": {
            "query": {"exists": {"field": "popularity"}},
            "script": {"source": "return doc['popularity'].value;"}
        }
    }
)

• Query features describe the query itself, like the number of terms. These are less common but can help the model handle different query types.

Designing your feature set

Choosing features isn’t random. Each feature should capture a signal that might explain why users prefer certain documents. Let's look at the features from the LTR notebook and understand the reasoning:

Notice the strategy here:

• Multiple signals for the same concept. We have both title_bm25 (lenient) and title_all_terms_bm25 (strict). The lenient version scores any document where at least one query term matches the title, and the strict version requires all the terms to be present. For short queries, the lenient match might be enough; whereas for longer, more specific queries, strict matching might be more important. The model can learn when to rely on each.
• Text features plus quality features. Text matching alone can return irrelevant documents that happen to contain the right words. The popularity feature lets the model boost well-known, quality content when text scores are similar.
• Coverage for different query types. Some queries target titles ("star wars"), and others target actors ("dicaprio movies"). Having features for both means that the model can handle diverse searches.

When designing your own features, ask yourself, "What signals would a human use to decide if this document is relevant?" Those are your candidate features.

Building the training dataset

Once features are defined, we extract them for every query-document pair in our judgment list. The result is a training dataset where each row contains:

• The query identifier.
• The document identifier.
• The relevance grade (from our judgment list).
• All feature values.

Here’s a simplified example:

A few things to notice:

NaN values are normal. When a query doesn’t match a field, the feature returns no score. The movie Star Wars has a high title_bm25 but no actors_bm25 because the query "star wars" doesn’t match any actor names.

Queries are grouped during training. The query_id column tells the model which documents to compare against each other. For "star wars", it learns that document 11 (grade 4) should rank above document 278427 (grade 1).

But here’s the important part: The model doesn’t memorize these specific queries. Instead, it learns general patterns, like "documents with high title_bm25 AND high popularity tend to have high grades." When presented with a new query, the model applies these learned patterns to rank the results.

Features must explain grade differences. Look at qid:1: The grade 4 document has a higher title_bm25 and higher popularity than the grade 1 document. These patterns are what the model learns.

Training the LTR model

With the training dataset prepared, we train an XGBoost model with a ranking objective. The model builds decision trees that learn patterns like:

• "If title_bm25 > 10 and popularity > 50, predict high relevance."
• "If title_bm25 is missing but actors_bm25 > 12, still predict moderate relevance."

Here's how the training process works in practice:

from xgboost import XGBRanker
from sklearn.model_selection import GroupShuffleSplit

# Create the ranker model:
ranker = XGBRanker(
    objective="rank:ndcg",
    eval_metric=["ndcg@10"],
    early_stopping_rounds=20,
)

# Shaping training and eval data in the expected format.
X = judgments_with_features[ltr_config.feature_names]
y = judgments_with_features["grade"]
groups = judgments_with_features["query_id"]

# Split the dataset in two parts respectively used for training and evaluation of the model.
group_preserving_splitter = GroupShuffleSplit(n_splits=1, train_size=0.7).split(
    X, y, groups
)
train_idx, eval_idx = next(group_preserving_splitter)

train_features, eval_features = X.loc[train_idx], X.loc[eval_idx]
train_target, eval_target = y.loc[train_idx], y.loc[eval_idx]
train_query_groups, eval_query_groups = groups.loc[train_idx], groups.loc[eval_idx]

# Training the model
ranker.fit(
    X=train_features,
    y=train_target,
    group=train_query_groups.value_counts().sort_index().values,
    eval_set=[(eval_features, eval_target)],
    eval_group=[eval_query_groups.value_counts().sort_index().values],
    verbose=True,
)

During training, the model tries different combinations of these rules and measures how well the resulting rankings match your judgment grades. It uses a metric called Normalized Discounted Cumulative Gain (NDCG) to score itself. A perfect NDCG of 1.0 means that the model's ranking exactly matches your judgments. Lower scores mean that some relevant documents are ranking below where they should be.

The training also uses a technique called early stopping. If the model's score stops improving for several rounds, training halts automatically. This prevents the model from memorizing the training data too closely, which would hurt its ability to generalize to new queries.

The companion notebook contains the complete training code.

Understanding what your LTR model learned

After training, XGBoost can show you which features the model relied on most. You can generate a feature importance chart using XGBoost's built-in visualization:

from xgboost import plot_importance

plot_importance(ranker, importance_type="weight")

The importance_type="weight" parameter shows how often each feature was used in tree splits. Here’s the resulting chart:

The F score counts how many times each feature was used to make split decisions across all trees in the model. Higher values mean that the model relied on that feature more often.

In this example:

• popularity (2178): The most important feature. The model frequently uses popularity to separate relevant from nonrelevant documents.
• title_bm25 (1642): Second-most important. Title matches matter a lot for movie searches.
• actors_bm25 (565): Moderately important. This is useful for queries that mention actors.
• title_all_terms_bm25 (211): Occasionally useful. The stricter matching helps for some queries.
• actors_all_terms_bm25 (63): Rarely used. The model found this feature less predictive.

This chart helps you iterate on your feature set. If a feature that you expected to be important shows near-zero importance, investigate why. Maybe the feature extraction is not working as intended, or maybe that signal doesn’t actually predict relevance in your judgment data.

Deploying and using the LTR model

Once trained, upload the model to Elasticsearch using Eland:

MLModel.import_ltr_model(
    es_client=es_client,
    model=ranker,
    model_id="ltr-model-xgboost",
    ltr_model_config=ltr_config,
    es_if_exists="replace",
)

Once uploaded, the model can be used as a rescorer retriever to be combined with other retrievers for multistage search pipelines:

GET movies/_search
{
  "retriever": {
    "rescorer": {
      "rescore": {
        "window_size": 50,
        "learning_to_rank": {
          "model_id": "ltr-model-xgboost",
          "params": {
            "query": "star wars"
          }
        }
      },
      "retriever": {
        "standard": {
          "query": {
            "multi_match": {
              "fields": ["title", "overview", "actors", "director", "tags", "characters"],
              "query": "star wars"
            }
          }
        }
      }
    }
  }
}

Response (simplified):

 "hits": {
    "total": {
      "value": 852,
      "relation": "eq"
    },
    "max_score": 25.165691,
    "hits": [
      {
        "_index": "movies",
        "_id": "11",
        "_score": 25.165691,
        "_source": {
          "title": "Star Wars"
        }
      },
      {
        "_index": "movies",
        "_id": "12180",
        "_score": 25.092865,
        "_source": {
          "title": "Star Wars: The Clone Wars"
        }
      },
      {
        "_index": "movies",
        "_id": "181812",
        "_score": 23.456198,
        "_source": {
          "title": "Star Wars: The Rise of Skywalker"
        }
      },
      {
        "_index": "movies",
        "_id": "140607",
        "_score": 23.320757,
        "_source": {
          "title": "Star Wars: The Force Awakens"
        }
      },
...

The first-stage query retrieves candidates using BM25. The LTR model then reranks the top 50 results using all the features it learned to weight.

For the sake of the example, the multi_match query alone would return some less relevant results on the first positions that LTR helped to fix:

{
  "hits": [
    {
      "_index": "movies",
      "_id": "11",
      "_score": 10.971989,
      "_source": {
        "title": "Star Wars"
      }
    },
    {
      "_index": "movies",
      "_id": "12180",
      "_score": 9.923633,
      "_source": {
        "title": "Star Wars: The Clone Wars"
      }
    },
    {
      "_index": "movies",
      "_id": "1022100",
      "_score": 8.9880295,
      "_source": {
        "title": "Andor: A Disney+ Day Special Look"
      }
    },
    {
      "_index": "movies",
      "_id": "278427",
      "_score": 8.845748,
      "_source": {
        "title": "Family Guy Presents: It's a Trap!"
      }
    },
    ...
  ]
}

Conclusion

The path from judgment lists to a working LTR model involves three key steps: designing features that capture relevance signals, building a training dataset that pairs those features with your judgment grades, and training a model that learns the patterns.

Our previous article becomes the starting point for this process. Your grades define what "relevant" means and how to measure it, and your features give the model the signals to predict it.

For the complete implementation with a dataset of 9,750 movies and 384,755 judgment rows, see the LTR notebook. For advanced use cases, like personalized search, see Personalized search with LTR.

Prerequisites

• Elasticsearch 8.15+ (for :: cast operator support; core ES|QL features available from 8.11)

Runtime fields versus ES|QL

Runtime fields were introduced in Elasticsearch 7.11 as a way to define fields at query time. Instead of reindexing data, you could write a Painless script that computes values on the fly:

PUT my-index/_mapping
{
  "runtime": {
    "full_address": {
      "type": "keyword",
      "script": {
        "source": "emit(doc['address'].value + ':' + doc['port'].value)"
      }
    }
  }
}

This works, but comes with trade-offs:

• Painless scripting overhead: Every runtime field requires scripting knowledge, and the syntax is Java-like, not query-like.
• Performance cost: Runtime fields evaluate per document at query time. Elasticsearch classifies them as "expensive queries" that can be rejected by cluster settings.
• Isolated computation: Each runtime field computes independently. There’s no way to chain transforms or use the output of one field in another within the same query.

ES|QL changes the equation. It has its own execution engine (not translated to Query DSL), runs queries concurrently across nodes, and provides a complete toolkit for field computation: EVAL, GROK, DISSECT, type casting, and pipeline chaining.

Let's see how each runtime field pattern maps to ES|QL.

Setting up the example data

All the code snippets in this article can be executed in the Kibana Dev Tools console.

To follow along, create a sample index with data that exercises all five patterns. This simulates a server logs scenario with mixed field types, raw messages, and some intentional data quality issues:

PUT server-logs
{
  "mappings": {
    "properties": {
      "host": { "type": "keyword" },
      "port": { "type": "keyword" },
      "raw_message": { "type": "text" },
      "response_time": { "type": "keyword" },
      "status_code": { "type": "keyword" },
      "region": { "type": "keyword" }
    }
  }
}

Now index some sample documents:

POST _bulk
{ "index": { "_index": "server-logs" } }
{ "host": "web-01", "port": "8080", "raw_message": "2024-01-15 INFO user=alice action=login duration=230ms", "response_time": "145", "status_code": "200", "region": "us-east" }
{ "index": { "_index": "server-logs" } }
{ "host": "web-02", "port": "443", "raw_message": "2024-01-15 ERROR user=bob action=upload duration=1200ms", "response_time": "not_available", "status_code": "500", "region": "eu-west" }
{ "index": { "_index": "server-logs" } }
{ "host": "api-01", "port": "3000", "raw_message": "2024-01-15 WARN user=charlie action=query duration=890ms", "response_time": "890", "status_code": "200", "region": "us-east" }
{ "index": { "_index": "server-logs" } }
{ "host": "api-02", "port": "3000", "raw_message": "2024-01-16 INFO user=diana action=export duration=3400ms", "response_time": "3400", "status_code": "200", "region": "ap-south" }
{ "index": { "_index": "server-logs" } }
{ "host": "web-01", "port": "8080", "raw_message": "2024-01-16 ERROR user=eve action=login duration=50ms", "response_time": "50", "status_code": "401", "region": "US-EAST" }

Notice that response_time is stored as a keyword (a common real-world mistake), and the last document has "US-EAST" instead of "us-east" (a data quality issue we’ll fix later).

Pattern 1: Field concatenation

A common runtime field use case is combining two fields into one. For example, creating a host:port identifier.

The runtime field approach

You can define it inline at query time. Query-time approach avoids modifying the mapping, but you still need Painless scripting, scoping it to a single search request:

GET server-logs/_search
{
  "runtime_mappings": {
    "endpoint": {
      "type": "keyword",
      "script": {
        "source": "emit(doc['host'].value + ':' + doc['port'].value)"
      }
    }
  },
  "fields": ["endpoint"],
  "_source": false
}

The ES|QL approach

You can run ES|QL queries using the _query API endpoint:

POST _query
{
  "query": """
    FROM server-logs
    | EVAL endpoint = CONCAT(host, ":", port)
    | KEEP host, port, endpoint
    | LIMIT 1
  """
}

Response:

{
  "columns": [
    { "name": "host", "type": "keyword" },
    { "name": "port", "type": "keyword" },
    { "name": "endpoint", "type": "keyword" }
  ],
  "values": [
    ["web-01", "8080", "web-01:8080"]
  ]
}

CONCAT accepts two or more arguments and always returns a keyword.

Note: For brevity, the remaining ES|QL examples in this article show just the query. Wrap them in POST _query { "query": "..." } to run them in Kibana Dev Tools.

When to use

If you need endpoint to persist across all queries and be available in Kibana dashboards, use a mapping-level runtime field. If you need it for a single search request within Query DSL, use a query-time runtime field. If you need it for ad-hoc analysis or exploratory work, ES|QL is simpler.

Pattern 2: Data extraction from unstructured text

Extracting structured data from raw log messages is another classic runtime field pattern.

The runtime field approach

Painless uses Java's regex Matcher class:

GET server-logs/_search
{
  "runtime_mappings": {
    "log_user": {
      "type": "keyword",
      "script": {
        "source": "def matcher = /user=(\\w+)/.matcher(params._source['raw_message']); if (matcher.find()) { emit(matcher.group(1)); }"
      }
    }
  },
  "fields": ["log_user"],
  "_source": false
}

This is verbose. You need to know Painless regex syntax, handle the Matcher object, and call emit() correctly.

The ES|QL approach: GROK

ES|QL provides two purpose-built commands for text extraction. GROK uses regex-based patterns:

FROM server-logs
| GROK raw_message "%{WORD:timestamp_date} %{WORD:log_level} user=%{WORD:user} action=%{WORD:action} duration=%{WORD:duration}"
| KEEP user, log_level, action, duration

Response:

{
  "columns": [
    { "name": "user", "type": "keyword" },
    { "name": "log_level", "type": "keyword" },
    { "name": "action", "type": "keyword" },
    { "name": "duration", "type": "keyword" }
  ],
  "values": [
    ["alice", "INFO", "login", "230ms"], ...
  ]
}

GROK uses the %{SYNTAX:SEMANTIC} pattern format. It extracts multiple fields in a single and readable command.

The ES|QL approach: DISSECT

For structured data with consistent delimiters, DISSECT is faster because it doesn’t use regular expressions:

FROM server-logs
| DISSECT raw_message "%{timestamp_date} %{log_level} user=%{user} action=%{action} duration=%{duration}"
| KEEP user, log_level, action, duration

The syntax is nearly identical to GROK, but DISSECT works by splitting on delimiters rather than matching regex patterns. This makes it faster for data that follows a consistent format.

When to use GROK vs DISSECT

Use DISSECT when your data has a predictable structure (same delimiters, same field order). Use GROK when you need regex flexibility, for example when fields may be optional or formats vary.

Pattern 3: Dynamic type conversion

When a field is mapped as keyword but contains numeric data (a surprisingly common scenario), runtime fields can cast it at query time.

The runtime field approach

GET server-logs/_search
{
  "runtime_mappings": {
    "response_time_long": {
      "type": "long",
      "script": {
        "source": """
          def val = doc['response_time'].value;
          if (val != 'not_available') {
            emit(Long.parseLong(val));
          }
        """
      }
    }
  },
  "fields": ["response_time_long"],
  "_source": false
}

You need to handle parsing exceptions manually. If Long.parseLong fails on an unexpected value, the script throws an error.

The ES|QL approach

ES|QL provides explicit conversion functions and a shorthand cast operator:

FROM server-logs
| EVAL response_ms = TO_LONG(response_time)
| KEEP host, response_time, response_ms

Or with the :: cast operator (available since 8.15):

FROM server-logs
| EVAL response_ms = response_time::long
| KEEP host, response_time, response_ms

Response:

{
  "columns": [
    { "name": "host", "type": "keyword" },
    { "name": "response_time", "type": "keyword" },
    { "name": "response_ms", "type": "long" }
  ],
  "values": [
    ["web-01", "145", 145]
  ]
}

Both produce the same result. The key difference from Painless: Failed conversions return null instead of throwing exceptions. The document with "not_available" simply gets null for response_ms, and ES|QL emits a warning.

Common conversion functions include:

The :: operator works with all these types (for example, field::double, field::datetime).

When to use

ES|QL's graceful null handling makes it safer for dirty data. Runtime fields with Painless give you fine-grained control over error handling but require more code. For type conversion specifically, ES|QL is almost always the better choice.

Pattern 4: Dynamic field handling

Runtime fields support "dynamic": "runtime" in mappings, which prevents mapping explosion by creating all new fields as runtime fields instead of indexed fields:

{
  "mappings": {
    "dynamic": "runtime",
    "properties": {
      "timestamp": { "type": "date" }
    }
  }
}

Any new field sent to this index becomes a runtime field automatically. This is useful when you ingest semi-structured data with unpredictable field names.

Where ES|QL fits

ES|QL provides query-time flexibility, but it still needs fields to be visible in the mapping. This is where runtime fields and ES|QL complement each other rather than compete.

If a field exists in _source but isn’t mapped, ES|QL cannot access it directly. The current workaround is to define a runtime field to make the unmapped field visible:

PUT dynamic-logs/_mapping
{
  "runtime": {
    "custom_field": {
      "type": "keyword",
      "script": {
        "source": "emit(params._source['custom_field'])"
      }
    }
  }
}

Once defined, ES|QL can query it:

FROM dynamic-logs
| WHERE custom_field == "some_value"
| KEEP timestamp, custom_field

This is one scenario where runtime fields remain essential. They act as a bridge, making unmapped data accessible to ES|QL.

Pattern 5: Field shadowing for error correction

Runtime fields can shadow (override) indexed fields by defining a runtime field with the same name as an existing field. This is useful for correcting data without reindexing.

The runtime field approach

Remember our data quality issue, where region has inconsistent casing ("US-EAST" versus "us-east")?

GET server-logs/_search
{
  "runtime_mappings": {
    "region": {
      "type": "keyword",
      "script": {
        "source": "emit(params._source['region'].toLowerCase())"
      }
    }
  },
  "fields": ["region"],
  "_source": false
}

This overrides the indexed region field for all queries. Every search, aggregation, and Kibana visualization will see the lowercase version.

FROM server-logs
| EVAL region = TO_LOWER(region)
| KEEP host, port, region

When you use EVAL with an existing column name, ES|QL drops the original column and replaces it with the computed value. This is the exact equivalent of field shadowing, but scoped to the current query.

You can also chain multiple corrections in a pipeline:

FROM server-logs
| EVAL region = TO_LOWER(region)
| EVAL region = CASE(region == "us-east", "US East", region == "eu-west", "EU West", region == "ap-south", "AP South", region)
| KEEP host, region

When to use

If the correction should apply to all queries and Kibana dashboards, use runtime field shadowing. If you need to correct data for a specific analysis, ES|QL is more flexible since you can apply different transformations in different queries without modifying the mapping.

The ES|QL pipeline advantage: Going beyond runtime fields

This is where ES|QL fundamentally surpasses runtime fields. Runtime fields are isolated: each one computes independently, and you cannot use the output of one runtime field as input for another in the same query.

ES|QL pipelines chain transforms. Here’s a single query that combines multiple patterns:

FROM server-logs
| GROK raw_message "%{WORD:log_date} %{WORD:log_level} user=%{WORD:user} action=%{WORD:action} duration=%{INT:duration_raw}ms"
| EVAL duration_ms = duration_raw::long
| EVAL region = TO_LOWER(region)
| WHERE log_level == "ERROR" AND duration_ms > 100
| STATS avg_duration = AVG(duration_ms), error_count = COUNT(*) BY region

This single query:

• Extracts fields from raw text (GROK).
• Converts the duration to a number (EVAL with cast).
• Normalizes region casing (EVAL with TO_LOWER).
• Filters for errors with high duration (WHERE).
• Aggregates by region (STATS).

To achieve the same result with runtime fields, you would need to define at least three separate runtime fields (for extraction, conversion, and normalization) and then write a Query DSL query with filters and aggregations. The ES|QL version is a single, readable pipeline.

You can even use expressions directly inside aggregations:

FROM server-logs
| EVAL response_ms = response_time::long
| STATS
    avg_response = AVG(response_ms),
    p95_response = PERCENTILE(response_ms, 95),
    slow_count = COUNT(CASE(response_ms > 1000, 1, null))
  BY host

Conclusion

What we covered:

• ES|QL provides a full toolkit (EVAL, GROK, DISSECT, type casting with ::) that replaces most runtime field patterns without any Painless scripting.
• Failed type conversions in ES|QL return null instead of throwing exceptions, making it safer for real-world data.
• Pipeline processing (chaining GROK into EVAL into WHERE into STATS) goes beyond what runtime fields can do in isolation.
• Runtime fields remain valuable for persistent computed fields, field shadowing across all queries, and as a bridge for unmapped data in ES|QL.

One important caveat: Both runtime fields and ES|QL compute values at query time, which means they pay the cost on every query. If you find yourself applying the same transformation repeatedly (type corrections, field extraction, data normalization), consider using ingest pipelines to fix the data at index time instead. Ingest pipelines let you parse, enrich, and transform documents before they’re stored, so queries can work with clean, properly typed fields directly. Runtime fields and ES|QL are great for exploration and ad-hoc analysis, but for production workloads, indexing the right data from the start is almost always the better choice.

The key takeaway: Runtime fields aren’t deprecated, and they aren’t going away. But for most query-time computation patterns, ES|QL offers a simpler, more powerful, and more performant approach. And when the transformation is known up front, an ingest pipeline is the most efficient option of all.

Next steps

• ES|QL documentation
• Runtime fields reference
• ES|QL timeline of improvements
• Getting started with runtime fields
• ES|QL processing data with DISSECT and GROK

In this article, we’ll explore the benefits of building a custom Elasticsearch MCP server and show how to create one in TypeScript that connects Elasticsearch to LLM-powered applications.

Why build a custom Elasticsearch MCP server?

Elastic provides some alternatives for MCP servers:

• Elastic Agent Builder MCP server for Elasticsearch 9.2+
• Elasticsearch MCP server for older versions (Python)

If you need more control over how your MCP server interacts with Elasticsearch, building your own custom server gives you the flexibility to tailor it exactly to your needs. For example, Agent Builder's MCP endpoint is limited to Elasticsearch Query Language (ES|QL) queries, while a custom server allows you to use the full Query DSL. You also gain control over how results are formatted before being passed to the LLM and can integrate additional processing steps, like the OpenAI-powered summarization we'll implement in this tutorial.

By the end of this article, you’ll have an MCP server in TypeScript that searches for information stored in an Elasticsearch index, summarizes it, and provides citations. We'll use Elasticsearch for retrieval, OpenAI's gpt-4o-mini model to summarize and generate citations, and Claude Desktop as the MCP client and UI to take in user queries and give responses. The end result is an internal knowledge assistant that helps engineers discover and synthesize best practices across their organization’s technical docs.

Prerequisites:

• Node.js 20 +
• Elasticsearch
• OpenAI API key
• Claude Desktop

What is MCP?

MCP is an open standard, created by Anthropic, that provides secure, bidirectional connections between LLMs and external systems, like Elasticsearch. You can read more about the current state of MCP in this article.

The MCP landscape is evolving every day, with servers available for a wide range of use cases. On top of that, it’s easy to build your own custom MCP server, as we’ll show in this article.

MCP clients

There’s a long list of available MCP clients, each with its own characteristics and limitations. For simplicity and popularity, we’ll use Claude Desktop as our MCP client. It will serve as the chat interface where users can ask questions in natural language, and it will automatically invoke the tools exposed by our MCP server to search documents and generate summaries.

Creating an Elasticsearch MCP server

Using the TypeScript SDK, we can easily create a server that understands how to query our Elasticsearch data based on a user query input.

Here are the steps in this article to integrate the Elasticsearch MCP server with the Claude Desktop client:

1. Configure MCP server for Elasticsearch.
2. Load the MCP server into Claude Desktop.
3. Test it out.

Configure MCP server for Elasticsearch

To begin, let's initialize a node application:

npm init -y

This will create a package.json file, and with it, we can start installing the necessary dependencies for this application.

npm install @elastic/elasticsearch @modelcontextprotocol/sdk openai zod && npm install --save-dev ts-node @types/node typescript

• @elastic/elasticsearch will give us access to the Elasticsearch Node.js library.
• @modelcontextprotocol/sdk provides the core tools to create and manage an MCP server, register tools, and handle communication with MCP clients.
• openai allows interaction with OpenAI models to generate summaries or natural language responses.
• zod helps define and validate structured schemas for input and output data in each tool.

ts-node, @types/node, and typescript will be used during development to type the code and compile the scripts.

Set up the dataset

To provide the data that Claude Desktop can query using our MCP server, we’ll use a mock internal knowledge base dataset. Here’s what a document from this dataset will look like:

{
    "id": 5,
    "title": "Logging Standards for Microservices",
    "content": "Consistent logging across microservices helps with debugging and tracing. Use structured JSON logs and include request IDs and timestamps. Avoid logging sensitive information. Centralize logs in Elasticsearch or a similar system. Configure log rotation to prevent storage issues and ensure logs are searchable for at least 30 days.",
    "tags": ["logging", "microservices", "standards"]
}

To ingest the data, we prepared a script that creates an index in Elasticsearch and loads the dataset into it. You can find it here.

MCP server

Create a file named index.ts and add the following code to import the dependencies and handle environment variables:

// index.ts
import { z } from "zod";
import { Client } from "@elastic/elasticsearch";
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import OpenAI from "openai";

const ELASTICSEARCH_ENDPOINT =
  process.env.ELASTICSEARCH_ENDPOINT ?? "http://localhost:9200";
const ELASTICSEARCH_API_KEY = process.env.ELASTICSEARCH_API_KEY ?? "";
const OPENAI_API_KEY = process.env.OPENAI_API_KEY ?? "";
const INDEX = "documents";

Also, let’s initialize the clients to handle the Elasticsearch and OpenAI calls:

const openai = new OpenAI({
  apiKey: OPENAI_API_KEY,
});

const _client = new Client({
  node: ELASTICSEARCH_ENDPOINT,
  auth: {
    apiKey: ELASTICSEARCH_API_KEY,
  },
});

To make our implementation more robust and ensure structured input and output, we'll define schemas using zod. This allows us to validate data at runtime, catch errors early, and make the tool responses easier to process programmatically:

const DocumentSchema = z.object({
  id: z.number(),
  title: z.string(),
  content: z.string(),
  tags: z.array(z.string()),
});

const SearchResultSchema = z.object({
  id: z.number(),
  title: z.string(),
  content: z.string(),
  tags: z.array(z.string()),
  score: z.number(),
});

type Document = z.infer;
type SearchResult = z.infer;

Learn more about structured outputs here.

Now let’s initialize the MCP server:

const server = new McpServer({
  name: "Elasticsearch RAG MCP",
  description:
    "A RAG server using Elasticsearch. Provides tools for document search, result summarization, and source citation.",
  version: "1.0.0",
});

Defining the MCP tools

With everything configured, we can start writing the tools that will be exposed by our MCP server. This server exposes two tools:

• search_docs: Searches for documents in Elasticsearch using full-text search.
• summarize_and_cite: Summarizes and synthesizes information from previously retrieved documents to answer a user question. This tool also adds citations referencing the source documents.

Together, these tools form a simple “retrieve-then-summarize” workflow, where one tool fetches relevant documents and the other uses those documents to generate a summarized, cited response.

Tool response format

Each tool can accept arbitrary input parameters, but it must respond with the following structure:

• Content: This is the response of the tool in an unstructured format. This field is usually used to return text, images, audio, links, or embeddings. For this application, it will be used to return formatted text with the information generated by the tools.
• structuredContent: This is an optional return used to provide the results of each tool in a structured format. This is useful for programmatic purposes. Although it isn't used in this MCP server, it can be useful if you want to develop other tools or process the results programmatically.

With that structure in mind, let’s dive into each tool in detail.

Search_docs tool

This tool performs a full-text search in the Elasticsearch index to retrieve the most relevant documents based on the user query. It highlights key matches and provides a quick overview with relevance scores.

server.registerTool(
  "search_docs",
  {
    title: "Search Documents",
    description:
      "Search for documents in Elasticsearch using full-text search. Returns the most relevant documents with their content, title, tags, and relevance score.",
    inputSchema: {
      query: z
        .string()
        .describe("The search query terms to find relevant documents"),
      max_results: z
        .number()
        .optional()
        .default(5)
        .describe("Maximum number of results to return"),
    },
    outputSchema: {
      results: z.array(SearchResultSchema),
      total: z.number(),
    },
  },
  async ({ query, max_results }) => {
    if (!query) {
      return {
        content: [
          {
            type: "text",
            text: "Query parameter is required",
          },
        ],
        isError: true,
      };
    }

    try {
      const response = await _client.search({
        index: INDEX,
        size: max_results,
        query: {
          bool: {
            must: [
              {
                multi_match: {
                  query: query,
                  fields: ["title^2", "content", "tags"],
                  fuzziness: "AUTO",
                },
              },
            ],
            should: [
              {
                match_phrase: {
                  title: {
                    query: query,
                    boost: 2,
                  },
                },
              },
            ],
          },
        },
        highlight: {
          fields: {
            title: {},
            content: {},
          },
        },
      });

      const results: SearchResult[] = response.hits.hits.map((hit: any) => {
        const source = hit._source as Document;

        return {
          id: source.id,
          title: source.title,
          content: source.content,
          tags: source.tags,
          score: hit._score ?? 0,
        };
      });

      const contentText = results
        .map(
          (r, i) =>
            `[${i + 1}] ${r.title} (score: ${r.score.toFixed(
              2,
            )})\n${r.content.substring(0, 200)}...`,
        )
        .join("\n\n");

      const totalHits =
        typeof response.hits.total === "number"
          ? response.hits.total
          : (response.hits.total?.value ?? 0);

      return {
        content: [
          {
            type: "text",
            text: `Found ${results.length} relevant documents:\n\n${contentText}`,
          },
        ],
        structuredContent: {
          results: results,
          total: totalHits,
        },
      };
    } catch (error: any) {
      console.log("Error during search:", error);

      return {
        content: [
          {
            type: "text",
            text: `Error searching documents: ${error.message}`,
          },
        ],
        isError: true,
      };
    }
  }
);

We configure fuzziness: “AUTO” to have a variable typo tolerance based on the length of the token that’s being analyzed. We also set title^2 to increase the score of the documents where the match happens on the title field.

summarize_and_cite tool

This tool generates a summary based on documents retrieved in the previous search. It uses OpenAI’s gpt-4o-mini model to synthesize the most relevant information to answer the user’s question, providing responses derived directly from the search results. In addition to the summary, it also returns citation metadata for the source documents used.

server.registerTool(
  "summarize_and_cite",
  {
    title: "Summarize and Cite",
    description:
      "Summarize the provided search results to answer a question and return citation metadata for the sources used.",
    inputSchema: {
      results: z
        .array(SearchResultSchema)
        .describe("Array of search results from search_docs"),
      question: z.string().describe("The question to answer"),
      max_length: z
        .number()
        .optional()
        .default(500)
        .describe("Maximum length of the summary in characters"),
      max_docs: z
        .number()
        .optional()
        .default(5)
        .describe("Maximum number of documents to include in the context"),
    },
    outputSchema: {
      summary: z.string(),
      sources_used: z.number(),
      citations: z.array(
        z.object({
          id: z.number(),
          title: z.string(),
          tags: z.array(z.string()),
          relevance_score: z.number(),
        })
      ),
    },
  },
  async ({ results, question, max_length, max_docs }) => {
    if (!results || results.length === 0 || !question) {
      return {
        content: [
          {
            type: "text",
            text: "Both results and question parameters are required, and results must not be empty",
          },
        ],
        isError: true,
      };
    }

    try {
      const used = results.slice(0, max_docs);

      const context = used
        .map(
          (r: SearchResult, i: number) =>
            `[Document ${i + 1}: ${r.title}]\\n${r.content}`
        )
        .join("\n\n---\n\n");

      // Generate summary with OpenAI
      const completion = await openai.chat.completions.create({
        model: "gpt-4o-mini",
        messages: [
          {
            role: "system",
            content:
              "You are a helpful assistant that answers questions based on provided documents. Synthesize information from the documents to answer the user's question accurately and concisely. If the documents don't contain relevant information, say so.",
          },
          {
            role: "user",
            content: `Question: ${question}\\n\\nRelevant Documents:\\n${context}`,
          },
        ],
        max_tokens: Math.min(Math.ceil(max_length / 4), 1000),
        temperature: 0.3,
      });

      const summaryText =
        completion.choices[0]?.message?.content ?? "No summary generated.";

      const citations = used.map((r: SearchResult) => ({
        id: r.id,
        title: r.title,
        tags: r.tags,
        relevance_score: r.score,
      }));

      const citationText = citations
        .map(
          (c: any, i: number) =>
            `[${i + 1}] ID: ${c.id}, Title: "${c.title}", Tags: ${c.tags.join(
              ", ",
            )}, Score: ${c.relevance_score.toFixed(2)}`,
        )
        .join("\n");

      const combinedText = `Summary:\\n\\n${summaryText}\\n\\nSources used (${citations.length}):\\n\\n${citationText}`;

      return {
        content: [
          {
            type: "text",
            text: combinedText,
          },
        ],
        structuredContent: {
          summary: summaryText,
          sources_used: citations.length,
          citations: citations,
        },
      };
    } catch (error: any) {
      return {
        content: [
          {
            type: "text",
            text: `Error generating summary and citations: ${error.message}`,
          },
        ],
        isError: true,
      };
    }
  }
);

Finally, we need to start the server using stdio. This means the MCP client will communicate with our server by reading and writing to its standard input and output streams. stdio is the simplest transport option and works well for local MCP servers launched as subprocesses by the client. Add the following code at the end of the file:

const transport = new StdioServerTransport();
server.connect(transport);

Now compile the project using the following command:

npx tsc index.ts --target ES2022 --module node16 --moduleResolution node16 --outDir ./dist --strict --esModuleInterop

This will create a dist folder, and inside it, an index.js file.

Load the MCP server into Claude Desktop

Follow this guide to configure the MCP server with Claude Desktop. In the Claude configuration file, we need to set the following values:

{
  "mcpServers": {
    "elasticsearch-rag-mcp": {
      "command": "node",
      "args": [   "/Users/user-name/app-dir/dist/index.js"
      ],
      "env": {
        "ELASTICSEARCH_ENDPOINT": "your-endpoint-here",
        "ELASTICSEARCH_API_KEY": "your-api-key-here",
        "OPENAI_API_KEY": "your-openai-key-here"
      }
    }
  }
}

The args value should point to the compiled file in the dist folder. You also need to set the environment variables in the configuration file with the exact same names defined in the code.

Test it out

Before executing each tool, click on Search and Tools to make sure that the tools are enabled. Here you can also enable or disable each one:

Finally, let’s test the MCP server from the Claude Desktop chat and start asking questions:

For the question “Search for documents about authentication methods and role-based access control”, the search_docs tool is executed and returns the following results:

Most Relevant Documents:
Access Control and Role Management (highest relevance) - This document covers role-based access control (RBAC) principles, including ensuring users only have necessary permissions, regular auditing of user roles, revoking inactive accounts, and implementing just-in-time access for sensitive operations.
User Authentication with OAuth 2.0 - This document explains OAuth 2.0 authentication, which enables secure delegated access without credential sharing. It covers configuring identity providers, token management with limited scope and lifetime, and secure storage of refresh tokens.
Container Security Guidelines - While primarily about container security, this document touches on access control aspects like running containers as non-root users and avoiding embedded credentials.
Incident Response Playbook - This mentions role assignment during incidents (incident commander, communications lead, etc.), which relates to access control in emergency scenarios.
Logging Standards for Microservices - This document includes guidance on avoiding logging sensitive information, which is relevant to authentication security.

The response is, “Great! I found 5 relevant documents about authentication methods and role-based access control. Here's what was found:”

The tool call returns the source documents as part of its response payload, which are later used to generate citations.

It’s also possible to chain multiple tools in a single interaction. In this case, Claude Desktop analyzes the user's question and determines that it needs to first call search_docs to retrieve relevant documents and to then pass those results to summarize_and_cite to generate the final answer, all without requiring separate prompts from the user:

In this case, for the query “What are the main recommendations to improve authentication and access control across our systems? Include references.”, we obtained the following results:

Based on the documentation, here are the main recommendations to improve authentication and access control across your systems:

Key Recommendations

1. Implement Role-Based Access Control (RBAC)
Ensure users have only the permissions necessary for their job functions. This principle of least privilege minimizes security risks by limiting access to what's truly needed. [1]

2. Regular Access Audits
Conduct regular audits of user roles and promptly revoke access for inactive accounts. This helps prevent unauthorized access from dormant or compromised accounts. [1]

3. Just-in-Time (JIT) Access
Implement just-in-time access for sensitive operations, where permissions are granted only when needed and for a limited duration. This reduces the attack surface by minimizing standing privileges. [1]

4. OAuth 2.0 for Secure Authentication
Use OAuth 2.0 to enable secure delegated access without sharing user credentials. This protocol allows third-party applications to access resources on behalf of users without exposing passwords. [2]

5. Token Security and Management
Configure your identity provider to issue tokens with limited scope and lifetime

Store refresh tokens securely
Validate access tokens consistently to maintain security [2]
References

Access Control and Role Management (Tags: security, access-control)
User Authentication with OAuth 2.0 (Tags: authentication, oauth)
These recommendations work together to create a defense-in-depth approach, where multiple security layers protect your systems from unauthorized access.

As in the previous step, we can see the response from each tool for this question:

Note: If a submenu appears asking whether you approve the use of each tool, select Always allow or Allow once.

Conclusion

MCP servers represent a significant step toward standardizing LLM tools for both local and remote applications. Though full compatibility is still in the works, we’re moving fast in that direction.

In this article, we learned how to build a custom MCP server in TypeScript that connects Elasticsearch to LLM-powered applications. Our server exposes two tools: search_docs for retrieving relevant documents using Query DSL; and summarize_and_cite for generating summaries with citations via OpenAI models and Claude Desktop as client UI.

The future of compatibility between different client and server providers looks promising. Next steps include adding more functionalities and flexibility to your agent. There’s a practical article on how you can parameterize your queries using search templates to gain precision and flexibility.

That’s exactly why we built read-only dashboards. It’s the control you’ve been asking for. Share dashboards with confidence, without worrying that the next person with edit access will change or break them.

Note: Read-only permissions are available in Elastic Cloud Serverless and from version 9.3 for Elastic Cloud Hosted and Elastic Self-Μanaged.

When “everyone can edit” gets in the way

In Kibana, sharing has usually meant space-level permissions. If someone can create dashboards in a space, they can also edit or delete anyone else’s. That’s great for collaboration until it isn’t. One accidental edit can ripple into wrong decisions, lost trust, and a lot of cleanup.

We’ve heard the workarounds: “We put ‘read-only’ in the dashboard name and hope people notice.” Or: “We tag them and cross our fingers.” Hope isn’t a permission model. You needed a real way to lock a dashboard without locking everyone out of the space.

What actually goes wrong

Deb and Kevin both have edit access to the log monitoring dashboard within the Operations space. Kevin makes some changes to the charts. When Deb comes back, the numbers don’t match what she presented. She has to track down what changed (often from memory), fix it, and wonder how many reports went out with bad data.

Read-only dashboards: Ownership and control that make sense

Read-only dashboards fix this by giving you control to decide whether other users can edit the dashboard. When you share a dashboard, you choose: edit (default, same as today) or view. In view mode, only you (and Kibana admins) can change or delete it. Everyone else can open it, use it, and trust it, but they can’t modify it.

What you get

• Dashboard integrity: In view mode, other users with edit access in the space cannot modify or delete the dashboard. If they try, they’re told it’s locked. Your charts and logic stay as you left them.
• You stay in control: You’re the owner. You can always edit, refine, and update. Sharing as view-only doesn’t lock you out; it locks in the version everyone else sees.
• Flexible lifecycle: You can switch a dashboard back to “can edit” anytime. And Kibana admins can still manage all dashboards (for example, if the owner leaves). No dead ends.

You can share finalized, mission-critical dashboards widely and know they’ll stay consistent. This is available in all Elastic tiers and offerings, including Serverless.

Who can do what?

Quick reference by role:

• Dashboard owner: You created it; you have full edit access.
• Kibana admin: Can manage all dashboards.
• User with space edit: Can create and edit their dashboards; can’t edit or delete view-only dashboards.
• User with space view: Can only view (and list) dashboards.

How to turn on read-only

You can set view-only when you save a new dashboard or later from the share menu.

When saving a new dashboard

• Build your dashboard, and click Save.
• In the “Save as new dashboard” modal, find Permissions.
• Change from Can edit to Can view.
• Click Save. Done. It’s read-only for everyone else.

For a dashboard you already own

• Open the dashboard.
• Open the Share dashboard menu.

• In the sharing modal, find Permissions and switch to Can view. The change applies immediately; other users in the space can no longer edit or delete it.

• You can mouse over the Share action to see what type of permissions a given dashboard has.

Seeing which dashboards are locked

On the main Dashboards list, dashboards you can’t edit or delete have a disabled selection checkbox. This provides an easy way to spot what’s view-only.

In the dashboard, you will also find that the Edit action is disabled and a tooltip will appear, explaining that the dashboard has been set as view-only.

Try it

Read-only dashboards are available now. Create a dashboard, flip it to Can view, and share it. Your team gets a single source of truth, and you get peace of mind. No more “please don’t edit” in the title.

We’d love to hear how you use read-only dashboards. Share your feedback in our community forum.

This post refocuses on the question,What are the right search interfaces an agent needs to build its own context? It first covers the trade-offs between shell tools and dedicated database tools. From there, it offers a practical framework for finding the right interfaces for your agent's needs.

What does "building context" actually mean for an agent?

In early retrieval augmented generation (RAG) pipelines, the developer engineered a fixed retrieval pipeline, and the large language model (LLM) was a passive recipient of the context. This was a fundamental limitation: Context was retrieved on every query, whether or not it was needed, with no check that it actually helped.

With the shift to agentic RAG, the agents now have access to a set of search tools to build their own context. For example, both Claude Code [1] and Cursor [2] let the agent choose between different search tools and even combine them for chained queries, depending on what the task actually requires.

What search interfaces exist for context engineering?

Context can live in different locations, such as on the web, in a local filesystem, or in a database. An agent can interact with each of these out-of-context data sources through different tools:

• Shell tools can execute shell commands and have access to the local filesystem. Some examples of built-in shell tools are Claude API's bash tool, OpenClaw's exec tool, and LangChain's shell tool.
• Dedicated database tools, such as tools from a Model Context Protocol (MCP) server (for example, the Elastic Agent Builder MCP server) or custom tools (for example, run_esql(query) or db_list_index()), can query databases.
• Dedicated file search tools can search and read local (or uploaded) files (without full shell access). Some examples of built-in file search tools are Gemini API’s File Search Tool or OpenAI’s File Search Tool.
• Web search tools can retrieve information from the web.
• Memory tools store and recall from long-term memory (regardless of how it’s stored).

As you can see, the shell tool is versatile and can be used to retrieve context from different data sources, including:

• Filesystem: The agent explores the directory structure (ls, find), searches for relevant content (grep, cat), and repeats until it has built sufficient context.
• Database: The agent can use database command line interface (CLI) tools (for example, elasticsearch-sql-cli), call HTTP APIs via curl, or run scripts, which is especially useful in combination with agent skills, which are reusable, documented examples injected into the agent's context to guide correct tool usage (for example, Elastic Agent Skills for Elasticsearch).
• Web: The agent can execute web searches via a curl command through a search provider’s API.

However, the shell tool provides direct system access and therefore requires safety measures, such as running in an isolated sandbox environment and logging all executed commands.

When to use which search interfaces

The right search interface depends on your data, your query patterns, and your use case. This section serves as a practical starting point.

Filesystems aren’t making databases obsolete

The filesystems-versus-databases discussion is not about the storage layer. For example, LangChain explains that its memory system doesn’t actually store memory in a real filesystem. Instead, it stores memory in a database and represents it as a set of files to the agent [3].

Filesystems are a natural fit for file-native use cases, such as coding agents. They also work well as a temporary scratch pad or working memory and for single-user or single-agent scenarios where concurrency isn't a concern. In these cases, a physical filesystem or representing the data as a filesystem gives you flexibility before committing to a purpose-built interface.

But filesystem storage has real downsides, such as weak concurrency, manual schema enforcement, and atomic transactions. These become more apparent when your application needs to scale or move to a multi-agent scenario. Anyone who ignores these downsides is doomed to painfully reinvent worse databases without the decades of engineering behind transaction safety or access control that production databases already provide. Additionally, in most enterprise contexts, you don't choose whether to use a database since it's already there, storing business-critical data.

Shell tool + filesystem

A shell tool is the natural starting point for filesystem search. Currently, coding agents are driving a lot of progress in the field. Because they work with code in local files, they’re naturally file-heavy use cases. Therefore, LLMs are fine-tuned in the post-training stage for coding tasks. That’s why many LLMs are not only good at writing code but also at using shell commands and navigating filesystems.

Using a shell tool with built-in CLIs, like ls and grep, to find files is effective. With grep, a query like "Find all files that import matplotlib" is fast, precise, and cheap. But when the agent needs to handle conceptual queries, such as "How does our app handle failed authentication?", pattern matching with grep can hit a ceiling quickly. Several alternatives that bring semantic search capabilities to the command line have emerged to fill this gap, including jina-grep.

However, grep and many of its semantic search alternatives run in O(n) over the corpus. For use cases over codebases, this might be fine. However, if your data grows, latency will become noticeable. In this case, an indexed datastore becomes necessary to maintain performance.

Shell tool + database

Another way to add more search capabilities, such as semantic or hybrid search, over your data is to store it in a database, as Cursor does, for example. Additionally, when data requires complex relational joins or aggregations, a database interface is nonnegotiable.

When the data lives in a database rather than on the filesystem, a shell tool can serve as a lightweight database interface for certain use cases. If your queries are simple enough for a CLI or a curl call, a dedicated database tool may add unnecessary complexity.

This approach is also suitable in early exploration stages, when you don't yet know what query patterns your agent will actually develop. In this case, Agent Skills can give the agent enough structure to query correctly without committing to a purpose-built tool. However, when the agent requires many iterations to figure out the right way to query the database for repeated tasks, the token overhead of using a shell tool as the interface no longer justifies the simplicity benefit of avoiding an extra tool.

Dedicated database tool

Especially when repeated query patterns are structured or analytical, dedicated database tools become necessary. A blog post from Vercel and Braintrust compared agents with different sets of search tools for real-world retrieval tasks over semi-structured data, such as customer support tickets and sales call transcripts (for example, “How many open issues mention 'security'?" or "Find issues where someone reported a bug and later someone submitted a PR claiming to fix it?") [4].

Agents with dedicated database tools used fewer tokens, were faster, and made fewer mistakes than agents with only a shell tool and filesystem. The lesson is that direct database tools are the right choice when the query requires analytical reasoning over semi-structured data.

Combining search interfaces

No single search interface handles every query well. For example, Cursor combines shell tools (for searches via grep) and semantic search tools and lets the agent select the right tool based on the user’s prompt. They report that the agent chooses grep for matching specific symbols or strings, semantic search for conceptual or behavior questions, and both for exploratory tasks.

The Vercel experiment reports the same: Its hybrid agent with access to both a shell tool and a dedicated database tool achieved the best performance out of all tested agents by first using the dedicated database tools and then verifying the results by grepping through the filesystem. However, this approach uses more tokens and time for reasoning about tool choice and verification.

The pattern across both examples is the same: Composition beats any single interface, but composition comes at the trade-off of added cost and latency.

Practical recommendations for finding the right set of tools

The right set of search interfaces is small, purposeful, and specific to your agent's actual query patterns. The current best practice is to have an agent with as few tools as possible instead of having an agent with hundreds of MCP tools. This is because the downside of exposing all possible tools up front is that it bloats the context window and confuses the agent about which tool to actually use. For example, Claude Code reportedly only has about 20 tools.

Instead, the idea of progressive disclosure is to start with a minimal set of tools and let the agent discover additional capabilities only when needed. Research from Anthropic [5] and Cursor [6] has shown that this approach yields a token savings between 47%–85%. Claude Code, for example, implements this directly, allowing the agent to incrementally discover how to query an API or a database, without that knowledge consuming context on every LLM call.

Once you’re familiar with the agent's query patterns, you can revisit the set of search tools that the agent has access to by default. A useful way to think about this trade-off is the "low floor, high ceiling" principle for deciding which tools make the cut. High-ceiling tools don't limit the agent's potential. For example, a versatile shell tool lets the agent write full database queries, including ambiguous ones, but at the cost of reasoning overhead, higher latency, and lower reliability.

Low-floor tools are the opposite. They’re specialized tools that wrap specific queries and are immediately accessible to the agent with minimal reasoning overhead, producing lower cost and higher reliability. But they need upfront engineering, can't cover every possible query, and can make it harder for the agent to choose the right tool.

Think of each tool on a spectrum: Low-floor tools are easy for the agent to use correctly but narrow in scope. High-ceiling tools are versatile but demand more reasoning to use well.

Most agents need a mix of different search tools. But each tool needs to earn its addition. We recommend starting with an all-purpose search tool (for example a search_database() tool or a shell tool). Then reuse the command logs you're already keeping for security purposes to track what your agent actually does, including tool calls, retries, and number of calls per user query. And, when you see a query pattern repeating or failing, that's the signal to build a purpose-built tool for it.

Summary

The filesystem-versus-database debate is distracting from the actual question that engineers need to be asking: What are the right search interfaces an agent needs to build its own context? The answer is most likely, Not a single one.

A shell tool is a versatile tool to interact with different out-of-context sources and thus a good starting point. But it’s less efficient and accurate for use cases with structured analytical queries than dedicated database tools.

The goal is to find the minimal set of search tools that handles your agent's actual query patterns well. Start with a shell tool, and log what your agent actually does. When you see a query pattern repeating and failing, it’s time to engineer specialized tools.

References

1. Thariq (Anthropic). Lessons from Building Claude Code: Seeing like an Agent (2026).

2. Cursor: Documentation. Semantic & agentic search (2026).

3. Harrison Chase (LangChain). How we built Agent Builder’s memory system (2026).

4. Ankur Goyal (Braintrust) and Andrew Qu (Vercel). Testing if "bash is all you need" (2026).

5. Anthropic. Introducing advanced tool use on the Claude Developer Platform (2025).

6. Cursor. Dynamic context discovery (2026).

The party is getting crowded

You're hosting a pizza party. You've got a few friends helping you serve, each stationed at different spots around the room. You give each friend a pizza, and they start handing out slices to hungry guests as they arrive.

At first, things run smoothly. A few guests trickle in, your friends serve slices, everyone's happy. But then word spreads about your sourdough pizzas. The doorbell keeps ringing. Guests pour in. Soon, there's a crowd forming around one of your friends, the one holding the pepperoni pizza, which everyone seems to want.

Your friend with the pepperoni pizza is overwhelmed. Guests are waiting, getting impatient, and a large queue has formed. Meanwhile, your friend holding the margherita pizza is standing around with barely anyone asking for a slice.

What do you do?

You order a couple more pepperoni pizzas and hand them to other friends. Now three friends are holding pepperoni instead of one. The crowd spreads out, and suddenly you can serve three times as many guests at once.

A few things become clear as you host more parties:

• Not all pizzas are equally popular. Some are in high demand, others have fewer takers. You don't need extra "copies" of the unpopular ones. You need extras of the ones with queues.
• Order more pizzas before the queue gets too long. If you wait until your friend is completely overwhelmed and guests are leaving angry, you've waited too long. Better to get an extra pizza when you see a crowd forming.
• Don't throw away pizzas too quickly. Just because the crowd around the pepperoni thinned out for five minutes doesn't mean the rush is over. Maybe they're just refilling drinks, or even talking among themselves (is that still a thing?). Keep the extra pizzas ready. If the lull continues for a while, then you can put them away.
• You can only hand out as many pizzas as you have friends who are helping. If you've only got four friends helping, ten pizzas won’t change the outcome. Only four can be served at once. Match your pizza count to your available hands.
• When a friend leaves, take their pizza. If one of your friends needs to head out, grab their pizza immediately. You can't have pizzas sitting unattended. Hand it to someone else, or put it away.

From pizzas to replicas

Let's map this back to Elasticsearch.

In our analogy, pizzas are replicas (copies of your index shards), your friends helping serve are search nodes, hungry guests are search queries, and that popular pizza with a crowd around it is a hot index with high search load.

When search traffic increases on a particular index, we create additional replicas and distribute them across your search nodes. Any replica can serve any query for that index, just like any friend holding pepperoni can hand out pepperoni slices. More replicas means higher throughput: Three replicas can handle three times the queries per second of a single replica.

Measuring the hunger

Before we decide how many pizzas to order, we need to know how hungry the crowd is.

Elasticsearch tracks the search load for every shard. It's a metric that captures how much search activity a shard is handling. We aggregate this across all shards of an index to understand the total search demand.

What matters most is the relative search load: What proportion of your project's total search traffic is hitting each index? If one index is receiving 60% of all searches while another gets 5%, we know where to add capacity.

The math behind the pizzas

We calculate the optimal number of replicas following this formula:

desired_replicas = min(ceil(L × N / (S × X)), N)

Where:

• L = the index's relative search load (between 0 and 1).
• N = the number of desired search nodes in your project.
• S = the number of shards in the index.
• X = a threshold to avoid hot spots (default: 0.5).

An example: four search nodes, one index with two primary shards receiving 80% of search traffic:

desired_replicas = min(ceil(0.8 × 4 / (2 × 0.5)), 4)
                 = min(4, 4)
                 = 4

This hot index gets four replicas distributed across the search nodes.

The threshold X (defaulting to 0.5) is important. We don't wait until a replica is completely overwhelmed; we scale up when it's at half capacity. Hand out the extra pizza when you see the crowd forming, not when guests are already leaving.

Scale up fast, scale down slow

When search load increases, we add replicas immediately. No reason to make users wait.

When search load drops, we wait a bit before taking any action. We need to see consistent low demand for about 30 minutes before reducing replicas. (This is to deal with spiky traffic where a quiet moment doesn't mean the party is over.)

This matters because adding a replica has a cost. The new replica copies data and warms its caches before serving queries efficiently. Removing replicas too eagerly means constantly paying this startup cost as traffic naturally fluctuates.

Respecting topology bounds

Replicas can never exceed the number of search nodes. Having more replicas than nodes provides no benefit (you can only serve as many pizzas as you have friends who are helping to serve slices).

When nodes are removed from your project, we reduce replicas immediately to match. No waiting for the cooldown, as you can't have unassigned replicas. The moment a friend leaves, we remove their pizza.

The bigger Serverless picture

Replicas for search load balancing works alongside other autoscaling systems:

• Search autoscaling adjusts the number of search nodes (how many friends are helping).
• Replicas for search load balancing distribute traffic by adjusting replica counts per index (how many pizzas of each kind we need).
• Data stream autosharding optimizes shard counts for writes (how to slice each pizza, covered in the previous post).

An important design principle: Replicas for load balancing don't directly trigger search autoscaling. Instead, by distributing search requests across more replicas, it enables increasing resource utilization across your search nodes. This higher utilization then triggers our existing autoscaling logic to add capacity if needed. Replicas for load balancing enables autoscaling to do its job, making sure your search nodes are actually being used, rather than having all traffic bottlenecked on a single replica while other nodes sit idle.

What this means for you

You don't need to predict which indices will be popular. You don't need to manually adjust replicas when traffic patterns change. You don't need to wake up at 3 a.m. because a surge overwhelmed your busiest index.

The system watches where queues are forming and orders more pizzas for those spots. Cold indices don't waste resources on unnecessary replicas. Hot indices get the capacity they need. Your budget goes where it matters.

Conclusion

In the autosharding post, we made sure your pizzas are sliced right. Now, with replicas for search load balancing, we make sure you have enough pizzas, in the right hands, when the hungry crowds arrive.

Try Elastic Cloud Serverless and let us handle the pizza logistics.

Prerequisites

• Elasticsearch 9.3 or Elastic Cloud Serverless: You can create a cloud deployment following these instructions, or you can use the start-local quickstart instead.
• Python 3.12: Download Python here.
• Hugging Face access token.

Chat completions using a Hugging Face inference endpoint

First, we’ll build a practical example that connects Elasticsearch to a Hugging Face inference endpoint to generate AI-powered recommendations from a collection of blog posts. For the app knowledge base, we’ll use a dataset of company blog articles, which contains valuable but often hard-to-navigate information.

With this endpoint, semantic search retrieves the most relevant articles for a given query, and a Hugging Face LLM generates short, contextual recommendations based on those results.

Let’s take a look at a high-level overview of the information flow we’re going to build:

In this article, we’ll test SmolLM3-3B capacity to combine its compact size with strong multilingual reasoning and tool-calling capabilities. Based on a search query, we’ll send all the matching content (in English and Spanish) to the LLM to generate a list of recommended articles with a custom-made description based on the search query and results.

Here’s what the UI of an article site with an AI recommendations generation system could look like.

You can find the full implementation of this application in the linked notebook.

Configuring Elasticsearch inference endpoints

To use the Elasticsearch Hugging Face inference endpoint, we need two important elements: a Hugging Face API key and a running Hugging Face endpoint URL. It should look like this: