Here's what's new.

How to upgrade an Elastic Cloud Hosted deployment via API

Upgrade API: POST /deployments/{id}/upgrade

Upgrading an Elastic Stack version across all resources in a deployment used to require you to construct and submit a full deployment plan update. Now, you can upgrade every resource in a deployment (Elasticsearch, Kibana, and all other components) to a target Elastic Stack version with one API call.

curl -X POST https://api.elastic-cloud.com/api/v1/deployments/{deployment_id}/upgrade \
  -H "Authorization: ApiKey $EC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"target_version": "9.4.0"}'

This pays off most when you operate many deployments and want to roll a new Elastic Stack version across your fleet quickly.

How to resize Elasticsearch tiers in Elastic Cloud Hosted

Scale API: GET / PATCH /deployments/{id}/elasticsearch/{ref_id}/tiers

The tiers API lets you resize any Elasticsearch tier (hot, warm, cold, frozen, master, coordinating or ML) with a single PATCH request scoped only to the tiers you want to change.Send a body keyed by tier ID. Anything you don't mention stays exactly as it was, so you can change one tier in isolation without touching the others.

curl -X PATCH https://api.elastic-cloud.com/api/v1/deployments/{deployment_id}/elasticsearch/main/tiers \
  -H "Authorization: ApiKey $EC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "hot_content": {
      "memory_size": 4096,
      "zone_count": 2
    },
    "warm": {
      "memory_size": 5120,
      "zone_count": 1
    },
    "master": {
      "memory_size": 1024,
      "zone_count": 3
    }
  }'

A few things to know:

• Valid tier IDs are hot_content, warm, cold, frozen, master, coordinating, and ml. Invalid keys are rejected with the list of tiers you can use.
• memory_size and zone_count are each optional per tier. Include only the one you want to change; the other stays untouched.
• Dedicated master handling is automatic. Include master with memory_size and zone_count greater than zero, and the endpoint flips the hot_content tier out of the master-eligible role so your new dedicated masters take over. Include master with memory_size set to zero, and the hot_content tier resumes acting as master-eligible. No separate flag to manage.
• Same validation as the full plan API. The modified plan runs through the existing validators, so size-vs-instance-configuration mismatches, zone-count limits, and topology rules are enforced consistently.

For teams automating tier-level scaling based on ingest load, rebalancing zones before a maintenance window, or promoting a deployment to dedicated masters as it grows, this collapses what used to be a multistep plan edit into a single targeted request.

How to update elasticsearch.yml settings in Elastic Cloud Hosted

User Settings API: GET and PUT /deployments/{id}/{resource_kind}/{ref_id}/user_settings

User-defined settings on a running deployment (elasticsearch.yml-style overrides, Kibana settings, APM config) now have their own endpoints. Read or update them directly with a focused JSON request, scoped to a single resource.

The endpoints cover Elasticsearch, Kibana, Elastic APM, App Search, Enterprise Search, and Integrations Server. Same endpoint shape across all of them; only the {resource_kind} path segment changes.

Read the current settings for a resource:

curl https://api.elastic-cloud.com/api/v1/deployments/{deployment_id}/elasticsearch/main/user_settings \
  -H "Authorization: ApiKey $EC_API_KEY"

Update them with a JSON body containing only the keys you want set:

curl -X PUT https://api.elastic-cloud.com/api/v1/deployments/{deployment_id}/elasticsearch/main/user_settings \
  -H "Authorization: ApiKey $EC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "search.allow_expensive_queries": false,
    "indices.recovery.max_bytes_per_sec": "200mb"
  }'

A few things to know:

• JSON in, JSON out. A JSON contract that's easy to template, diff, and check into source control.
• Existing YAML is auto-migrated. Any tier-level user settings you currently have in YAML are converted to JSON the first time you PUT, so existing deployments work without a manual rewrite or a one-time migration step.
• Allowlist and denylist rules still apply. Settings the platform doesn't permit are rejected with the same validation the full plan API runs.
• Asynchronous. The action is acknowledged immediately and applied in the background, consistent with the other targeted endpoints in this release.
• Non-breaking. The full PUT /deployments/{id} endpoint continues to work; this is an additive surface, not a replacement.

For teams that tune settings as part of a release, sweep a configuration change across a fleet, or want to manage cluster overrides as code, this is a single targeted PUT per deployment.

How to manage deployment tags in Elastic Cloud Hosted

Tags API: GET and PUT /deployments/{id}/tags

Deployment tags attach metadata (environment, owner, cost center, application) to deployments for filtering, reporting, and policy. The Tags API gives you a GET for the current tags and a PUT to replace them, both in one targeted call.

curl -X PUT https://api.elastic-cloud.com/api/v1/deployments/{deployment_id}/tags \
  -H "Authorization: ApiKey $EC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"tags": [{"key": "env", "value": "prod"}, {"key": "owner", "value": "search-platform"}]}'

PUT replaces all tags on the deployment. Validation enforces up to 64 tags per deployment, keys up to 32 characters, and values up to 128 characters, with explicit 400 errors on violations. Authorization follows the same model as the deployment update endpoint.

How to link snapshot repositories across Elastic Cloud Hosted deployments

Snapshot repository APIs: Full CRUD at /deployments/{id}/elasticsearch/{ref_id}/snapshot/repository, plus a new Elastic Cloud console workflow

A common operational pattern in ECH is needing one deployment to access snapshots from another: migration validation, populating a staging environment, or selectively restoring one tenant's indices across a fleet. Snapshot repository linking lets you do this without copying full snapshot data into a new deployment.

The capability creates a link between a source deployment's managed snapshot repository and a target deployment. The target can browse the source's snapshots and restore only what it actually needs; nothing is copied at link time. Cross-bucket credentials and the repository are wired up for you, given your user has an admin role over both deployments.

This works specifically and only for Elastic Cloud managed snapshot repositories, whose credentials are managed by Elastic Cloud.

You can manage these links in two ways: from the Elastic Cloud console UI for ad hoc, point-and-click work, or via the API for automation and infrastructure-as-code workflows.

Link snapshot repositories from the Elastic Cloud console

For one-off restore drills, staging refreshes, or anytime you'd rather not script it, the Elastic Cloud console now provides a guided flow to link a source deployment's snapshot repository into a target deployment. Open the target deployment, go to the snapshot repository management screen, pick the source deployment from the list of deployments you have access to, and confirm. The console handles credentials and registration behind the scenes. No JSON, no keystore wrangling.

Once the link is in place, the same screen lists all linked deployments:

Removing a link is equally direct: Pick the linked repository, and unlink it. The source deployment is unaffected; only the read-only link from the target is torn down.

Manage repositories linking via the API

The snapshot repository linking operations (create, list, delete) are also available as API endpoints for automation and infrastructure-as-code workflows.

Create a link

curl -X POST https://api.elastic-cloud.com/api/v1/deployments/{target_deployment_id}/elasticsearch/{ref_id}/snapshot/repository \
  -H "Authorization: ApiKey $EC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "source_deployment_id": ""
  }'

The endpoint is idempotent. A retry after a partial failure picks up where the previous attempt left off, so you don't end up with half-configured links. The work runs asynchronously: You get an immediate acknowledgment, and the Elastic Cloud control plane handles the rest (fetching the source repository configuration, injecting credentials into the target cluster's keystore, and registering the read-only repository in Elasticsearch). The linking proceeds, provided the caller has the right permissions on both deployments.

List linked snapshot repositories

curl https://api.elastic-cloud.com/api/v1/deployments/{deployment_id}/elasticsearch/{ref_id}/snapshot/repository \
  -H "Authorization: ApiKey $EC_API_KEY"

Each entry in the response includes a repository_name field with the actual name of the repository as registered in Elasticsearch (for example, _clone_abcd1234). You don't need to derive it from internal ID conventions; take the name straight from the response, and use it anywhere you'd reference a repository in the Elasticsearch snapshot and restore APIs.

Remove a link

curl -X DELETE https://api.elastic-cloud.com/api/v1/deployments/{deployment_id}/elasticsearch/{ref_id}/snapshot/repository/{repository_name} \
  -H "Authorization: ApiKey $EC_API_KEY"

DELETE takes the repository_name directly (the same value returned from GET above). That keeps the API symmetric and avoids looking up the source deployment by ID when you're tearing a link down.

A typical lifecycle:

POST   .../snapshot/repository            (link target → source)
GET    .../snapshot/repository            (confirm the link, read back repository_name)
DELETE .../snapshot/repository/{name}     (unlink when done)

This is particularly useful for multi-environment workflows where a staging deployment reads from production snapshots, tenants-per-cluster setups that need selective restores, or multitenanted restore processes that require a specific index restore.

Choosing instance configurations per tier in Elastic Cloud Hosted

Instance configuration customization: Elastic Cloud console UI

Elastic Cloud Hosted now lets you pick an instance configuration per data tier (hot, warm, cold, frozen) directly from the Elastic Cloud Console.

Match deployment hardware to a specific workload by picking an instance configuration per data tier (hot, warm, cold, frozen):

1. Navigate to your deployment's overview page, and click Edit on the hardware profile.
2. In the hardware profile flyout, select the instance configuration that best fits each data tier's workload.
3. Review the changes, and click Update. A confirmation dialog shows exactly what will change before you apply.

Whether you need more storage density on your warm tier, higher compute on hot for search-intensive workloads, or want to move to a newer machine type in your region, you can now make that change from the console instead of crafting API requests. If your selection aligns with an existing hardware profile, that profile is automatically applied. Otherwise, the deployment is labeled custom, and any combination supported by your region is allowed.

The API continues to support instance configuration changes for both data tiers and stateless resources (Kibana, machine learning [ML] nodes, master nodes) for automation use cases.

Full documentation for instance configuration customization is available in the Elastic Docs.

Elastic Cloud Hosted API reference: upgrade, scale, configure, tag, snapshot

These endpoints, plus the new Elastic Cloud console workflows, give ECH a verb-shaped surface for fleet operations: upgrade, scale, set, tag, share, customize. One call per operation, the same validators as the full deployment plan, available from API and Terraform. That's what makes fleet-scale automation practical.

Get started:

• Already on Elastic Cloud Hosted? Generate an API key from your Elastic Cloud account page, and try the examples above. Full reference is in the Elastic Cloud API documentation.
• Prefer infrastructure as code? The Elastic Cloud Terraform provider wraps these APIs for declarative deployment management.
• New to Elastic Cloud? Start a free trial, and deploy your first cluster in minutes.

Vector Search on Elasticsearch Serverless

Elasticsearch Serverless is built on Stateless Elasticsearch, a fully decoupled compute and storage architecture where index data lives in remote object storage and search nodes maintain only a local cache. For vector search to be fast on this architecture, the scoring engine needs to work directly with the local cache, not copy it to the heap first.

Elasticsearch simdvec is the engine behind every vector distance computation in Elasticsearch. It provides hand-tuned AVX-512 and NEON kernels, bulk scoring with explicit prefetching, and off-heap memory access that keeps data flowing from storage straight to CPU registers. On stateful Elasticsearch, simdvec has always had a direct fuel line: Memory-mapped files feed native pointers straight into SIMD intrinsics. On serverless, the data was sitting right there in the blob cache's memory-mapped regions, in exactly the right form, but there was no path connecting it to the scoring engine.

We've now built that path. simdvec runs on Serverless with the same off-heap, native SIMD scoring as stateful. And because serverless gives us control over the entire storage layer, this is just the beginning.

Premium fuel only: why simdvec requires off-heap memory for vector scoring

simdvec's speed comes from working directly with off-heap memory. It takes a native pointer to memory-mapped data and passes it straight to C++ SIMD intrinsics. No intermediate copies, no heap allocations. The data flows from storage straight to CPU registers. This matters more than it sounds: simdvec's kernels process vectors faster than the data can be copied, so any copy in the path becomes the bottleneck, not the scoring itself.

On stateful Elasticsearch, this just works. Lucene memory-maps index files from local disk, and the scorer extracts a native pointer directly from the mapped region. This is the path that delivers the benchmark numbers we've published, and it's what we wanted to bring to serverless. To see how, we first need to understand how serverless stores and accesses data.

The serverless blob cache: how Elasticsearch stores vector data

In the stateless architecture, the primary copy of all index data lives in remote object storage, such as S3. Each search node maintains a local cache (called the blob cache) that keeps recently and frequently accessed portions of the index data on local SSD. The frozen tier on stateful Elasticsearch uses the same architecture: Searchable snapshots are backed by a similar blob cache that memory-maps regions from remote storage onto local disk. When a search hits cached data, it's served from fast local storage. When it misses, the blob cache fetches the data from the remote store and caches it for future queries.

The blob cache is organized into fixed-size memory-mapped regions, 16MB by default. It manages its own lifecycle: tracking which regions are in use, applying a least-frequently-used eviction policy when the cache is full, and reference counting to ensure regions aren't evicted while being read. The regions are still memory-mapped through the OS, but the blob cache controls which regions exist, which are populated, and when they're reclaimed. On stateful, those decisions are left entirely to the OS.

Crucially, because each region is memory-mapped, the blob cache already holds vector data in exactly the form simdvec needs. But before we built the connection, there was no way to get at it. Every vector comparison was copied into a heap array and handed to a slower scorer. No direct memory pointers, no SIMD, and garbage collection pressure on every call.

Unified scoring: one SIMD path for all storage tiers

We introduced a new abstraction that lets the scorer safely borrow direct memory from whatever storage layer is underneath, just long enough to run the SIMD computation. If the data is available as direct memory, simdvec's native kernels run. If not (data not yet cached or spanning a region boundary), the scorer falls back to a heap copy. In practice, the fallback is rare.

This gave us a single scoring entry point across all tiers:

1. Stateful (local disk): The scorer extracts a native pointer from the OS memory map.
2. Blob cache (serverless, frozen tier): The scorer borrows a direct memory slice from a cache region.
3. Fallback: The scorer copies bytes to the heap. Rare in practice.

The scorer doesn't know which tier it's running on, and it doesn't need to. It also means we no longer maintain separate scoring implementations; previously, there was a fast native path for stateful and a slower path for everything else. Now every improvement to simdvec benefits all tiers automatically, including its most powerful capability: bulk scoring.

Bulk vector scoring across blob cache regions

A single query may score thousands of candidate vectors. simdvec's bulk scoring processes these in batches with multi-accumulator inner loops, query amortization, and cache-line prefetching, up to 4x faster than single-vector alternatives when data exceeds CPU cache.

Search over an Inverted file (IVF) index is where bulk scoring has the most impact. The query selects a set of candidate posting lists and sweeps through the quantized vectors, scoring them in large batches against the query vector. On stateful, those vectors live in one contiguous memory-mapped file, so bulk scoring resolves them with straightforward pointer arithmetic and scores a batch in a single native call.

On serverless, a sweep through a posting list may cross blob cache region boundaries. We extended the direct memory abstraction with a bulk access method that resolves multiple vector offsets to their respective cache regions in a single call. If all vectors in the batch are cached and none cross a region boundary, the scorer gets a direct memory slice and passes the whole batch to simdvec's native bulk kernel with the same prefetching and pipelining as stateful. When a vector does cross a boundary, the system falls back to per-vector scoring: still zero-copy, just without the batching benefit. With 16MB regions and 1024-byte vectors, that happens roughly once every 16,000 vectors.

simdvec's bulk scoring architecture, the key differentiator highlighted in the simdvec benchmarks, now operates on serverless with the same characteristics that make it fast on stateful. So how does it perform in practice?

simdvec on Elasticsearch Serverless: vector search lap times

We benchmarked with an 18 million vector MSMARCO dataset at 1024 dimensions, using IVF with Better Binary Quantization (BBQ) 1-bit quantization. All results are on a warm blob cache with the full dataset resident in local cache regions, so we're measuring the scoring path rather than remote fetch latency.

Throughput. Under concurrent load, search throughput nearly doubled, jumping from 398 to 739 ops/s. Single-client gains were 23-39%, but the real difference shows up under concurrency: The improvement was 2-3x larger because eliminating heap copies removes the GC pressure and allocation contention that previously throttled concurrent scoring.

Tail latency. The direct memory path transformed tail latency under load:

• p99.9 dropped from 237 ms to 30 ms (87% reduction).
• p99.99 dropped from 9.1 seconds to 55 ms (99.4% reduction).

p100 dropped from 11.4 seconds to under 100 ms.

The worst-case outliers that previously took seconds now complete in tens of milliseconds. The heap-copy-induced queueing that caused latency spikes is gone.

Recall is identical. The same vectors are scored, producing the same results. And we're just getting started.

Beyond parity: what Elasticsearch Serverless can do for vector search that stateful can't

Reaching parity with stateful was the goal. But the more interesting realization is what the stateless architecture lets us do that stateful can’t.

On stateful, the OS controls memory-mapped file behavior: which pages stay resident, when to evict, how aggressively to read ahead. The application can offer hints, but they apply to entire file mappings, and the kernel may ignore them. Worse, search and indexing happen concurrently on the same node, so a hint that benefits one access pattern can hurt another. In practice, to balance different needs, you have to be conservative.

On serverless, two things are fundamentally different. The blob cache manages its own memory-mapped regions with full application-level control. And serverless separates indexing and search onto dedicated tiers: Search nodes never merge, indexing nodes never serve queries. No conflicting access patterns means we can be aggressive with memory advice. Here’s what we’re working on:

• Per-region memory advice. The blob cache knows what type of data each region holds. It can issue random-access hints for rescoring regions, where raw float32 vectors are read in unpredictable order and the kernel’s default readahead would waste memory on pages that will never be used. It can apply sequential readahead for scans through quantized vectors. On the indexing tier, merges read data sequentially, so aggressive readahead brings pages in before they're needed, with no risk of harming concurrent random reads that simply aren't happening on that node.
• Cache-aware prefetching. simdvec already prefetches at the CPU cache-line level. On serverless, we can coordinate this with the blob cache's knowledge of region residency, prefetching at multiple levels: remote store to cache, OS pages to RAM, and cache lines to CPU. The blob cache can tell the scorer which regions are resident before scoring begins, avoiding work on data that would trigger a remote fetch.
• Workload-aware eviction. The blob cache can prioritize retaining data that vector search depends on: IVF centroid indexes that are checked on every query or quantized vectors that are scored in bulk, over data that's accessed infrequently. The OS page cache evicts based on generic heuristics with no understanding of what the data represents. On serverless, eviction policy can be tuned to the workload.

The blob cache gives us a level of control over the memory hierarchy that the OS page cache simply can’t. This is why we see serverless as the most promising platform for the next generation of vector search performance work. Not just matching stateful, but surpassing it. And vectors are just the beginning.

Vector search on Elasticsearch Serverless: what we shipped and what's next

simdvec now runs everywhere Elasticsearch runs (stateful, serverless, and frozen tier) with the same native SIMD scoring, the same bulk scoring, and the same off-heap efficiency. The abstraction we built is general-purpose and already wired through every layer in the storage chain, so the same approach could benefit term lookups, aggregations, sorting, and stored field retrieval in the future.

Elasticsearch Serverless is where we're investing most heavily in vector search performance. Every improvement to simdvec, every optimization to the blob cache, and every new storage-level improvement lands here first. If you're choosing where to run your vector workloads, serverless is the platform that keeps getting faster. You can get started with a free Elastic Cloud trial.

This post is a deep dive into the implementation. For context on how synthetic _id fits into the broader metrics performance story, see How we rebuilt Elasticsearch as a leading columnar metrics datastore to achieve up to 6.6x improvement in storage efficiency and 50% improvement in indexing throughput for OpenTelemetry metrics.

We'll start by explaining why the _id field is expensive for time-series workloads. We'll then describe how synthetic _id works and how it uses a bloom filter to optimize document deduplications instead of maintaining a traditional inverted index. Finally, we'll share the performance results from our benchmarks and serverless production deployments.

The hidden cost of _id in time-series indices

Time-series indices are a specialized index mode optimized for metrics, logs, traces, and other timestamped data. They store sequences of data points (like CPU usage, stock prices, or sensor readings) that track changes to specific entities over time. In Elasticsearch, each of these data points is indexed as a document with a unique identifier called _id. This identifier is used to look up, update, or delete specific documents. When a document is indexed in Elasticsearch, the system checks whether a document with the same _id already exists. Depending on the operation type (op_type), an existing document is either replaced (index) or the new document is rejected (create); the latter is the most common path for metrics ingestion.

To perform this lookup efficiently, Elasticsearch builds an inverted index for the _id field. This inverted index maps each _id value to its location in the index, enabling fast document lookups. Until version 8.11, the _id value was also stored separately in order to be returned in search results and other APIs. From 8.11 and onwards, we optimized Elasticsearch to only store this value temporarily for document replication purposes, the value being quickly merged away and reconstructed on demand.

For many use cases, building the inverted index and storing it is an acceptable overhead. But for time-series data, like metrics or traces, the cost can add up quickly. Our experiments showed that building the inverted index for the field _id adds 6% CPU overhead compared to indexing without it. In some extreme cases, we benchmarked that it could reduce indexing throughput by 25%.

This overhead is especially painful for time-series workloads where data points are typically small (often just a timestamp and a few numeric values) and compress extremely well. The _id field, however, doesn't benefit from the same compression. As a result, the inverted index for _id can represent a disproportionate share of the total storage. In our benchmarks with OpenTelemetry (OTel) metrics, the _id inverted index alone consumed around 5 bytes of the total 25 bytes per data point.

We considered several approaches to eliminate this overhead:

• Stop indexing _id and checking for duplicates: This would be the simplest solution, but without deduplication, duplicate data points could corrupt aggregations. A gauge average, for instance, would be skewed by repeated values.
• Accept duplicates during indexing, deduplicate at query time: This preserves correctness but adds overhead to every query, degrading dashboard responsiveness.
• Deduplicate during segment merges: Duplicates would eventually be removed, but queries on unmerged segments would still return results with duplicates.
• Synthetic _id: Compute the document identifier on the fly from fields that already uniquely identify each data point, and use a lightweight bloom filter for deduplication instead of a full inverted index.

We chose synthetic _id because it maintains correctness at ingest time while eliminating the storage and CPU overhead of the traditional approach. And we decided to implement it for time-series indices because they’re very well suited for this optimization.

In time-series indices, the _id isn’t arbitrary. Each document has a time series identifier (_tsid) and a timestamp (@timestamp). The _tsid is generated from the dimensions fields of the document (like host.name, pod.name, or sensor_id), while the @timestamp marks the point in time of the document. Together, these two fields uniquely identify the document: There can only be one data point for a given time series at a given moment in time. This means we can derive the _id from the _tsid and @timestamp field values, rather than storing it separately.

How does synthetic _id work in Elasticsearch?

With synthetic _id, Elasticsearch computes the document identifier on the fly as the combination of the _tsid and @timestamp fields. This computed value is used wherever _id would normally be used: in API responses, for document lookups, and for deduplication. However, it’s never stored in an inverted index nor is it stored on disk for later retrieval.

The challenge is deduplication. When a new document arrives, Elasticsearch must verify that no document with the same _id already exists. Without an inverted index on _id, how can we perform this check efficiently?

How synthetic _id simulates an inverted index without building one

Our Elastic Lucene experts suggested a clever idea: Since _tsid and @timestamp are already stored as doc values, we could expose our own custom Lucene postings format that simulates an inverted index without actually building one.

This means that when Elasticsearch needs to look up a document by its _id, it uses the same code path as usual: It queries the underlying Lucene index to look up the _id term. But instead of hitting a real inverted index, our custom postings format intercepts the call, extracts the _tsid and @timestamp encoded in the synthetic _id, and uses their doc values to locate the document. Because time-series indices are sorted by these fields, documents belonging to the same time series are stored contiguously. This allows Elasticsearch to skip large subsets of nonmatching documents (sometimes entire segments) to find the target document(s) quickly.

While this process is efficient, it can involve several random-access reads: looking up the _tsid value, scanning for matching documents, and reading timestamps. For the common case in time-series indices where we don’t expect the document to already exist, we wanted to fail fast without touching doc values at all.

Bloom filters for fast membership testing

We solve this problem using a bloom filter, a probabilistic data structure that can quickly answer the question Could this element be in the set? with a small risk of false positives but no risk of false negatives. In other words, a bloom filter might occasionally say yes when the answer is actually no, but it will never say no when the answer is yes.

When a document is indexed, its synthetic _id is added to the bloom filter. When a new document arrives, we first check the bloom filter. If the bloom filter says no, we know for certain that no document with this _id exists and we can proceed with indexing immediately. If the bloom filter says maybe yes, we fall back to the more expensive verification using the _tsid and @timestamp doc values.

Synthetic _id indexing workflow: step by step

Let's walk through what happens when a document is indexed into a time-series index with synthetic _id enabled:

1. Compute the synthetic _id: Elasticsearch calculates _id as a combination of _tsid || @timestamp.
2. Check the live version map: Like today, we first check an in-memory map of recently indexed documents. If the document is present in this map, we can handle the duplicate immediately.
3. Filter segments by timestamp: Time-series indices are sorted by _tsid and @timestamp. We can skip any segment whose timestamp range does not overlap with the incoming document's timestamp.
4. Check the bloom filter: For each candidate segment, we test whether the _id might exist using the bloom filter.
5. Verify if needed: If the bloom filter returns a positive result, we look up the document using the _tsid and @timestamp doc values. Since documents are sorted by these fields, this lookup is efficient.
6. Index the document: If no existing version is found, the document is indexed. The _id is added to the segment's bloom filter, but no inverted index is built and the field value is never stored.

In the common case where new data arrives with recent timestamps, step 3 eliminates most segments from consideration, and step 4 quickly confirms that the document is new. The expensive verification in step 5 only happens on bloom filter false positives, which are expected to be rare.

Bloom filter false positive rate: how Elasticsearch keeps it low

One challenge with bloom-filter-based deduplication is controlling the false positive rate without sacrificing the storage efficiency we were after. To size bloom filters effectively, we consider the number of data points in each segment and target both a low false positive rate and a bit set saturation below 50%.

The saturation target serves a specific purpose: When segments are merged, we OR the bit sets rather than rebuilding bloom filters from scratch. This makes merges fast but means the false positive rate converges toward 100% as segments are merged repeatedly. Keeping saturation below 50% before merging buys headroom, delaying that convergence.

The low false positive rate target is justified by access patterns: Recent segments are checked far more often than older ones, since we prune the search space based on data point timestamps. Older, heavily merged segments with degraded bloom filters are unlikely to be checked.

Synthetic _id performance benchmarks: indexing and storage

We ran extensive benchmarks to validate our implementation.

Indexing throughput

A core goal of this effort was to match or improve on existing indexing throughput. In principle, the new approach does less work: Building an inverted index for _id requires hashing each value, building and maintaining complex data structures in memory, and flushing them to disk. These structures must also be reconstructed during segment merges, adding CPU and I/O overhead in high-throughput use cases.

Building a bloom filter isn't free (we still hash each value), but the memory footprint is smaller and there are no complex data structures to maintain or flush. The bloom filter is also cheap to merge: When possible, we simply OR the bit sets together rather than rebuilding from scratch.

The main cost of synthetic _id comes from verifying potential duplicates using doc values. However, this cost is mitigated by two factors: First, bloom filter false positives are rare, so most documents skip this step entirely. Second, time-series indices are sorted by _tsid and @timestamp, which means doc value lookups can skip large blocks of nonmatching documents efficiently.

In practice, that's exactly what we observed. Even accounting for the extra seeks needed to verify matches against the tsid and timestamp when a bloom filter returns a positive, throughput came out comparable or better than before. The savings from not building and merging the inverted index outweigh the occasional cost of a false positive check, as confirmed by our nightly benchmarks:

Storage savings

In our benchmarks with OTel metrics, synthetic _id reduced storage by approximately 5 bytes per data point. For a dataset where documents average 25 bytes per data point, this represents a 20% reduction in storage from this single optimization alone.

These results were soon confirmed by our nightly benchmarks.The chart below shows the storage footprint reduction over time as we enabled the synthetic _id feature on March 19, 2026:

Our standard time series database (TSDB) benchmark showed a reduction from 2.5 GiB to 1.9 GiB (24%). Similarly the time-series downsampling benchmark showed a comparable reduction from 3 GiB to 2.3 GiB (23%).

Another benchmark, more focused on metrics, showed an even better reduction, from 3.0 GiB to 2.0 GiB (34%):

API compatibility

An important design goal was maintaining compatibility with existing Elasticsearch APIs. With synthetic _id, all document APIs continue to work as expected: Bulk, Get, Update, Delete, Reindex, and Update/Delete by Query. This compatibility layer also limited the blast radius of the change, ensuring any issues would be contained to the internal implementation.

When the _id isn’t provided in an API request, Elasticsearch computes it from the _tsid and @timestamp fields. To check if the document already exists, it first queries the bloom filter and, if needed, falls back to doc values. The _id is also synthesized on demand from doc values when returning documents in search results or API responses.

One case that requires special handling is searching or filtering by _id prefix or pattern. Such queries require scanning many documents to find matching documents, and while this works correctly, it incurs a performance penalty compared to a direct _id lookup. We don’t expect this use case to be common for time-series indices though.

Elasticsearch 9.4 and Elastic Cloud Serverless availability

The synthetic _id feature will be released in Elasticsearch 9.4.0 and is already available on Elastic Cloud Serverless.

No configuration is required: The feature is enabled by default, and newly created time-series indices (including those created on datastream rollover) will automatically benefit from this optimization. Existing time-series indices created before 9.4 will continue to create inverted indices for the _id field.

We expect synthetic _id to perform well across all time-series use cases. However, in some very specific, update-heavy use cases, if you encounter performance issues, the feature can be disabled by setting index.mapping.synthetic_id to false for new indices.

Summary: synthetic _id storage and performance gains

In this article, we’ve presented how synthetic _id eliminates the storage and compute overhead of document identifiers in time-series indices. By computing _id on the fly from _tsid and @timestamp, and using a bloom filter for deduplication, we achieve comparable or better indexing performance with up to 34% reduction in storage footprint while maintaining full API compatibility. For users running large-scale time-series workloads, this translates directly into lower infrastructure costs.

Roadmap: what comes after synthetic _id

Synthetic _id is part of a broader effort to reduce storage overhead in Elasticsearch.

• Sequence number trimming: Every document carries a sequence number for replication and concurrency control. For append-only time-series data, these become redundant after segments are merged. Elasticsearch 9.4 now trims them during merges to reclaim even more storage: We'll cover this optimization in detail in an upcoming blog post.
• Synthetic _id beyond time-series: We’re exploring how to bring synthetic _id to regular indices by letting users declare which fields uniquely identify their documents and configuring index sorting on those fields to enable efficient lookups.

Stay tuned!

jina-vlm is a 2.4B-parameter vision-language model that pairs a SigLIP2 vision encoder with a Qwen3 language decoder, using attention pooling over image tiles for token-efficient handling of arbitrary-resolution inputs. Beyond the model itself, the paper's main contribution is its “leave-one-out” ablative data-mixture: By removing one task, domain, modality, or language category at a time during training, you can figure out which slices of data are significant or redundant, and whether learning in one domain transfers to others. The result is a compact model that, despite its size, achieves state-of-the-art multilingual VQA performance.

Rio delivered everything you'd hope for: warm, sunny beach weather, the easy walk between Copacabana and Ipanema, the view from Christ the Redeemer, the colors of Escadaria Selarón. A welcome contrast to a still-chilly European spring.

What was trending at ICLR 2026: RLVR, test-time compute and retrieval

Conferences like ICLR give everyone a chance to take the field’s pulse and find out what’s hot, what’s not, and what’s coming up. After a few days of walking the aisles at the poster sessions and dropping in on oral sessions, you start to get a sense for things. You start to see the same words on poster after poster, and you notice which sessions are the most crowded.

Here are a few things we picked up on:

Reinforcement Learning with Verifiable Rewards (RLVR) is now the dominant paradigm for post-training refinement. Almost every reasoning-focused poster we stopped at was using some form of Group Relative Policy Optimization (GRPO) for math correctness, code execution, and formal-logic checks, rather than Reinforcement Learning from Human Feedback (RLHF). Direct Preference Optimization (DPO) fine-tuning, which felt like the default a year ago, was conspicuously rare. It makes sense: If you can use code to check for correctness, you no longer need to get annotated data and the training loop goes much faster.

Test-time compute has stopped being a curiosity and become a design problem. Test-time compute – the time a system spends generating a response – is an increasingly important study variable. Papers now measure it as part of their experimental setup and developers try to optimize for it. Models are now built with the expectation that inference will be expensive and clever, not just a single forward pass through a neural network.

Vision-Language Models (VLMs) are everywhere, and Vision-Language-Action models (VLAs) are not far behind. A big chunk of the conference was about how to make multimodal AI work better, like better tokenization for images, better positional encodings for non-text media, and more efficient ways to compress visual information before it overwhelms your model. Vision-Language-Action models that extend multimodal AI recipes to robotics and embodied agents are no longer niche research. They brought in the crowds at their presentations and hosted vibrant debates.

Reports of the death of State-Space Models (SSMs) have been greatly exaggerated. Attention models still dominate AI, but Mamba, SSM variants and recurrent neural networks still draw attention and research, both as full replacements for Transformers and as components inside hybrid attention-based stacks. Whether they'll ever genuinely displace Transformers is an open question, but the line of research is alive and well.

Agentic AI safety is taken very seriously. A lot of papers and presentations discussed problems like machine unlearning and jailbreaking, and some of the most interesting work was on prompt injection through agentic tool use, like when a model dutifully follows instructions hidden in a webpage or an API response it just fetched. A repeated, slightly unsettling observation: models that follow instructions better tend to be more vulnerable to this kind of attack, not less. This capability-vulnerability tension is going to define a lot of the next few years of safety research.

Hallucination and factuality are increasingly framed as retrieval problems. Several talks made that point explicitly: A generative model that has to invent facts will inevitably hallucinate them, while a model that retrieves information can ground its responses in verifiable ways. That framing is, of course, exactly the bet that search AI engineers have been making all along.

ICLR 2026 invited talks: hidden universe imaging and open AI development

Two of the invited talks stood out to us, albeit for very different reasons:

Images of the Hidden Universe

Katie Bouman presented a tour of how physics, prior knowledge, and machine learning combine to reconstruct information that the universe never gives us directly, like the silhouettes of supermassive black holes and the invisible dark matter structures. She walked us through the Event Horizon Telescope's imaging of M87 and Sagittarius A, building images up from indirect and incomplete radio measurements, and then extended the same machinery to mapping dark matter through gravitational lensing.

This talk was a useful reminder of why machine learning matters outside the LLM bubble. The more you already know, the more you can learn from a little bit more information. This principle generalizes beyond astronomy to knowledge in general, and to machine learning in particular. Any decision system that uses sparse, noisy observations is confronted with it.

_____________________________________________________________________________________

Marin: Open Development of Frontier AI

Percy Liang opened his presentation with a blunt observation: As AI capabilities skyrocket, openness plummets. His response is Marin, a platform for community-driven AI research where every experiment is open, every suggestion or discussion is on public fora, and anyone can review or rerun a result.

What makes Marin interesting isn't just creating open weight models - plenty of projects do that - but creating an open process for making models. Project pre-registration, peer review, and reproducibility have long been part of the natural sciences, and Marin attempts to maintain that tradition for AI. Model training is treated as a matter of public scientific record.

The talk presented concrete scientific results from this approach (optimizer findings and scaling-law results), suggesting that community-scale science isn't just an aspiration but a workable methodology.

_____________________________________________________________________________________

Bouman and Liang made a pleasingly complementary pair: one a reminder of how much ML has to offer the world outside ML, the other a challenge to how the field organizes itself.

ICLR 2026 research highlights: embedding models, retrievers and sparse representations

We attended many oral presentations and poster sessions. The papers below stood out because of their potential to impact how we make and use embedding models.

Rethinking pretraining for representations

Decoder-only models have dominated the LLM leaderboards for years, but one paper makes a case for encoder models.

Seq vs Seq: An Open Suite of Paired Encoders and Decoders does a repeatable, open-data, architecture-controlled comparison of encoder-only and decoder-only models trained identically. They used the same data, same architecture, same training recipe, and differed only in their training paradigms: Bidirectional Masked Language Modeling (MLM), typically associated with encoders, vs. Causal Language Modeling (CLM), usually used in decoders. Their results confirm prior findings that encoders excel at classification and retrieval while decoders excel at generation. A key finding is that cross-objective continuous pretraining does not close the performance gap between the encoders and decoders. A 400M parameter encoder beats a 1B parameter decoder in classification and retrieval, and vice versa for generative tasks. All artifacts including data, checkpoints, and code are open-sourced.

Their study delivers a definitive empirical finding for the AI community: Encoder-only pretraining is substantially more efficient for classification and retrieval tasks than adapting decoders to act like encoders, even with post-training on high-quality data. This challenges the recent trend of adapting large decoder LLMs (like LLM2Vec) for embedding tasks. Dedicated encoder pretraining from scratch remains the most reliable path to strong retrieval performance. Additionally, the public release of 200+ checkpoints with batch-ordered training data makes their work an invaluable resource for studying how retrieval-relevant representations emerge during training and how they scale with parameter count and tokens.

New paradigms for training retrievers and embedders

Revela: Dense Retriever Learning via Language Modeling reframes dense retriever training as a language modeling problem. Rather than using supervised training with query-document pairs, it trains a retriever model jointly with a language model by conditioning next-token prediction on all the other documents in the batch. This innovative in-batch attention mechanism modifies the model’s Transformer blocks by injecting the similarity scores of documents in each batch into the cross-document attention weights. Training is done on raw text, without query-document pairs, hard negatives, or synthetic data generation. The resulting 3B parameter model outperforms E5-Mistral-7B-Instruct (with 7B parameters) as well as proprietary closed-weight embedding models like OpenAI, Cohere, and Voyage. On retrieval benchmarks, it matches E5 despite using roughly 1000 times less training data and approximately 10 times less compute.

This demonstrates that next-token prediction can still serve as an effective training objective for high-quality dense retrieval AI. This is important because plain text data – what you need for next-token prediction – is widespread and inexpensive and this paper shows that it’s all you need to train competitive embedding models.

Let LLMs Speak Embedding Languages: Generative Text Embeddings via Iterative Contrastive Refinement advances the proposition that LLMs should learn to "speak an embedding language," i.e., generate sequences of “soft tokens” optimized for semantic representation rather than human readability. They outline innovative loss functions and objectives in support of this goal, and show that the resulting models have very competitive performance, while generating only a handful of additional tokens.They also show that generating more tokens at inference time steadily improves embedding quality in a way analogous to chain-of-thought scaling in reasoning LLMs. KV-caching reduces the computational overhead of the generation process to within 1.1 times that of standard single-pass embedding models. This approach represents a new paradigm for representation learning, complementary to encoder-only and single-pass approaches.

Towards Improved Sentence Representations using Token Graphs frames the problem of generating embeddings for sentences from token-level representations as a relational learning problem rather than a compression problem. Instead of pooling tokens, it uses a supplementary neural network that processes a dynamically constructed graph made from output token similarities. This added network is compact, with very few trainable parameters, and can be implemented without doing any additional training on the main language model. The result is competitive with current frontier models.

This approach can be dropped into any language model at a very reasonable additional training cost, giving it immediate practical significance. Furthermore, the resulting models hold up well in the presence of noise, a known problem, especially for long-context models.

Sparse and ultra-efficient embeddings

LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query Inference introduces an asymmetric dual-encoder architecture for embeddings-based retrieval in which the query encoder is much smaller and faster than the document one. The key insight is that while document embeddings benefit from the modeling power of a large language model, query embeddings are much less demanding. During training, they propose to learn per-token query embeddings, then, at query time, those embeddings are retrieved and averaged to produce a full query embedding. Documents must still be encoded at storage time using a potentially large encoder, but there is no need to invoke an embedding model at query time at all. The result retains approximately 95% of the performance of the query encoder it replaced. This has immediate implications for computational constrained, time-sensitive, or resource-efficient text information retrieval systems.

CSRv2: Unlocking Ultra-Sparse Embeddings addresses the computational cost of embeddings-based retrieval using dense, high-dimensional vectors. It tackles that cost with Contrastive Sparse Representation (CSR), which maps dense vectors into a much higher-dimensional space where only a few vector entries are non-zero, so that search can use highly efficient sparse-vector search techniques like inverted-indexes.

CSR approaches tend to break down when the number of dimensions with non-zero values becomes very low. This paper addresses this problem with an innovative training approach that makes ultra-sparse representations viable, opening up the possibility of much faster, less computationally demanding retrieval without loss of accuracy.

Multi-step and multimodal retrieval

Q-RAG: Long-Context Multi-Step Retrieval via Value-Based Embedder Training frames the problem of multi-step retrieval-augmented generation (RAG) in terms of optimizing the embeddings used in RAG search. RAG systems are typically based on a single retrieval step: Input to an LLM becomes a query to a vector store, and a selection of the results are presented to the LLM as a basis for composing a response. However, agentic approaches that involve multi-step interactions between the LLM and vector store can improve RAG performance significantly, especially for large input contexts that might contain millions of tokens. This paper seeks to optimize the embedding model used for retrieval to better support this usage scenario with Reinforcement Learning with Verifiable Rewards (RLVR).

This paper is one of the more elegant intersections of two of the conference's biggest themes — RLVR and retrieval — and it gives a glimpse of what retrieval looks like when it has to operate inside an agentic loop, not just before one.

Foundations and evaluation

HUME: Measuring the Human-Model Performance Gap in Text Embedding Tasks undertakes the unusual task of systematically measuring human performance on the Massive Text Embedding Benchmark (MTEB), the most widely used benchmark for embeddings-based information retrieval. Using 16 datasets in 5 languages, they find that average human retrieval accuracy is 77.6%, while the best embedding models currently score over 80%. However, this performance gap is uneven. Models may outperform humans on standard tasks but fall apart when faced with low-resource languages, where human intuition still holds a significant lead.

This paper also shows that "superhuman" scores on low-agreement tasks are mostly artifacts of fitting noise, not genuine capability. This underlines the problem of our current suite of embedding benchmarks: New models are not improving benchmark performance very much. To make progress, we need new, harder challenges and a total rethink of how we evaluate models.

Training dynamics for foundation models

How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining identifies a significant but underexplored problem in AI model training. Large training sets can create a problem with models forgetting things that they’ve learned as they’re presented with more data. Curriculum-based pretraining — sorting data from low to high quality — should help, but in practice the results have been disappointing. The reason, the authors argue, is that the model encounters the highest-quality data late in the training schedule when the learning rate is at its lowest. Its gradient contribution is therefore greatly reduced. They confirm that hypothesis empirically by showing that curriculum training significantly beats random shuffling if training uses a constant learning rate.

They propose two simple strategies to fix this: Let the learning rate decay more slowly, or replace learning rate decay with weight-averaging over the multiple final checkpoints. Combining the two yields a 1.64% average benchmark improvement over standard practices with no additional data refinement. The broader message - that data composition and optimization schedule need to be co-designed - applies well beyond pretraining, and is a useful frame for embedding training too.

What ICLR 2026 means for retrieval and embedding research

Science has always been conducted through print and publication, but in-person conferences are still the only way to put people together in a room. Over five days, we met a steady stream of researchers from very different backgrounds — academia and industry, large labs and small startups, half a dozen countries — and conversations ranged from research trends to philosophical questions that have haunted AI from the beginning. Are LLMs really reasoning, or are they doing something more like very high-dimensional memorization with interpolation? Where exactly is the line, and does it matter for what we can build on top of them?

These conversations rarely produce answers, but they sharpen the questions, which is most of what good research is.

For the information retrieval work we do at Jina by Elastic, the future looks bright. Retrieval, long relegated to merely applied research, is increasingly recognized as the engine for keeping language models grounded. Better encoders, better embedding training paradigms, sparser representations, and retrieval that operates at the core of reasoning loops – these things matter to us all. What we saw and heard at ICLR 2026 convinces us that this is where a meaningful share of the next round of progress will come from.

We're already looking forward to seeing where the field is next year.

How DiskBBQ uses two centroid tiers for asymmetric quantization

DiskBBQ now uses two centroid tiers (fine-grained document centroids and coarser query centroids) so queries are quantized once per parent centroid instead of once per document centroid.

The old mental model is "one centroid does everything for a posting list." The new model splits responsibilities:

• Document centroids (fine-grained): Still used for posting-list structure and document centering.
• Query centroids (coarser): A parent centroid reused across multiple document centroids.

So instead of quantizing the query independently for every document centroid we visit, we quantize per parent centroid and reuse that work across all of its children. Since we were already using two-tier clustering logic as the index size grew, it was a natural fit. We can reuse the work we already do during querying.

These images are a simple representation of our goal: Quantizing per centroid gives us overhead per centroid. Let’s get rid of it!

The goal is to significantly reduce the number of times we actually need to quantize a given query.

The math behind asymmetric BBQ in Elasticsearch

To center the data prior to computing quantized query and document vectors, $q$ and $d$, we rewrite the dot product $q^td$ as $(m+q-m)^t(m+d-m)$ and expand. We can perform exactly the same operation but using different centroids for the query vector $q$ and document vector $d$. Specifically,

As for standard Better Binary Quantization (BBQ), we quantize $q-m_q$ and $d-m_d$ in order to estimate the per (document, query) pair component of the dot product. The quantities $m_d ^tq$  and $m_q ^t(d-m_d)$ are scalars so just two extra additions per dot product we compute. For $m_d ^tq$, we compute naturally when finding the nearest centroid. For $m_q ^t(d-m_d)$, this can be stored with the quantized document vectors, which are just 4 bytes overhead. Below, we’ll discuss how to manage the other term on the fly.

Asymmetric BBQ in DiskBBQ

We cluster the document centroids (using k-means, for example) into $k_q<k_d$ clusters, for $k_q$ and $k_d$ the query and document centroid count, respectively. This means there’s a many-to-one mapping from document centroids to query centroids. We’ll denote the document centroids by their index $i ∈ [k_d]$ and define this mapping to the query centroids as $j : [k_d]$→$[k_q]$.

Since there’s a unique query centroid for each document centroid, we only need to cache one value for $m_q^t(d-m_d)$ per quantized document vector, that is, for each document vector $d_k$ in posting list $i$, we need to cache $m_j(_i)^t(y_k-m_i)$ with the quantized document vector.

When we come to compute the dot products between a query and the document vectors in a cluster, we look up the quantized query vector corresponding to $q-m_j(_i)$ and we compute $m_i^tq$ once and use it to process the whole posting list. The quantization process is significantly more expensive than computing the dot product, so this is a big net win.

The $(q-m_q)^t(d-m_d)$ term is estimated using the usual BBQ machinery, that is, these vectors will be quantized and the dot product value estimated from the quantized vectors. Then we can use (1) to compute the final dot product estimate. Notice that this means we only need to quantize the query at most $k_q$ times. Furthermore, we typically visit many centroids from the same parent centroid in a search because they’re close to one another.

Euclidean distance corrections for asymmetric quantization

For Euclidean, we can write $||q-d ||^2=||q||^2+||d||^2-2q^td$ and treat the $q^td$ term exactly as above. In fact, there’s a slightly nicer form. Substituting, we have that:

We can rewrite this as follows:

The corrective terms are the norm of query vector $q$ minus the document centroid $m_d$, the norm of the document vector $d$ minus the query centroid $m_q$, and the norm of the difference of query and document centroids. As before, $||d-m_q||^2-||m_d-m_q||^2$ can be stored as a single float with each document.

What changed in DiskBBQ indexing and scoring

At indexing/merge time, centroids can be clustered into parent groups when centroid count is large enough. Posting metadata moved from "centroid ordinal + centroid score" to a shape that explicitly carries query-centroid ordinal and document-centroid score. That decoupling is what lets scoring read documents and query centering from different places. For Euclidean, let’s break it down further by our mathematics above:

$||q-m_d||^2$ <- This is the distance from a “query vector $q$” to “document centroid $m_d$”. We already gather this when we find the nearest centroids during querying. No new work.

$||d-m_q||^2$ <- This is the distance from “document vector $d$” to “query centroid $m_q$”. However, recalling our original quantization work, this can simply replace a previously stored float value. No new storage is required.

$||m_d-m_q||^2$ <- This is just the distance between query centroid $m_q$ and document centroid $m_d$. This is just a single extra floating point value per postings list.

The practical change for dot product spaces is even simpler; the only correction value change is $m_j(_i)^t(y_k-m_i)$ being stored instead of $y^tm_i$.

These changes don’t introduce new computation costs and marginally reduce storage costs because we no longer quantize queries with document centroids. Those raw centroids don’t need to be present with the posting lists.

One cost we did add is a small cache of quantized query values. This is to account for clustering edge cases. For example, it's possible that query $q$ is very close to query centroid $qc_0 =\{dc_0, dc_1, dc_2\}$ but not quite as close as $qc_1 = \{dc_3, dc_4, dc_5\}$. That said, the actual nearest three document centroids could have a relative order: $[dc_0, dc_3, dc_1]$. So, to prevent the query from being quantized twice, we keep a limited cache of the most recent quantized values for a given query.

Here’s a visualization of the situation described above. In the typical iteration scenario, we don’t want to risk unnecessarily quantizing the query against the same query centroid multiple times.

DiskBBQ asymmetric quantization: performance results

The flame graphs below show a before and after comparison. Before, about 20% of the time was spent quantizing queries when we visited each cluster. After our adjustment, it dropped to about 4%.

Of course, the bulk of the cost is still just scoring the vectors in each cluster. But every little bit helps.

Here’s a better view of the full end-to-end performance and recall. The data set was 1 million DBpedia docs encoded with the GTE-Base model. Here, “sec” indicates the number of clusters per secondary (parent) cluster. Note that symmetric quantization is still impacted by the secondary cluster size as it also impacts the two-tier clustering indexing we do already.

However, the impact on our current index structure is still dominated by centroid scoring and scoring vectors in the cluster. Asymmetric quantization removes a frustratingly expensive part of our scoring overhead, but the impact isn’t dramatic given our current structure.

What's next for DiskBBQ quantization

This simple piece of mathematics decouples our query quantization from our document quantization, giving us better storage efficiency and faster queries. This is in Elasticsearch Serverless now and will be in Elastic Stack version 9.4.0.

This now means that query quantization time isn’t a direct concern for future decisions. We can make larger index changes without worrying about the consistent overhead of quantization directly with document centroids.

This was a nerdy one. I hope you survived all the math (and that I copied it all down correctly). It’s always fun to be able to tackle complex problems with simple mathematics, and the results are actually positive in real use cases and data.

Why do dashboards need an API?

The existing Saved Object API exports dashboards as a single stringified JSON blob: visualization state, internal references, UUIDs, and metadata all tangled together. Changing one field in a Kibana Lens chart means parsing 200 lines of internal state that was never meant to be read. There's no clean way to diff it, review it in a pull request, or edit it programmatically.

That makes standard GitOps workflows impossible: no drift detection, no automated promotion across environments, no rollback to a known-good state without overwriting the whole document by hand.

For large language models (LLMs), it's not easy either. The Saved Object format is too complex and error-prone for language models to generate or modify reliably; a simpler, validated schema is a prerequisite for natural-language dashboard authoring, which is exactly how we're using it in the Elastic AI Chat and dashboard skills for third-party tools integration.

What the new Dashboards API provides

The API works with your existing dashboards; it's not limited to dashboards created through the new interface.

Five endpoints cover the full lifecycle: 

• Create a new dashboard (POST /api/dashboards).
• Read an existing dashboard (GET /api/dashboards/{id}).
• Update an existing dashboard or create a new one if the ID doesn’t exist (upsert) (PUT /api/dashboards/{id}).
• Delete a dashboard (DELETE /api/dashboards/{id}).
• List all existing dashboards with pagination and query parameters (GET /api/dashboards).

What makes it different from the legacy Saved Object API:

• Typed schemas for each panel. Every visualization type (XY chart, metric, pie, gauge, heatmap, data table, treemap, mosaic, waffle, tag cloud, region map) has its own validated schema with sensible defaults. Markdown panels, controls, and drilldowns are also supported.
• ES|QL and data view support. Each visualization can be backed by a data view or an Elasticsearch Query Language (ES|QL) query; the schema cleanly separates the two modes.
• Dashboard-level filters. Condition filters (is, is_one_of, range, exists), group filters (and/or nesting), raw Query DSL filters, and spatial filters, all typed and defined at the dashboard level.
• Structural validation on write. POST and PUT validate your definition up front, so errors surface at write time rather than render time.
• Layout control. Panels specify grid position on a 48-column grid; dashboards can be organized into collapsible sections.
• Library panels. Panels can exist only as part of the dashboard, or they can be saved in the library if they need to be reused in multiple dashboards. The API supports both types of panels.

How the Kibana Dashboards API was built: the transforms layer

At the outset of this project, it was quickly discovered that the biggest culprit to the unusability of the Saved Objects APIs was the stringification of the dashboard contents. The dashboard’s entire panels array, for instance, was stored in one key, called “panelsJSON,” which could be thousands upon thousands of characters long, all stuffed into one line. We lovingly dubbed these fields “JSON Bags.”

If we could stop the stringification process for these keys and instead store the JSON directly in Elasticsearch, all of our problems would be solved. Unfortunately, after we de-stringified the JSON and truly saw what was inside, we found a jumbled mess.

Kibana has stored user-created UI content since day one. This content was stored with exactly one purpose in mind: allowing the UI to restore that content later. Because the actual shape of the storage was inconsequential, it was stored in Elasticsearch as a direct snapshot of the UI state. UI state is optimized for the specific UI that created it and is not meant to be looked at. Structures like deep nesting, objects with randomly generated IDs, parallel arrays, and internal references between different portions of the state are helpful in UI state but detrimental to readability.

Here, we faced a choice:

• Leave the existing UI code in place and build a new alternative API schema for the public endpoints that bypasses the existing dashboard system and allows for some of its most-used functionality.

OR

• Inject a new API schema into the existing dashboard system and build middleware to allow the same API schema to underpin both the UI and the public endpoints.

We chose the second option and set about the monumental task of crystalizing and redefining every key in every panel of the jumbled mess of existing UI state into something we would be comfortable publishing in a formal API schema. This process was completed over a long time frame, by multiple teams, in live code. Backward compatibility was of utmost importance as, throughout the entire process, dashboards were being saved and loaded by our users every day.

The middleware that made this possible is called the transforms layer; it’s responsible for translating back and forth between the legacy, stringified, state shape and the cleaner API shape. Transform functions are registered for each panel that a dashboard can host. The teams that owned those panels would iteratively improve on the schema and make matching changes to the UI with each PR to ensure that the UI continues to operate correctly.

As each panel registers its transform functions with the dashboard, it also has the ability to register a schema that becomes part of the contract of the public Dashboards API. Schemas registered like this must be fully complete to reduce the risk of breaking changes to the API in the future. If a panel type doesn’t have a schema registered, it will be stripped from public API responses. This allows us to iteratively include more content in the public APIs into tech preview and beyond, while ensuring that users who rely on the API can trust that the shape will not change.

Using the Kibana Dashboards API: a walkthrough

The Kibana Dashboards API is available in Elastic 9.4 and works with existing dashboards.

The best place to get started is the Dashboards API documentation, where you’ll find all schema definitions and examples.

The following example uses the Kibana sample logs data to walk through the API; not a realistic GitOps workflow, but a simple way to see how it works.

• Open a dashboard, and click Export JSON from the top menu. This dashboard contains one control and two sections: one with two metrics and another one with two time series.

• You’ll see the JSON of this dashboard displayed in a flyout. Now click Open in Console.

• You’ll be redirected to the Developer tools, and a new POST request is automatically prefilled for you with the HTTP request.

• Include the space ID in the POST request URL.

POST kbn:/s/production/api/dashboards?

• Go to the destination space, and see how the dashboard has been created.

Terraform support: Dashboards as native HCL

The Elastic Stack Terraform provider ships a new elasticstack_kibana_dashboard resource that maps the Dashboards API to native Terraform HCL. This means you get terraform plan, terraform apply, drift detection, import, and all the standard Terraform lifecycle operations for dashboards.

The provider doesn't cover every API capability yet; it focuses on the features that matter most for GitOps workflows.

Terraform example

Example: a Kibana dashboard in Terraform HCL

Start with the basics: a dashboard with a time series chart tracking request counts, a metric panel showing a KPI, and a markdown panel for context.

resource "elasticstack_kibana_dashboard" "service_overview" {
  title       = "Service Overview"
  description = "Key metrics for the payments service"
  tags        = ["production", "payments"]

  time_range       = { from = "now-1h", to = "now" }
  refresh_interval = { pause = false, value = 30000 }
  query            = { language = "kql", text = "service.name:payments" }

  panels = [
    {
      type = "vis"
      grid = { x = 0, y = 0, w = 32, h = 15 }
      xy_chart_config = {
        axis = {
          x = { title = { visible = true } }
          y = {
            title       = { visible = true }
            scale       = "linear"
            domain_json = jsonencode({ type = "fit" })
          }
        }
        legend      = { visibility = "visible", inside = false, position = "right" }
        fitting     = { type = "none" }
        decorations = { fill_opacity = 0.3 }
        query       = { language = "kql", expression = "" }
        layers = [{
          type = "area"
          data_layer = {
            ignore_global_filters = false
            sampling              = 1
            data_source_json      = jsonencode({ type = "data_view_spec", index_pattern = "logs-*" })
            y = [{
              config_json = jsonencode({ operation = "count", empty_as_null = true, color = { type = "auto" } })
            }]
          }
        }]
      }
    },
    {
      type = "vis"
      grid = { x = 32, y = 0, w = 16, h = 15 }
      metric_chart_config = {
        ignore_global_filters = false
        sampling              = 1
        data_source_json      = jsonencode({ type = "data_view_spec", index_pattern =   "logs-*", time_field = "@timestamp" })
        query                 = { language = "kql", expression = "" }
        metrics = [{
          config_json = jsonencode({
            type = "primary", operation = "count", empty_as_null = false,
            color = { type = "auto" },             format = { type = "number", decimals = 2, compact = false }
          })
        }]
      }
    }
  ]
}

This is readable, reviewable, and diffable. When someone changes the time range or adds a filter, the terraform plan output tells you exactly what changed.

Dashboard access control in Terraform

The Terraform provider supports the dashboard access control model, letting you restrict write access and set ownership:

resource "elasticstack_kibana_dashboard" "protected" {
  title = "Production SLOs"
  # ...

  access_control = {
    access_mode = "write_restricted"
  }
}

When access_mode is set to "write_restricted", only the creator can make changes. This is especially useful for production dashboards where you want to ensure all changes flow through your Terraform pipeline.

The GitOps workflow for Kibana dashboards

With the new Dashboards API and Terraform support, you can now treat dashboards like any other infrastructure artifact:

1. Define dashboards in HCL, alongside your Elasticsearch indices, data views, and alerting rules.
2. Review changes through pull requests: terraform plan shows exactly what's changing in each visualization.
3. Deploy across environments using Terraform workspaces or variable files per environment.
4. Detect drift when someone edits a dashboard manually in the Kibana UI: terraform plan will show the difference.
5. Roll back by reverting to a previous commit and running terraform apply.

Elastic users can now enjoy the benefit of GitOps-enabled dashboards, with a typed, HCL-native experience that goes beyond just managing dashboards.

Getting started with the Terraform provider

The elasticstack_kibana_dashboard resource requires Elastic Stack 9.4 or later and is available in the latest versions of the Elastic Stack Terraform provider.

To get started:

• Set up your provider with Kibana connectivity:

terraform {
  required_providers {
    elasticstack = {
      source  = "elastic/elasticstack"
      version = "~> 0.14"
    }
  }
}

provider "elasticstack" {
  kibana {}
}

• Define your first dashboard using any of the examples above.
• Run terraform plan to preview, and then run terraform apply to create.
• Import existing dashboards into Terraform state:

terraform import elasticstack_kibana_dashboard.my_dashboard /

For the full resource schema and documentation, visit the Terraform Registry.

Kibana Dashboards API roadmap

The Dashboards API is in technical preview in Elastic 9.4, and both the API and Terraform provider are actively evolving. On the roadmap:

• All panel types will be supported by the API. There are a few panels that are still missing API support, like the Links panel, Machine Learning panels, Alerts, Log analysis panels, Vega visualization, and Maps.
• More panel types in Terraform. Typed HCL config blocks for image, links, Service Level Objective (SLO), and Synthetics panels that the API already supports.
• Dashboard-level filters in Terraform. Filter pills, controls, and drilldowns that the API already supports.

Try it out and let us know what you think. Send your feedback by using the Submit Feedback icon in the top menu, file issues on the Terraform provider GitHub repo, or join the conversation on Discuss. We’re looking forward to hearing from you!

Using the BrowseComp-Plus public dataset and a cost-controlled agent harness, accuracy moved from 60% to 70% to 92% across three stages of iterations, with input tokens dropping by up to 75% versus standard RAG. Most of the final jump came from feeding the agent's own wrong answers back into the extractor to create new KIs.

Pre-computed context only works inside a system that retrieves well, manages the data over time, and learns from its own failures.

Agents stall before they reach the answer

Frontier models are extremely capable when armed with the ability to access sources of information — they can crawl web pages, parse spreadsheets, navigate logs, run queries. The challenge is doing this without running out of tokens before arriving at the answer.

This challenge is familiar to engineering leaders who are building agents with access to data. The agent gets a task, decides it needs information, searches, retrieves, evaluates, decides it needs more, searches again, reads, stitches together a partial picture, loops. By the time the model is ready to answer, most of the token and latency budget is gone. Sometimes the answer is in the corpus and the context window fills before the agent gets to it. Sometimes the agent picks the wrong thread and never recovers.

OpenAI ran into this building their own internal data agent. Raw data access didn't scale. They had to layer in human annotations, institutional knowledge, and learned corrections before it was reliable enough for daily use. We're seeing the same with customers using Elastic Agent Builder. Model intelligence is rarely the limitation; what breaks agents is the context before the reasoning step.

The shapes of the workloads vary wildly across different domains:

• Log Anomaly Triage: processing of machine-generated system alerts. Because individual data points lack sufficient context, the goal is to extract insights across multiple anomalies to separate benign, seasonal patterns from truly actionable incidents.
• Financial Payment Analytics: querying historical logs to track end-to-end transaction journeys based on fuzzy identifiers. It requires cross-record retrieval to map full service lineages and diagnose payment failures or latency.
• Product Support: assisting customers with questions on products using internal documentation, building insights across multiple documents.

This post covers a strategy we're testing for that problem: do the data orientation work once, ahead of time, and let the agent read the result.

Bottom-up context strategy: extracting knowledge from the source

There are two parts to the approach: extracting context ahead of time using Knowledge Indicators, and giving the agent a clean way to query it.

Use agents to learn how to extract and maintain context effectively

Most of the data an agent needs is already somewhere in the enterprise — records in databases, documents in Google Drive, Confluence, or SharePoint, logs and metrics in Elasticsearch, files in S3. The strategy works against the sources where they already live, with the access controls they already have.

Every domain utilizes sources differently. So instead of creating generic extraction pipelines that treat every source and domain the same, we're planning to use agents to tailor the extraction needs. An agent reads a sample of the source, works out the shape — the schema, the field semantics, the ways the data is typically queried — and writes that understanding out as structured metadata. Some of that metadata is facts pre-computed from the data; some of it describes how to query the source itself.

Document:
  docid: 
  url:   
  text:  
Return a JSON object with key "facts" containing 0–15 atomic facts.
Long, fact-dense documents (Wikipedia articles, news features, profiles,
academic-staff pages) typically warrant 8–15 facts. Short or generic
documents may warrant 0–3.
Each fact MUST be self-contained: title + description together fully answer
the implied W-question (who/what/when/where/how/which) without requiring the
source document. A future agent should be able to commit to an answer by
reading just title+description — the description must include the answer
value, supporting evidence (date, location, named witness, exact quantity,
physical detail), and a short verbatim quote (≤30 words) when it adds
disambiguating signal. This bias toward density is intentional even at the
cost of slightly longer descriptions.
Each fact:
  {
    "title":            "",
    "description": x     "<2-3 sentences ≤350 chars carrying the answer + evidence: entity, relation, value, date/location/source detail, and an inline verbatim quote when it disambiguates. Avoid restating the title verbatim.>",
    "subject":          "",
    "predicate":        "",
    "object":           "",
    "evidence_span":    "",
    "confidence":       <0..100 integer>,
    "tags":             ["", ...]
  }
Coverage priorities — extract a fact for EACH of the following whenever it's grounded in the doc text:
- Every named person mentioned + their role / position / title (no matter how briefly named —
  a one-line mention of "the secretary, Mary" still warrants its own fact).
- Every named organisation + its relation to the main entity.
- Every concrete date + the event that occurred on it (graduation 22 June 2003, trip 1 Nov 2022, etc.).
- Every named location + what happened there.
- Every distinctive descriptive detail: clothing colour, building material, exact age, weight,
  height, vehicle, distinguishing feature, last-seen description.
- Every cross-entity relationship: X collaborated with Y, X worked for Y, X spoke at Y's
  conference, X's child is Z, X co-edited a book with Y.
Anti-patterns — do NOT do these:
- Don't only extract facts about the most famous / dominant entity in the doc. Secondary
  individuals named once still warrant their own fact.
- Don't fill the budget with generic claims (founded-year, location, leadership) at the
  expense of specific concrete details that sit deeper in the doc body.
- Don't skip a fact because it seems minor — minor facts are often what disambiguate two
  similar entities at retrieval time.
Predicate guidance:
- Use a precise snake_case predicate (≤32 chars). Prefer reusing common terms when they fit:
  located_in, founded_in, founded_by, held_event, published_article, won_award, member_of,
  position_held, born_in, died_in, created_by, parent_of, succeeded_by, field_of_study,
  co_authored_with, organized_by, attended_by, physical_description, last_seen_wearing,
  clothing_worn, cross_link.
- Coin a new specific predicate when none of those fit. AVOID the catch-all `affiliated_with`.
Title and description constraints (CRITICAL — items violating these are dropped):
- title and description MUST read as natural standalone fact statements.
- They MUST NOT contain the strings: "BrowseComp", "qid", "qid:",
  "use this fact", "anchor a criterion", "without re-reading".
- They MUST NOT mention the document, the dataset, or this task.
Fact constraints:
- Favor specificity (proper nouns, dates, numbers) over generic claims.
- Skip the doc entirely (return empty facts list) for navigation pages, login walls,
  error pages, very short or generic content.
- evidence_span must be a verbatim substring of the doc text supplied above.

The extraction prompt was created by the agent. You can find earlier prompts created by the agent here.

Knowledge Indicators

We call this unit of pre-computed metadata a Knowledge Indicator — KI for short. KIs aren't a new concept for us: they already run in production in Elastic Observability Streams, where the same extraction pattern is applied to raw log data. There, an agent samples logs from a stream and extracts structured facts about the environment — which services are running, the infrastructure they sit on, how they depend on each other, the log schemas they use — and the KIs feed downstream into topology graphs, rules, dashboards, and agent investigations. The KIs auto-expire after 7 days if a service stops showing up, so the index stays current without manual cleanup.

The work in this post applies the same pattern to a different shape of corpus — documents instead of logs — but the unit is the same. Some KIs are facts; others describe sources. They share the same shape:

{
  "type": "knowledge_indicator",
  "id":   "ki-bcpc-d1478-tony-blair-position-held-prime-minister-of-the-un",
  "title": "Sir Tony Blair served as the Prime Minister of the United Kingdom from 1997 to 2007.",
  "description": "Blair was a British politician who held the office of Prime Minister for ten years starting in May 1997, and was the first person to lead the Labour Party to three consecutive general election victories. The text describes him as \"a British politician who served as Prime Minister of the United Kingdom from 1997 to 2007\".",
  "references": ["index://browsecomp-plus-corpus"],
  "tags": [
    "entity:tony-blair",
    "doc:1478",
    "tony-blair",
    "prime-minister",
    "politics",
    "1997-2007"
  ],
  "evidence_doc_ids": ["1478"],
  "payload": {
    "type": "feature",
    "subtype": "dataset_fact",
    "properties": {
      "subject":   "Tony Blair",
      "predicate": "position_held",
      "object":    "Prime Minister of the United Kingdom",
      "docid":     "1478"
    },
    "evidence": [
      "Sir Anthony Charles Lynton Blair (born 6 May 1953) is a British politician who served as Prime Minister of the United Kingdom from 1997 to 2007"
    ],
    "confidence": 100,
    "status":     "active",
    "last_seen":  "2026-05-10T11:12:41Z"
  }
}

In practice this is harder than it sounds, and the difficulty is why this work needs a system with optimized search, retrieval, and data engineering to work effectively.

The part that makes this work is measuring how the metadata holds up. We watch how agents use it to answer real questions, where they fail, where they fall back to scanning the raw source, where they give up. Those failures are the signal. They tell the setup agent what it missed, the metadata gets regenerated, and the next batch of failures looks different. Without that loop, the extraction agent overfits to whatever small set of questions it was first tuned against, and the strategy doesn't scale past a demo.

The shape of the loop, end to end:

The user contributes once - a set of example questions and a pointer at the source. Everything inside the loop runs on its own, and keeps running as new traces come in.

Give the agent a way to ask

The second part is the interface. The agent shouldn't need to know which source it's querying, what the index schema looks like, or whether the answer is going to come from knowledge indicators or from a fallback into the raw source. It should ask one question and get back something it can use.

Querying Knowledge Indicators

We settled on a natural-language interface for querying the metadata. The agent sends a question; an LLM behind the interface rewrites it into the underlying query against the index. The agent doesn't need to know the schema, the retrieval strategy, or which source the answer is coming from.

POST /api/_get_context
{
  "query":        "Wilkinson 2014 creatine review rheumatoid arthritis article title",
  "size":         10,
  "execute":      true
}

And this gets transformed by the use of an LLM to the following:

FROM .chat-context-items METADATA _id,_index,_score
| FORK
    (
      // Branch 1 — exact entity tag match for Wilkinson
      WHERE tags : "entity:wilkinson" OR tags : "wilkinson"
      | KEEP id, type, title, description, tags, references, evidence_doc_ids, _id, _index, _score
      | SORT _score DESC | LIMIT 25
    )
    (
      // Branch 2 — lexical match on title for specific keywords
      WHERE MATCH(title, "Wilkinson 2014 creatine review rheumatoid arthritis")
      | KEEP id, type, title, description, tags, references, evidence_doc_ids, _id, _index, _score
      | SORT _score DESC | LIMIT 25
    )
    (
      // Branch 3 — semantic match on description for the research intent
      WHERE MATCH(description.semantic, "Wilkinson 2014 review on creatine supplementation for rheumatoid arthritis")
      | KEEP id, type, title, description, tags, references, evidence_doc_ids, _id, _index, _score
      | SORT _score DESC | LIMIT 25
    )
| FUSE
| SORT _score DESC
| LIMIT 10

The response carries more than the matching KIs. Each KI comes with its tags and source references, and the API returns aggregations across the result set — counts by tag, by source, by entity — so the agent can see the shape of what's there before reading any individual KI. If the first ask returns a fan of results spread across three sources, the agent knows how to narrow. If they're all tagged with the same entity, the agent knows to follow that thread. Tags and aggregations are how it navigates the index quickly, without rereading.

The three metrics we tracked

We want to know whether pre-computing context represented in knowledge indicators helps an agent answer faster and more accurately than the standard RAG pattern when both are working against the same corpus with the same budget.

Three things we care about:

• Accuracy. How often does the agent commit to the correct answer? Answer match against the gold answer (judged by an LLM), plus F1 scores.
• Input tokens consumed. Every step the agent takes costs tokens. Fewer tokens to the same answer is the whole point of the strategy. If with-context answers more questions but burns the same budget doing it, that's not a win for any real deployment.
• Whether the agent converges within the step budget. A wrong-but-committed answer and a "ran out of steps and gave up" answer are both failures, but they fail for different reasons and the fix is different. We track timeouts separately.

We're not chasing leaderboard accuracy. We're testing whether the strategy holds up where the budget is tight, which is where every real agent lives.

Experiment Setup

Before the numbers, a quick walk through the setup:

Dataset. BrowseComp-Plus — 830 hard factual questions in the test split, each paired with gold source documents inside a roughly 100k-document web corpus. The questions are deliberately cryptic and multi-criteria; an agent typically has to chain two to five retrievals to converge on a short, exact answer like a name, title, or date.

Agent harness. Both setups use the same harness, based on the LangChain deepagents middleware stack. The agent has local shell access to call the skill's scripts. The only thing that changes between runs is which skill is loaded. We use Claude Sonnet 4.6 as the agent's model.

Agent harness budget. The harness has a 43-step recursion budget. BrowseComp-Plus leaderboard runs typically give agents far more headroom. We're not trying to compete on raw accuracy — given enough steps the agent will eventually get to the right answer either way. The point of this experiment is to hold the step and token budget low, and measure how well each retrieval strategy converges within a realistic budget.

Force-commit safety net. If the agent reaches the 43-step ceiling without emitting an Answer: line, the harness makes one final LLM call asking it to commit to its best guess from the trace so far. A timeout is not automatically a failure — the agent can still land on the right answer at the limit.

Baseline (search-and-fetch RAG). The baseline skill exposes two helpers:

• Search runs an ES|QL semantic and lexical MATCH(text, …) query against the corpus and returns up to ten hits, each with docid, url, _score, and three body snippets of up to 700 words. The agent reads relevant passages inline, in a single call, without the full document body.
• get_by_doc_id is the escape hatch for fetching a full body when the snippets don't cover the answer.

A caveat on the baseline: this is the standard search-and-fetch pattern most teams use today, but it's not the most optimized RAG setup possible. Someone tuning RAG hard against this benchmark — chunking strategies, reranking, query expansion — would close some of the gap. The comparison is against the setup most customers actually run, not the theoretical best.

In-context setup. The in-context skill exposes two helpers:

• get_context POSTs a natural language question to query relevant KIs.
• execute_esql is available as a raw fallback against the corpus.

The skill's SKILL.md teaches the agent to chain get_context calls for entity-anchored multi-hop retrieval and only drop to body searches after two or more KI calls have come up short.

How KIs are indexed. Each KI's title and description are mapped as Elasticsearch semantic_text fields backed by the .jina-embeddings-v5-text-small inference endpoint. It's a hybrid search (semantic + lexical) match against pre-extracted knowledge.

We took a 96 question sample (from the 830 question dataset) and transformed 25k (out of the 100k-document web corpus) documents into KIs (a 25% sample) which included the golden docs in each question. This produced around 240k KIs, built on Gemini Flash over roughly 7 hours.

Stage 1: Setup Agent Optimization

At setup time, we took a small slice of 15 questions and worked with the agent to improve the KI extraction process — iterating on the prompt to reduce the number of steps and tokens the agent harness consumed to reach an answer.

Against that same set of 15 questions and after 4 agent loop iterations refining the KI extractor prompt, the in-context setup posted:

The headline isn't really the 30% token saving. Exact-match accuracy more than doubled, while wall time stayed flat and tokens went down. Five new questions came out right that the baseline got wrong, and zero baseline wins were lost in the swap.

A note on what these numbers mean. The accuracy figures here are not directly comparable to the BrowseComp-Plus leaderboard. Leaderboard runs optimize for accuracy with generous step and token budgets. This experiment deliberately constrains both to simulate the real-world case we care about — agents working against limited budgets where retrieval efficiency, not raw reasoning headroom, decides whether the answer lands.

The wins clustered around questions where a KI directly answered the query. The baseline on the same questions burns through call after call of keyword search, often hitting its recursion limit before it converges. On four of the new wins, the baseline didn't fail because it was wrong, it failed because it ran out of steps. The KI route gets to the answer in a fraction of the budget, which is what the 80–95% token savings on those specific questions reflect.

However, it wasn't uniform. On a couple of questions the in-context setup actually used more tokens than the baseline (one was +151%, another +723%), because no KI covered the question well and the agent fell through to body searches after exhausting the KI route. That's the failure mode the feedback loop is built to close — every one of these traces is a signal that the extractor missed something the agent needed, and the agent would continually improve the extractor prompt to capture these facts for the domain more effectively.

Stage 2: Scaling up, but the failure mode changes

Strong numbers on a small slice. The next thing was to find out how much of that survives when you widen the eval set.

We ran the same setup over a 96-question expanded set and compared again to the snippet-baseline RAG.

With-context wins on accuracy and cost — 7 more questions correct and about 3.6× cheaper per question. But it also times out more, 37 against 28. That's worth understanding, because it changes how to read the timeout number.

A baseline search call returns up to 10 hits with three 700-word snippets each — one call can drop ~21k words of body text into the agent's context. The agent has to read that, find the answer-bearing sentence, and commit. Each miss means another search and more body text. With-context's get_context call returns ~10 KIs — single-sentence pre-extracted facts, ~2k tokens total. When a KI states the fact, the agent reads one sentence and commits. So with-context spends its budget on more retrieval calls, each one cheaper and sharper; baseline spends it on fewer, more expensive ones and then works through the text they return.

Higher Timeouts

Under a tight step budget, a timeout isn't a failure. It means the agent ran out of room before writing its answer. This is the realistic case for any budget-constrained deployment: the agent often has the answer in hand and simply hasn't committed it yet. 21 of with-context's 37 timeouts were judged correct on this run, because by step 43 the answer is usually already in the conversation from a KI retrieved already within the context, and the force-commit safety net picks it up. Baseline's timeouts fail outright more often, because its conversation is mostly raw body text and the force-commit pass is guessing from a haystack.

The safety net is catching answers the agent already had but didn't commit — so the agent is timing out on questions it could have answered itself, sooner and for fewer tokens. We want the agent to commit faster. That's a balance between two levers: the agent's instructions — when to keep retrieving, when to commit, when to fall through to body search — and the context it's working from — whether the KIs in front of it are sharp enough to commit on. Getting that balance right isn't a one-time fix. It's what the feedback loop is for: watch where the agent stalls, where it commits late, where it commits wrong, and feed that back into both the instructions and the extraction. Stage 3 is the first turn of that loop.

Stage 3: Teaching the agent from its own mistakes

Stage 2 left with-context ahead on accuracy and 3.6× cheaper per question, but 29 failures still on the table. We pulled the traces and looked at what was actually going wrong.

The failures grouped into three shapes:

• Wrong-twin commits (19 of 29). The agent retrieved KIs that were topically right but couldn't disambiguate between near-neighbours, and picked the closer-looking one. q79's University of Aberdeen came back as University of Edinburgh. q193's Secret Oral Teachings in Tibetan Buddhist Sects came back as The Mystic Spiral. q775's Boston came back as Jerusalem. Same failure mode each time — confident commit to the wrong entity.
• Wrong-value commits (3 of 29). Same mechanism, applied to numbers. q209's 9 came back as 7. q624's 65% came back as 26%. q1090's 500 Egyptian pounds came back as $1,500.
• No-candidate failures (7 of 29). The agent's organic retrieval did not find relevant candidates and the force-commit safety net produced a guess that didn't survive judging.

In all three cases, the agent retrieved something, the something didn't sharply distinguish the right answer from a plausible neighbour, and the agent committed wrong. The fix requires identifying distinguishing criteria before the agent responds.

For each failure we had four things: the question, the correct answer, the agent's wrong commit, and the gold doc body. We then used an LLM to write a single disambiguation KI per failure, with a title that names both entities and the criterion that separates them:

"Joseph Dalton Hooker (19th-century British botanist, Director at Kew) is associated with the second origin narrative — distinguished from 16th-century German botanist Leonhart Rauwolf."

A guardrail rejected any KI whose title didn't lexically contain both the gold answer and the wrong prediction, so the disambiguator was guaranteed to land inside the semantic-text embedding rather than buried in a description field the retrieval might skip. 33 of 38 attempted KIs passed the guardrail and got indexed into the same retrieval layer the agent already queries. The five rejections were cases where the wrong-pred was empty or contained unicode quoting that broke the lexical match.

We then re-ran the 96-question set as a single live evaluation:

Force-commits landed correct on 22 of 28 fires this run, which is why timeouts dropped and accuracy jumped.

{
  "title": "The manuscript was co-authored by TJ Wilkinson, TD O'Brien, and AB Lemmey.",
  "description": "The 2014 review article on creatine supplementation for rheumatoid arthritis was authored by TJ Wilkinson, TD O'Brien, and AB Lemmey, all from the Aberystwyth University research group.",
  "subject":   "Wilkinson, O'Brien, Lemmey",
  "predicate": "co_authored_with",
  "object":    "Oral creatine supplementation review",
  "tags":      ["doc:51481", "entity:wilkinson", "td-obrien", "ab-lemmey", "2014", "co-authors"],
  "evidence_doc_ids": ["51481"]
}

Example of a disambig knowledge indicator

21 of the 29 failures flipped. Wrong-twin: 16 of 19. Numeric: 1 of 3 — the two misses didn't surface their disambig KI on this run, which looks stochastic. No-candidate: 4 of 7, because a disambig KI with an answer-bearing title surfaces during retrieval before the force-commit safety net fires.

Accuracy improved and the agent got faster doing it. Input tokens dropped 12%, timeouts dropped from 37 to 27, and the typical winning trace lands in around 25 steps.

8 persistent failures remain. Three failed because the agent committed a different wrong twin on this run than the disambig was built for — example: q83's Stage-2 disambig separated Hooker from Rauwolf, but this run committed Francisco Hernández, who isn't in the KI title. The challenge is figuring out how to retrieve and guide the agent away from new wrong twins. An improved loop names multiple plausible wrong-preds, or iterates the disambig build over multiple runs. The loop needs to run continuously because the failure modes drift.

The headline result is what this says about that loop. Stage 1 and Stage 2 knowledge indicators once and read it back. A static index has a ceiling — better retrieval over the same static facts only goes so far. Stage 3 is the same system using its own failures as the next batch of context. Every wrong-but-confident commit is a diagnostic signal: which two entities the corpus failed to disambiguate, and what the wrong commit was. That becomes a KI the next run reads first.

Conclusion: extraction, retrieval, and a feedback loop

Across the three stages, accuracy went from 60% to 70% to 92%, and input tokens dropped by up to 75% versus standard RAG. To be clear this experiment is not claiming a fixed multiplier you can expect everywhere. The baseline could be tuned harder and a different domain could shift the failure modes around. What the experiment does show is simpler: a system built around pre-computed context represented as knowledge indicators can beat search-and-fetch on a tight budget, and the gap is big enough to justify the work.

Extraction has to be tuned to the domain

A generic extractor produces generic facts, and generic facts don't disambiguate. The 96-question run worked because the extractor prompt was tuned to the questions from the human created eval dataset — what types of entities, dates, and details to pull out. Point the same extractor at a logs corpus or a payments corpus and the KIs become meaningless.

This tuning isn't a setup step you do once either. It runs as an agent in a loop, always improving. The extraction agent writes the prompt, the eval shows where it failed, the agent rewrites the prompt against those failures. Stage 1 did exactly this over 4 iterations. And the loop keeps running after launch, because sources change and the questions drift. Every new domain needs an extractor that fits its source, plus a process that keeps it fitting.

Retrieval is what gets the right fact to the agent

In Stage 2 the agent picked the wrong neighbour 19 times out of 29 failures. The right fact was usually in the index. The agent just couldn't tell it apart from a plausible twin. That's a retrieval problem, and text similarity alone won't fix it.

This is why hybrid semantic and lexical search, tags, and aggregations exist — the agent needs to see the shape of the result set before it commits. Aggregations give it counts by tag, source, and entity, so it knows if the answer is spread across three sources or sitting in one. Tag filtering lets it narrow to a single entity in one call. Together these let the agent scan a large result set fast and decide where to look, instead of reading KIs one at a time and burning steps.

The feedback loop is what moves the ceiling

Stages 1 and 2 extracted context once and read it back. Accuracy stalled at 70%. Extracting more facts the same way wasn't going to help, because the problem wasn't missing facts. It was the agent picking the wrong one of two that looked alike, and tuning the extractor alone couldn't fix that.

What moves the ceiling is the failures themselves. When the agent commits to a wrong answer, the trace tells you something precise — exactly which two entities the index couldn't separate, and which one the agent picked instead. That's specific enough to act on. The system takes those failures and builds new KIs aimed straight at the gap, each one naming both entities and what separates them. Stage 1's tuning improves how facts get pulled; the feedback loop adds targeted facts the extractor would never have written on its own. Stage 3 did this, and accuracy moved from 70% to 92%.

What's next

This is just the first strategy we're excited to share — bottom-up extraction into knowledge indicators — and there are more we're hoping to explore that leverage our platform for retrieval, data management and agent feedback. We think there's real room to make agents dramatically cheaper and sharper when tokens are tight, and we can't wait to show you where this goes next.

The agent builds dashboards from scratch, but it also works with what you already have. Open the AI Chat sidebar while viewing a dashboard and it attaches automatically. Ask why a metric spiked, break it down by region, or add a comparison panel. Your existing dashboard becomes the starting point, not just the end product.

Behind the scenes: How we built dashboards in the AI Chat

We teach the agent specific tasks through skills — structured descriptions of how to operate on a given problem. But building a dashboard skill meant teaching an LLM to generate valid Kibana dashboards, and the legacy Saved Object API made that painful: deeply nested JSON, subtle version-to-version changes, brittle references. We needed a different approach

A purpose-built API for programmatic dashboards

The new Dashboards API was built for exactly this scenario. Instead of exposing raw internal state, it offers typed, validated schemas for every panel type. The API handles the translation between clean external structures and Kibana’s internal representations, so the agent can focus on what the dashboard should contain rather than on how to format it.

One skill, one tool, many operations

The dashboard-management skill exposes a single manage_dashboard tool that accepts an ordered array of operations. Each operation is a discrete action: Set metadata, add a markdown panel, create ES|QL-backed visualizations from natural language, edit existing panels, group panels into collapsible sections, or reposition items on the grid.

The agent can describe an entire dashboard: title, description, sections, and every panel inside them in a single call:

{
 "operations": [
   { "operation": "set_metadata", "title": "Checkout latency investigation" },
   {
     "operation": "add_section",
     "title": "Overview",
     "panels": [
       { "query": "p95 checkout latency over the last 24h", "chartType": "xy" },
       { "query": "checkout error rate by region", "chartType": "metric" }
     ]
   }
 ]
}

Operations execute in order, so later steps can reference and build upon earlier ones. This design keeps the conversation focused on intent rather than on implementation details.

The visualization pipeline: Natural language to ES|QL to visualizations

When you ask for a dashboard, the agent explores your data — indices, field mappings, types — then plans the visualizations and calls manage_dashboard.

Each panel runs through its own pipeline: chart type selection, ES|QL generation, visualization configuration, and validation. We isolated this from the main agent thread — visualization construction takes several model calls per panel, and mixing it into the main context would bloat the window and muddy the reasoning.

Inside manage_dashboard, all panels build concurrently, then reassemble in order. The result is a complete dashboard with embedded panels — no orphaned visualizations, no sync issues.

Why we moved visualization creation inside the dashboard tool

Our first approach used a separate create_visualization tool — one call per panel, then hand each attachment to the dashboard tool. It worked, but every visualization needed its own tool call, its own lifecycle, and an explicit handoff. Worse, editing a visualization in the conversation didn't update the dashboard panel, which confused users.

We folded visualization creation directly into manage_dashboard. The same parallel workflows run, but panels assemble into the dashboard structure without intermediate attachments. Fewer calls, no sync issues, one lifecycle.

Standalone visualizations still work — you can drop existing charts into a dashboard via attachment references — but for building from scratch, inline creation is the cleaner path

For security teams

SOC analysts and detection engineers can't afford a round trip to the dashboard editor mid-investigation. With the AI Chat, ask for alert volume by rule type, host, or MITRE tactic and see it in your thread in about a minute. As the hunt develops, layer in panels — process execution anomalies, network connections, timeline comparisons — without breaking context.

Save when you're done. The dashboard becomes a reference for the post-incident review, a starting point for the next analyst, or a weekly threat briefing — no re-explanation needed.

Read more about how security teams can use dashboard creation and other recently launched AI Chat capabilities in this blog post.

For observability and site reliability engineers (SREs)

When a service degrades at 2 a.m., there's no time to build dashboards from scratch. With the AI Chat, an SRE can describe the metrics they need (p99 latency by service, error rate against deployment events, pod restarts over the last hour) and get a full dashboard in the investigation thread in about a minute. The agent can refine it step by step as the picture sharpens: Add a panel, change the time window, break down by region.

Save the dashboard, and it's immediately available in the war room (same panels, same framing) for everyone joining the incident bridge. After the incident, it becomes the foundation for the postmortem.

What’s next

We're working on token optimization, richer full-screen interactions, broader panel support, and continued quality improvements. Technical preview is the right time to shape priorities — if something is missing, tell us via the "Submit feedback" icon in the top menu.

Try it

Upgrade to Elastic 9.4 (or start a trial), open the AI Chat in full-screen mode, and try it on a real investigation. Ask the agent to chart the metrics you're looking at, then ask for the next breakdown. When the story holds, save and share — same panels, same framing, no re-explanation needed. Requires an enterprise license (get started).
The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

How Kibana fetches data when loading a dashboard

When a Dashboard is opened, most of the panels (internally, we call these embeddables) kick off one or more Elasticsearch queries. But instead of the simple call-and-response of a synchronous (sync) search, we use the power of asynchronous (async) search (docs).

With async search, query results are kept available in Elasticsearch outside of any particular HTTP request. This is important because it

• makes data loading resilient to network turbulence
• powers our background search feature which allows users to work on other things in Kibana while they wait for a long-running dashboard or Discover session

After the initial query is submitted, Kibana monitors the search to detect when it is complete and retrieve the result set.

How traditional polling affects Kibana dashboard load times

In traditional polling, Kibana submits a query, closes the initial connection, then periodically checks Elasticsearch for completion.

We do give Elasticsearch a short amount of time after query submission to simply complete the search and return results. If the search completes that quickly, it amounts to a simple call-and-response. But for longer searches, the initial connection is closed and Kibana begins periodically checking the search for completion. This is called polling.

Performance drawbacks of traditional polling

Looking at the figure above, perhaps you can already see the performance drawback to this approach: the search is most likely to finish during one of Kibana’s sleep intervals, leading to lost time.

In the worst case scenario (when a search completes at the beginning of a sleep period) the entire duration of the polling interval will be wasted.

The impact of a backoff strategy

It’s standard practice when polling to apply a backoff strategy. This means that the longer the duration of the search, the less frequently we poll.

However, this also means that the potential lost time scales with the duration of the search.

How polling intervals create sawtooth latency patterns

Putting these factors together, our lost time becomes a stepwise sawtooth function.

Here, the peaks are worst-case scenarios and the troughs are best case scenarios. This illustrates that traditional polling costs us between nothing and the full duration of the polling interval, depending on the search duration (and network conditions).

Continuous polling: how Kibana eliminates wait time

The problem with traditional polling is a fundamental lack of coordination between Kibana and Elasticsearch. Ideally, Kibana knows immediately when results are available. So, what if we inverted the polling pattern to where nearly all of the time is spent checking Elasticsearch and no time is spent sleeping?

With this combination of long polling and no more sleep periods, results are delivered as soon as they are ready.

HTTP/1 degradation

The theory is solid. So why does this Kibana deployment look so degraded when we turn on continuous polling?

The key is that this deployment is running over HTTP/1. In HTTP/1, HTTP requests are mapped 1:1 to TCP connections. So several long-lived polling requests are hogging the browser’s finite connection pool, causing other requests to be queued.

In HTTP/2+ on the other hand, network requests can share TCP connections via multiplexing, so we don’t run into this problem.

So, on HTTP/2+ continuous polling is a virtue but on HTTP/1 it becomes a vice.

How Kibana detects HTTP protocol for optimal polling

HTTP/2 is the recommended protocol and it’s the Kibana default since 9.0, so it would be a shame not to ship this performance enhancement. On the other hand, the HTTP/1 experience is so degraded that it isn’t acceptable to risk it on any on-prem deployments who haven’t yet upgraded their protocol. The answer is clear: we need to detect which protocol is in use and apply the optimal polling strategy.

It is certainly possible for the Kibana server to know which protocol it is speaking. But, there’s a catch: the limiting factor is the browser’s connection pool. That means that what really matters is what the browser is speaking.

Because of proxies, these are not always the same.

If we based our optimization on the server protocol, we could get things wrong in one of two ways.

1. Apply continuous polling when we shouldn’t and degrade the experience.
2. Fail to apply continuous polling when we should and miss out on the optimization.

Luckily, modern browsers provide a way to detect the protocol of the last network hop of any completed request through the use of a PerformanceObserver. So, we watch for the protocol of the first query submission and optimize based on that.

new PerformanceObserver((list) => {
  const entries = list.getEntries();
  const entry = entries.find(({ name }) => name.includes('/internal/search/'));
  if (entry) {
    this.protocolSupportsMultiplexing = ['h2', 'h3'].includes(entry.nextHopProtocol);
  }
});

Lab results: continuous polling vs. traditional polling in Kibana

To validate continuous polling, we created dashboards with query delays ranging from 1 to 23 seconds and measured load times with and without the optimization enabled. We then loaded the dashboards with and without continuous polling to measure the gains (we had a lot of fun with race-for-the-prize).

The pattern echoes our original sawtooth diagram. For some query durations, the gains are small while for others they amount to several seconds.

Conclusion

This optimization successfully replaces the latency inherent in traditional polling with a more efficient continuous polling strategy. The primary challenge was implementing this optimization conditionally to prevent performance degradation on HTTP/1 deployments. We solved it using the browser’s PerformanceObserver to reliably detect the protocol in use for the final network hop.

Laboratory testing validates the theory, showing continuous polling delivers results as soon as they are ready. On average, this leads to a meaningful improvement in user experience, making data load up to 25% faster.

This work is the latest step in our commitment to driving down time-to-insight for our users. By making Kibana a more transparent proxy to Elasticsearch data, we push the limits of performance within our sphere of influence. More to come!

(In 2025, Thomas Neirynk gave an excellent overview of the methods and motivation behind improving Kibana dashboard performance. This is an update on that initiative.)

What is a Kibana dashboard builder?

What if you could describe the dashboard you want in plain English and watch it appear, complete with interactive charts, a drag-and-drop layout, and one-click export to Kibana?

That's exactly what example-mcp-dashbuilder does. It's an open source (Model Context Protocol (MCP) application that connects AI assistants to Elasticsearch, letting you create full Kibana Dashboards through conversation. No clicking through menus. No manually writing visualization configs. Just describe what you need, and the AI explores your data, writes the Elasticsearch Query Language (ES|QL) queries, builds the charts, and delivers a live, interactive dashboard, all within your editor's chat window.

From prompt to dashboard in seconds

Here's what it looks like in practice. You type something like:

"Build me a web traffic dashboard from logstash-* with total requests, bytes transferred over time, top geographic sources, and a response code breakdown"

The AI then:

1. Discovers your data: Lists indices, inspects field mappings.
2. Writes ES|QL queries: Tailored to your schema, using the right aggregations.
3. Creates visualizations: Bar charts, line charts, metrics with sparklines, heatmaps, pie charts.
4. Organizes everything: Collapsible sections, meaningful titles, proper layout.
5. Renders an interactive preview: Right in the chat, with tooltips, a time picker, and drag-and-drop.

Each chart appears inline as it's created, so you can see progress in real time. Then view_dashboard shows the complete dashboard with all panels laid out in Kibana's 48-column grid.

Single chart preview inline.

Powered by ES|QL

All data retrieval uses ES|QL, Elasticsearch's piped query language. The AI doesn't just pass through raw queries, it also uses built-in knowledge of ES|QL syntax along with information on the structure of your data to write correct, efficient queries for each visualization type.

The server includes a comprehensive ES|QL reference as an MCP resource. Before writing any query, the AI reads this reference to understand the available commands, functions, and patterns. Combined with a dataviz best-practices guide (also served as a resource), the AI knows not just how to query, but what makes a good visualization:

• Use BUCKET(@timestamp, 1 day) for time series; always SORT by the time field.
• Limit pie charts to six slices with | SORT value DESC | LIMIT 6.
• Choose bar charts for category comparisons, line charts for trends, metrics for key performance indicators (KPIs).

AI-driven data exploration with open-ended analysis

Building a dashboard you've already designed in your head is one thing. Asking "What's interesting in this index?" and getting a useful answer is harder; it requires the AI to know how to explore, not just how to draw.

example-mcp-dashbuilder ships an analysis://guidelines resource that defines a structured exploration flow: Profile the data, run targeted aggregations, surface patterns worth investigating, build charts for the most interesting findings, and propose drill-down queries the user might want next. Trigger phrases, like "analyze my logs" or "find patterns in this index," cause the AI to read the playbook before doing anything else, so an open-ended prompt produces a coherent investigation rather than a random pile of charts.

The result: You can hand the AI an unfamiliar index and get back a starting point: a dashboard plus a short list of "Here's what I noticed, want me to dig into any of these?" prompts.

Kibana dashboard export and import: the full round trip

The export/import round trip is where example-mcp-dashbuilder becomes genuinely useful for teams already working in Kibana.. example-mcp-dashbuilder is its own thing, a conversational dashboard surface that lives inside your editor, but it doesn't trap your work there. Dashboards built here can move into Kibana when you want them to, and existing Kibana dashboards can come the other way for AI-assisted editing.

Export to Kibana

When you're happy with your dashboard, one command exports it:

"Export this dashboard to Kibana"

Every panel is translated to a real Kibana Lens visualization. The translation preserves:

• ES|QL queries: Transferred directly as Lens ES|QL data sources.
• Grid positions: The same 48-column system Kibana uses, so your layout looks identical.
• Custom colors: Series palettes, metric backgrounds, heatmap color ramps.

The result is a fully functional Kibana dashboard. Not a screenshot. Not an embed. A real dashboard you can share and continue editing in Kibana.

Kibana dashboard and dashboard in Cursor chat side by side.

Import from Kibana

The round trip works in the other direction too:

"Import the Kibana dashboard with ID abc-123"

This fetches an existing Kibana dashboard, translates its Lens visualizations back to editable chart configs, preserves the grid layout and sections, and loads everything into example-mcp-dashbuilder. From there, you can modify it with natural language and re-export.

This makes the AI a collaborator in your existing Kibana workflow, not a replacement for it.

Custom themes and colors

Want a branded dashboard? Just ask:

"Create a pink-themed dashboard with custom colors"

Every visualization type supports custom color configuration:

• Charts: palette accepts an array of hex colors for series and slices.
• Metrics: color sets the background color.
• Heatmaps: colorRamp defines the gradient, from low to high values.

The AI picks up on theme requests naturally. Say, "Ocean theme," and it'll choose blues and teals. Say, "Match our brand colors" and provide hex values, and they'll carry through to Kibana on export.

A themed dashboard with custom colors.

How example-mcp-dashbuilder works: MCP architecture

example-mcp-dashbuilder is built on MCP, the open standard for connecting AI assistants to external tools and data. Here's the architecture at a high level:

The MCP server exposes 25 tools that the AI can call directly, everything from running ES|QL queries to exporting dashboards, alongside a handful of internal "app-only" tools that the inline preview uses to fetch data, persist layout changes, and detect time fields. It serves three resources: a dataviz best-practices guide, an ES|QL reference, and a deep-analysis playbook that kicks in for open-ended prompts ("analyze my logs", "what's interesting in this index"). And it runs over either stdio or HTTP; the HTTP transport supports streamable responses and session management, so multiple clients can connect to one server.

The MCP App is the interactive preview. It's built with React, Elastic Charts, and Elastic UI, bundled into a single self-contained HTML file. When the AI calls view_dashboard or creates a chart, the host renders this HTML in a sandboxed iframe. The app communicates with the server entirely through the MCP Apps protocol, using callServerTool() over postMessage to fetch data, save layouts, and detect time fields. There's no localhost server, no port to configure, no external network dependency.

This means it works with any MCP-compatible client: Cursor, Claude Desktop, Claude.ai, VS Code with Copilot, and more.

What chart types does example-mcp-dashbuilder support?

At time of writing, six chart types that cover the most common dashboard scenarios are supported:

Dashboards support collapsible sections for organization, a time picker with automatic time field detection,and the ability to save and switch between multiple dashboards; parallel chat sessions stay isolated from each other via a dashboardId threaded through every tool call.

How to install and run example-mcp-dashbuilder

example-mcp-dashbuilder is open source and ready to use. You'll need Node.js 22+, an Elasticsearch instance (local or Elastic Cloud), and an MCP-compatible client.

Claude Desktop: Download the latest .mcpb from GitHub Releases, and double-click it. Claude Desktop will prompt you for your Elasticsearch credentials.

Cursor / Claude Code / VS Code Copilot: Point your MCP config at the released tarball; no clone, no npm install:

{
  "mcpServers": {
    "example-mcp-dashbuilder": {
      "type": "stdio",
      "command": "npx",
      "args": ["https://github.com/elastic/example-mcp-dashbuilder/releases/latest/download/example-mcp-dashbuilder.tgz"]
    }
  }
}

Set ES_NODE, ES_API_KEY (or ES_USERNAME / ES_PASSWORD) and KIBANA_URL as environment variables. If you'd rather work from source, clone the repo and run npm run setup for an interactive wizard that handles both local Elasticsearch and Elastic Cloud (Cloud ID + API key).

And start building:

"Explore the logs index and build me the most insightful dashboard you can"

The AI takes it from there. 😉

Roadmap: what's coming to example-mcp-dashbuilder

This is an early release, and we're actively developing it. Some areas we're focused on:

• More chart types: Gauge, donut, treemap, data table, and tag cloud to match Lens's full capabilities.
• Push dashboards to Git: Write dashboard configurations into a repository for version control and code-review workflows.
• Better error UX: More detailed feedback when ES|QL queries fail, with suggestions for common fixes.
• Richer analysis flows: Extending the deep-analysis playbook to cover more data shapes (logs, metrics, traces).

We'd love to hear what you build with it. Try it out, file issues, and let us know what visualizations and workflows would be most useful for your team.

GitHub: elastic/example-mcp-dashbuilder

Acknowledgments

Thanks to Walter Rafelsberger and Tim Schnell for their contributions to the implementation.

FAQ

What is example-mcp-dashbuilder? example-mcp-dashbuilder is an open source MCP (Model Context Protocol) application that connects AI assistants to Elasticsearch. It lets you describe a Kibana dashboard in plain English and automatically generates ES|QL queries, creates visualizations, and delivers a live interactive dashboard inside your editor's chat window.

What query language does example-mcp-dashbuilder use to retrieve data? All data retrieval uses ES|QL, Elasticsearch's piped query language. The MCP server includes a built-in ES|QL reference that the AI reads before writing any query, ensuring correct syntax and efficient aggregations for each visualization type.

Can I export dashboards built with example-mcp-dashbuilder to Kibana? Yes. Running "Export this dashboard to Kibana" translates every panel into a real Kibana Lens visualization, preserving ES|QL queries, the 48-column grid layout, custom colors and series palettes. The result is a fully functional Kibana dashboard, not a screenshot or embed.

Can I import an existing Kibana dashboard into example-mcp-dashbuilder for AI-assisted editing? Yes. Providing a Kibana dashboard ID fetches the existing dashboard, translates its Lens visualizations into editable chart configurations, and loads them into example-mcp-dashbuilder. You can then modify the dashboard using natural language and re-export to Kibana.

Which MCP clients are compatible with example-mcp-dashbuilder? example-mcp-dashbuilder works with any MCP-compatible client, including Cursor, Claude Desktop, Claude.ai, and VS Code with Copilot. It supports both stdio and HTTP transport, with no localhost server or port configuration required.

What chart types does example-mcp-dashbuilder support? The current release supports six chart types: bar, line, area, pie, metric (with sparkline), and heatmap. Planned additions include gauge, donut, treemap, data table, and tag cloud to match Kibana Lens's full capabilities.

What do I need to run example-mcp-dashbuilder? You need Node.js 22 or higher, an Elasticsearch instance (local or Elastic Cloud), and an MCP-compatible client. Set the environment variables ES_NODE, ES_API_KEY (or ES_USERNAME/ES_PASSWORD), and KIBANA_URL. For Claude Desktop, download the .mcpb file from GitHub Releases and double-click to install.

Why stored-vector search used to require two requests

Previously, when you wanted to find documents similar to a stored vector, you needed to:

• Call GET to fetch the vector value from Elasticsearch.
• Call _search referencing that vector value in Elasticsearch:
  • Serialize the vector value via JSON.

This means paying serialization and network costs twice:

• Serialization and deserialization of the vector for both requests.
• Network latency costs in both directions.
• Potential egress costs in cloud deployments.

In Python, the pattern would be:

from elasticsearch import Elasticsearch

es = Elasticsearch(HOST)

# 1) Fetch the seed vector from Elasticsearch
seed_doc = es.get(
    index=source_index,
    id=seed_id,
    _source_includes=[vector_field],
)
query_vector = seed_doc["_source"][vector_field]

# 2) Send it back in a kNN query
resp = es.search(
    index=target_index,
    query={
        "knn": {
            "field": vector_field,
            "k": 10,
            "num_candidates": 100,
            "query_vector": query_vector,
        }
    },
)

While these two calls seem cheap, the overhead is unnecessary. Let’s make this better.

How query_vector_builder.lookup works in Elasticsearch 9.4

In Elasticsearch 9.4, we added lookup to simplify the API and eliminate unnecessary costs:

from elasticsearch import Elasticsearch

es = Elasticsearch(HOST)

resp = es.search(
    index="products",
    query={
        "knn": {
            "field": "product-vector",
            "k": 10,
            "num_candidates": 100,
            "query_vector_builder": {
                "lookup": {
                    "index": "seed-products",
                    "id": "product-123",
                    "path": "product-vector"
                }
            },
        }
    },
)

This request now grabs the dense_vector value stored in the product-vector field, in the document with ID product-123 in the seed-products index. This example is a “more like this” search, finding the nearest vectors to the one with ID product-123. You can refer to any index, effectively using lookup as a query vector store.

How much latency lookup vector search can remove

The goal is to simplify the experience and make it faster. The performance gains aren't just from eliminating the client round trip. Many Elasticsearch instances involve multiple nodes, and traffic between nodes can carry its own serialization and network costs. Elasticsearch actively biases execution toward the local node, which cuts network serialization costs on the server side, too.

To illustrate the potential performance improvements, here’s a benchmark we ran. We used a modified version of our so_vector, where instead of using the query vectors, one path did the GET and then _search pattern and the other used lookup. Running on two nodes in the same zone in GCP, the results were strong. Latency was consistently improved by almost 3x. Even when nodes are within the same data center and the same availability zone, network and serialization costs can have a real impact.

This benchmark ran with 2M documents, and the latency improvement will depend on your overall search costs. Even when the speedup is smaller, lookup still removes the extra client-side request. Less code, fewer round trips.

A simpler path for stored-vector search

Sometimes small changes can have an outsized impact. While this is a simple feature, I hope it removes some unnecessary friction in your Elasticsearch usage and makes us that much more lovable.

Downsampling (available since Elasticsearch 8.7) shrinks the footprint of your time series data by summarising data points into broader time buckets. It frees up storage and speeds up queries by orders of magnitude. Recently, our engineering focus has shifted from simply optimising the underlying downsampling engine to expanding what it can do. The new features give you more control over how your data is summarised.

In this post, we’ll explore the new downsampling features including:

• A choice between two distinct sampling methods: lightweight last-value sampling (for maximum storage savings) and high-fidelity aggregate sampling (for precise mathematical accuracy, such as counter resets).
• Expanded support for new metric types, including histograms.
• Full ES|QL support for downsampled gauges.

To begin, here is a quick review of the terminology used for time series data streams (TSDS):

• Metrics are the actual measurements that change over time, such as CPU usage.
• Dimensions are the identifying names and values associated with a measurement, which collectively determine the unique time series ID (_tsid).
• The timestamp marks the exact moment a measurement was taken.
• Finally, a (downsampled) bucket represents the result of reducing a metric's values across a specified time interval for a single _tsid.

How does downsampling work?

The downsampling process is initiated via the downsample API and can be automated using ILM (Index Lifecycle Management) or data stream lifecycle which downsamples and replaces the index with the raw data after the indexing has finished.

Since downsampling operates on a whole backing index (which must be read-only), and because the backing indices are time-bound for time series data, the system can generate the downsampled buckets from the data in a single index.

The downsampling task is optimised for efficiency: it reads all documents sorted by their time series dimensions and their timestamp in descending order. This specific sorting ensures that all data points belonging to a single time series bucket are collected sequentially without interleaving with other time series data. Once all documents that contribute to a single bucket have been read, their field values are collected for summarisation.

We will use the following data to show the effects of downsampling throughout this post. This data represents two nodes reporting their CPU usage and their number of requests every 10 seconds.

# Original data
{ "@timestamp": "2025-09-08T21:25:01.930Z", "node": "node-0001", "cpu.usage": 49.9, "requests": 3 }
{ "@timestamp": "2025-09-08T21:25:11.920Z", "node": "node-0001", "cpu.usage": 39.9, "requests": 6 }
{ "@timestamp": "2025-09-08T21:25:21.940Z", "node": "node-0001", "cpu.usage": 59.9, "requests": 2 }
{ "@timestamp": "2025-09-08T21:25:13.780Z", "node": "node-0002", "cpu.usage": 19.9, "requests": 100 }
{ "@timestamp": "2025-09-08T21:25:23.870Z", "node": "node-0002", "cpu.usage": 29.9, "requests": 102 }

Let’s see what downsampling (up to version 9.3) produces from this data using a 10-minute interval:

# Downsampled data
{ "@timestamp": "2025-09-08T21:20:00.000Z", "node": "node-0001", "cpu.usage": {"min": 39.9, "max": 59.9, "sum": 149.7, "value_count": 3}, "requests": 2 }
{ "@timestamp": "2025-09-08T21:20:00.000Z", "node": "node-0002", "cpu.usage": {"min": 19.9, "max": 29.9, "sum": 49.8, "value_count": 2}, "requests": 102}

A time series is uniquely identified by its dimension values, the node field in our example. Therefore, all documents summarised into a single bucket will share the same dimension values, meaning only one instance of the dimension values needs to be stored per bucket.

The timestamp and metrics are the fields whose values vary per document. For the timestamp (@timestamp), a rounding operation is performed to align it with the beginning of the bucket interval. For instance, in our example, the resulting timestamps are normalised to 2025-09-08T21:20:00.000Z (UTC).

For the metrics, up to 9.3 we used to downsample them based on their metric type, for gauges (cpu.usage) we stored the min, max, sum, and count of the encountered values and for counters (requests) the observed last value.

As you can see, up to 9.3 we effectively used a last value sampling method for counters, and the aggregate method for gauges. From 9.4 onward, for each field type, both sampling methods are available for all metric types and you get to choose which one best fits your data and the available system resources.

How downsampling sampling methods work

Different use cases require different trade-offs. For maximum storage reduction, last-value sampling keeps only the most recent observation per bucket. For accurate aggregations, the aggregate method keeps min, max, sum and count. Some applications demand maximum storage reduction and fast downsampling. Other use cases prioritise results that retain the highest possible fidelity to the original data, optimising for accuracy over sheer speed or space savings.

For this reason, in versions 9.3 and 9.4 we worked on offering two distinct ways of downsampling metrics. In 9.3 we introduced a new sampling method called last value and in 9.4 we differentiated the way we downsample counters between the last value and the aggregate method.

Last value sampling method

The last value method consistently downsamples data across all field types. For each time series bucket, it creates a single document. This document is timestamped with the start of the bucket, and all fields retain the last observed value from that period. The fact that it only needs one value makes it very efficient since it does not need to go over all values. Looking at our previous example, the downsampled documents look like this:

# 10 minute interval, last value method
{ "@timestamp": "2025-09-08T21:20:00.000Z", "node": "node-0001", "cpu.usage": 59.9, "requests": 2 }
{ "@timestamp": "2025-09-08T21:20:00.000Z", "node": "node-0002", "cpu.usage": 29.9, "requests": 102 }

While this method sacrifices data accuracy by discarding data points, it is a standard practice in time series solutions. Its primary benefit is preserving long-term trends while lowering the cost of data storage and querying. It is also lightweight, reducing the resources needed to generate downsampled data.

Aggregate sampling method

The aggregate sampling method does not skip any metric values, it collects all of them and then summarises them appropriately. The aggregate method processes each metric type differently, as explained below. Our example data, when downsampled with the aggregate method:

# 10 minute interval, aggregate method
{ "@timestamp": "2025-09-08T21:20:00.000Z", "node": "node-0001", "cpu.usage": {"min": 39.9, "max": 59.9, "sum": 149.7, "value_count": 3}, "requests": 3 }
{ "@timestamp": "2025-09-08T21:25:11.920Z", "node": "node-0001", "requests": 6 }
{ "@timestamp": "2025-09-08T21:25:21.940Z", "node": "node-0001", "requests": 2 }
{ "@timestamp": "2025-09-08T21:20:00.000Z", "node": "node-0002", "cpu.usage": {"min": 19.9, "max": 29.9, "sum": 49.8, "value_count": 2}, "requests": 100}

In the next sections we will see how the aggregate method summarises each metric type.

Gauges

Gauges are a fundamental type of metric. Unlike counters or histograms, their values can both increase and decrease, reflecting the current state of a system component (in our example, cpu.usage). A single value per bucket isn't enough to keep gauge aggregations accurate, since gauge values fluctuate. Instead, we track multiple statistical aggregates over each downsample interval:

1. min tracks the lowest value recorded for the gauge within the aggregation interval.
2. max tracks the highest value recorded for the gauge within the aggregation interval.
3. sum is the arithmetic sum of all individual gauge readings taken during the interval.
4. value_count tracks the total number of individual gauge measurements that contributed to the aggregation interval.

With these four statistics, we can answer the direct aggregations exactly (we know the true minimum, maximum, sum and count) and compute the average over the interval by dividing the recorded sum by the value_count. This approach preserves the shape of the gauge's behaviour over time, regardless of the downsampling interval.

In our example, we see that for node-0001 the downsampled document is stored in the format of an aggregate metric double:

{"min": 39.9, "max": 59.9, "sum": 149.7, "value_count": 3}

Some operations such as value filtering or standard deviation aren't covered by the summary statistics. In 9.4, we address this by using the average value in ES|QL. This is a major milestone because now the pre-aggregated gauges can be fully supported by ES|QL. Any ES|QL dashboard built on top of raw data can now use downsampled data with no errors, just with the expected loss of accuracy from using the average value per downsample interval.

We chose the average as the most representative signal we have available for the original samples in each downsample interval. For example, let’s think of the following query:

FROM my-data | STATS stddev = std_dev(cpu.usage)
FROM my-downsampled-data | STATS stddev = std_dev(cpu.usage)

The first query is applied to the original data and uses the individual data points. The second query is applied on the aggregated cpu usage values so we do not have individual data points. We could use min or max, but then the values used would be skewed towards the min or the max point. For this reason, we decided to use the average that would better capture the values within an interval.

Counters

Counters (requests in our examples), which are measurements that only ever increase (for cumulative temporality), seem straightforward for downsampling: just keeping a single value should be enough. However, when a process restarts, the counter value resets to zero.

The most common aggregation on a counter is its rate of change. Missing a counter reset when downsampling can drastically skew this rate, leading to inaccurate monitoring. Therefore, our downsampling process ensures that the rate calculation algorithm can still detect a reset, even when analysing downsampled data.

A rate algorithm detects a reset when a counter's current value is lower than its previous recorded value. To maintain accuracy in downsampled data, we need to ensure that the last value before the reset is preserved, and the next value seen is correctly compared against it.

So, let’s see how the counter requests in a downsampled bucket with reset look like:

# Main downsampled document
{ "@timestamp": "2025-09-08T21:20:00.000Z", "node": "node-0001", "requests": 3, "cpu.usage": ...}
# Pre-reset value
{ "@timestamp": "2025-09-08T21:25:11.920Z", "node": "node-0001", "requests": 6 }
# Post-reset value
{ "@timestamp": "2025-09-08T21:25:21.940Z", "node": "node-0001", "requests": 2 }

The main downsampled bucket stores the first counter value observed in that time frame. We choose the first value because it is the closest to the bucket's start timestamp. In addition to that, when a counter reset occurs within this time bucket, we store auxiliary documents with their original timestamps that hold the values to preserve the reset event:

1. Last value before reset is stored in an auxiliary document. This records the maximum value the counter reached before the reset event.
2. Value after reset (conditional) may also be stored in a second auxiliary document. This is optional because if the first value of the next downsampled bucket is already lower than the stored pre-reset value, the rate algorithm can infer the reset without needing this intermediate data point.

Let’s consider a more elaborate example, we have a stream of counter values within a time window, simulating a reset:

Bucket 1: 1000 1003 1010 1040 1060 (reset) 20 30 40 70 80

Bucket 2: 90 ...

This sequence results in the following downsampled documents:

• Downsampled bucket 1: stores the first value (1000).
• Auxiliary document: stores the pre-reset value (1060).
• Post-reset value: we do not need to store value 20, because the next downsampled bucket's starting value (90) is already lower than the pre-reset value (1060). The rate calculation will correctly detect the reset by comparing 90 against 1060.
• Next downsampled bucket 2: stores the first value (90).

The total change (delta) in the counter value is calculated via:

Notice that it remains the same for the original and the aggregated data because we have all the values available in both data sets: $(1060 - 1000) + 90 = 150$. In comparison, the last value misses the increase within the first bucket and only observes an increase of 90, 40% less.

Although the delta is accurately observed, the final downsampled rate may not be exactly the same as the raw data rate because of the rate extrapolation and interpolation. Still, this method keeps the result close to the raw rate, even when counters reset.

How aggregate method handles histogram metrics

The aggregate downsampling method merges histogram metrics into a single representative histogram per bucket, preserving distribution shape while reducing volume. The process depends on the histogram type.

For exponential histograms, which are designed for efficient storage of data with a wide range, their values and counts are aggregated and merged into a single, representative exponential histogram. This maintains the proportional distribution and statistical characteristics of the original datasets.

Conversely, for the older histogram field type and the newer tdigest field type, the merging is performed using the TDigest algorithm. This ensures that essential metrics derived from the distributions, such as the median or the 95th percentile, remain statistically reliable after the downsampling process is complete.

Let’s see an example with exponential histograms:

# Original data
{ "@timestamp": "2025-09-08T21:25:01.930Z", "node": "node-0001", "histogram": { "scale": 0, "sum": 7.0, "min": 1.0, "max": 4.0, "positive": {"indices": [0,1], "counts": [3,2]}}}
{ "@timestamp": "2025-09-08T21:25:11.920Z", "node": "node-0001", "histogram": { "scale": 0, "sum": 30.0, "min": 4.0, "max": 16.0,
  "positive": {"indices": [2,3], "counts": [4,1]}}}
{ "@timestamp": "2025-09-08T21:25:21.940Z", "node": "node-0001", "histogram": { "scale": 0, "sum": 55.0, "min": 0.5, "max": 32.0, "zero": { "count": 2, "threshold": 0.5 }, "positive": {"indices": [1,4], "counts": [5,3]}}}

# 10 minute interval, aggregate method
{ "@timestamp": "2025-09-08T21:20:00.000Z", "node": "node-0001", "histogram": { "scale": 0, "sum": 92.0, "min": 0.5, "max": 32.0, "zero": { "count": 2, "threshold": 0.5 }, "positive": { "indices": [0,1,2,3,4], "counts": [3,7,4,1,3] }}}

As you can see, the downsampled histogram is capturing the values of all three histograms and not just the last one as the last value method would have done.

Note: the histogram field type does not record which algorithm was used to build it. Histograms built with TDigest are downsampled correctly with the aggregate method. Histograms built with High Dynamic Range (HDR) are not, because the aggregate method cannot detect that. For HDR data, use the last value method instead.

How to configure Elasticsearch downsampling methods

You can configure the sampling method of a time series data stream using either data stream lifecycle or ILM:

# Data stream lifecycle
PUT _data_stream/my-data-stream/_lifecycle
{
  "data_retention": "7d",
  "downsampling_method": "aggregate",
  "downsampling": [
     {
       "after": "1m",
       "fixed_interval": "10m"
      },
      {
        "after": "1d",
        "fixed_interval": "1h"
      }
   ]
}

# ILM
PUT _ilm/policy/datastream_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover" : {
            "max_age": "5m"
          },
          "downsample": {
  	        "fixed_interval": "5m",
  	        "sampling_method": "aggregate"
  	      }
        }
      },
      "warm": {
        "actions": {
          "downsample": {
            "fixed_interval": "1h",
  	        "sampling_method": "aggregate"
          }
        }
      }
    }
  }
}

Note: all downsampling actions in a single ILM policy should have the same sampling method configured.

How do I switch between downsampling methods?

It is possible to switch from one sampling method to another, but a downsampled index can be downsampled further only with the same sampling method.

The data stream lifecycle takes care of this, so if you change the sampling method in a data stream lifecycle configuration, the data stream lifecycle will apply it to the original indices, but if there are already downsampled indices it will continue applying the compatible one.

On the other hand, if you are using ILM and the policy has more than one downsampling action, it is recommended to create a new policy with the new sampling method. This way, the existing indices will continue downsampling with the old sampling method and the new ones will transition to the new one. More specifically, we recommend the following steps:

1. Copy the ILM policy and change the sampling method in the downsampling actions.
2. Update the relevant index templates to use the new ILM policy.

How do last-value and aggregate downsampling compare?

Now let’s compare the two methods on more realistic data. We generated an hour of OTel metrics from a receiver. This was done using metricsgenreceiver with the following configuration:

• scenario: hostmetricsreceiver
• interval: 5s

Next, we downsampled the data into 10-minute buckets using both sampling methods. We then compared key performance characteristics, including the resulting document count, the size of the dataset (after being force-merged into a single segment), and the time taken for the downsampling process.

Both downsampling methods (last value and aggregate) significantly reduce the initial 1.51 million documents by 98%. However, the aggregate method retains slightly more documents to account for counter resets.

When considering the overall data set size, both methods show a substantial reduction compared to the raw data, but a more noticeable difference emerges between the two: the last value method achieves a greater size reduction than the aggregate method. This difference is not solely due to counter resets; it's also because the aggregate method stores more detailed information per gauge to maintain the accuracy of common aggregations.

The last value downsampling method is also faster, completing the process 7% sooner than the aggregate method. This efficiency is because the last value method does not need to collect every value; after the last value is found, the rest are ignored.

The last value method wins on data size and speed; the aggregate method is more accurate on queries, which is its main advantage.

The following section will demonstrate this with three ES|QL query examples using the recently introduced TS command which uses time series semantics and lets us use time series aggregations to examine how time series change over time. These queries will run against all three indices (raw, last value, and aggregate) to compare how the downsampled data relates to the raw data concerning query accuracy.

Gauges

For downsampling gauges, we will compare the following time series aggregations:

• The minimum of min_over_time
• The maximum of max_over_time
• The average of avg_over_time
• The standard deviation (std_dev) of last_over_time
• The standard deviation (std_dev) of avg_over_time

TS  | STATS 
min = MIN(MIN_OVER_TIME(system.cpu.logical.count)), 
max = MAX(MAX_OVER_TIME(system.cpu.logical.count)), 
avg = AVG(AVG_OVER_TIME(system.cpu.logical.count))

TS  | STATS stddev = std_dev(last_over_time(system.cpu.logical.count))
TS  | STATS stddev = std_dev(avg_over_time(system.cpu.logical.count))

The aggregate sampling method preserves accuracy for most results, such as min, max, and average, because it uses pre-aggregated data from the buckets. This data accurately reflects the raw data for all metrics except the standard deviation of the last value.

Since the pre-aggregated data does not include the raw last value, the system defaults to using the average value for the bucket to calculate the last value over time. This substitution reduces the overall variance and, consequently, the standard deviation compared to the raw data.

Conversely, the calculation based on the last value alone loses accuracy for min, max, and average because it omits certain data points. However, this method maintains the accuracy of the standard deviation of the last value query, as both rely on the same single last value within the bucket.

Counters

For counters, we will focus solely on the rate, as it is the most practical aggregation. Specifically, we will query the minimum and maximum rate observed across the time series.

TS  | STATS 
min_rate = min(rate(system.disk.io)), 
max_rate = max(rate(system.disk.io))

The maximum rate observed closely matches that of the raw data for both sampling methods. However, the minimum rate detected shows a significant difference because the last value method results in the loss of all reset information calculating a significantly lower value.

Choosing the right downsampling method

As we demonstrated in the previous sections, downsampling can substantially reduce data size and improve query performance, regardless of the sampling method used. The default sampling method is still the aggregate method, so accuracy is preserved by default. However, if your primary concern is reducing resource consumption during the downsampling process and in the resulting data, you now have the option to accept a slight decrease in accuracy for a more resource-efficient outcome.

What's next for Elasticsearch downsampling?

Three areas are in active development: sparse data performance, layered downsampling, and lifting the read-only index requirement.

Performance improvements for sparse data

Downsampling has been optimised keeping in mind that all the documents of a time series data stream will have the same fields. For example, in a data stream containing node metrics, we expected that all nodes would be defined by the same set of dimensions and all documents would contain the same metrics about these nodes.

However, in practice we see that this is not always the case, and quite often a time series data stream contains documents that represent measurements, that are defined by different sets of dimensions and contain different metrics. For example, the same data stream could have kubernetes metrics but also application metrics.

This is reflected in recent improvements, since 9.3, time series indices use doc values skippers, a form of sparse indices, instead of inverted indices and BKD trees, allowing them to be very efficient in this case. Following this direction, we will focus on improving the downsampling algorithm to use optimisation opportunities that this type of data provides.

The future of downsampling

We believe the improvements described above enable adoption of downsampling for most metrics applications. Three constraints still shape how teams use it:

• downsampling depends on ILM or data stream lifecycle for automation,
• requires the source index to be read-only,
• and replaces the raw data once complete.

We plan to address all three with a more flexible downsampling solution built around multiple downsampling layers.

Layers will lift the requirement to delete the original data, so you can start downsampling much earlier and query the pre-aggregated data even for the most recent information.

Multiple layers will also let you downsample at different granularities and manage each layer separately. For example, you could move the original data to the frozen tier sooner and keep only the pre-aggregated data in the hot tier, balancing cost and query performance per layer.

We expect to share design details and an early preview of the layered model in an upcoming post.

What does the Elasticsearch cuVS GPU plugin actually do?

The Elasticsearch cuVS GPU plugin takes over HNSW graph construction during two operations: segment flush and force merge. The plugin requires a supported NVIDIA GPU attached to the data node, the cuVS plugin installed, and vectors.indexing.use_gpu: true in elasticsearch.yml.

1. Segment flush. Vectors accumulate in the JVM write buffer. At flush, the plugin batches them, copies them to GPU VRAM via PCIe, constructs a CAGRA graph in VRAM, converts it to HNSW, copies the result back to system RAM via PCIe, and then Lucene writes the segment file to local NVMe. The segment is subsequently memory-mapped into the OS page cache for query serving.
2. Force merge. When segments are merged, the same GPU path speeds up graph reconstruction, delivering approximately 7x faster force-merge times according to the published benchmark.

Everything else (HTTP request handling, query serving, cluster state management, ILM execution) runs on the CPU and uses system RAM. The GPU is a coprocessor for two write-path operations, not a replacement for the host compute environment.

The separation of GPU build from CPU query serving matters because GPU-attached nodes don't need to be the query-serving tier.They can be dedicated to the write path and hand off built shards to cheaper CPU hot-serving nodes. That's the core insight behind Pattern B.

Pattern A: combined GPU build and serve in one Elasticsearch node

Pattern A is the simpler deployment. Every GPU node builds shards and serves queries on those shards. This is how the cuVS benchmark was run: a single g6.4xlarge (64 GB RAM, 1x NVIDIA L4) running indexing on a single Elasticsearch process. The node is capable of both indexing and search concurrently, though the published benchmark measured indexing throughput specifically.

When to use combined GPU nodes (Pattern A)

• Small clusters (fewer than 5 data nodes).
• Proof-of-concept or edge deployments where operational simplicity outweighs cost optimization.
• Corpora small enough that the full HNSW graph fits in each node's page cache alongside normal operations.

Node configuration (Pattern A)

Every data node gets a GPU and the same elasticsearch.yml config:

# elasticsearch.yml: Pattern A (combined GPU + serving node)
node.roles: [data_hot, data_content]

# auto (default): use GPU when available. true: fail to start if GPU unavailable.
# Use true on dedicated GPU nodes to catch misconfiguration at boot.
vectors.indexing.use_gpu: true

# JVM heap: ~32 GB max (compressed OOPs boundary)
# Remaining system RAM goes to OS page cache for HNSW graph

No shard-routing tricks, no ILM tier separation. Shards land where Elasticsearch's default allocator puts them, and every node can both build and serve.

Note: there is also an index-level setting, index.vectors.indexing.use_gpu, that can override the node-level default on a per-index basis. Valid values are auto (default, uses GPU when available), true (requires GPU, fails if unavailable), and false (disables GPU for this index).

Sizing implications

Because each node holds long-lived shards, it needs enough system RAM for:

• JVM heap (~32 GB)
• OS page cache holding the HNSW graph for its share of the corpus
• OS and CUDA overhead (~10-15 GB)

For a 300M-vector int8 corpus with HA (two copies), each of 10 data nodes holds ~65 GB of HNSW data in page cache. Add JVM and overhead and you land at roughly 128–192 GB of system RAM per node. That's much more than the 64 GB reference benchmark because the benchmark held only 2.6M vectors on a single node.

System RAM per node scales with shards-per-node, not with total corpus size. More nodes at lower RAM, or fewer nodes at higher RAM. The tradeoff is operational complexity versus hardware cost.

A note on the published benchmark. The ~12x indexing throughput and ~7x force-merge numbers were measured using 2.6M vectors at 1,536 dimensions with float32 hnsw on a single g6.4xlarge node. This post's examples use 1,024 dimensions with int8_hnsw. Performance characteristics may vary with different dimension counts and quantization levels, so benchmark your own corpus for production sizing.

Pattern B: dedicated GPU ingest nodes with ILM shard handoff to CPU

Pattern B separates the cluster into two data tiers with different hardware profiles and different roles:

• GPU ingest tier. Small number of nodes with NVIDIA GPUs and modest system RAM (64 GB). These nodes accept writes, build HNSW segments on GPU via cuVS, and own the active write shard.
• CPU hot-serving tier. Larger number of nodes without GPUs and larger system RAM (192 GB). These nodes receive migrated shards from the GPU ingest tier via ILM rollover and serve all query traffic.

Once a shard rolls over and migrates to the CPU hot-serving tier, the GPU ingest node no longer owns it. The GPU ingest node's page cache footprint stays small because it only ever holds the currently-writing index.

When to use GPU ingest + CPU serving (Pattern B)

• Production clusters with 5+ data nodes, where ILM rollover is already part of the operational cadence.
• When GPU-attached hardware is expensive and you want to minimize how many nodes need GPUs.
• When query-serving workloads benefit from dedicated CPU and RAM that isn't shared with GPU build operations.
• When ILM is already part of the operational model (which it should be for any time-series or lifecycle-managed vector corpus).

Node configuration (Pattern B)

GPU ingest nodes (elasticsearch.yml):

# GPU ingest node: builds shards, does not serve queries long-term
node.roles: [data_hot]
node.attr.tier_function: gpu_ingest

vectors.indexing.use_gpu: true

CPU hot-serving nodes (elasticsearch.yml):

# CPU hot-serving node: receives migrated shards, serves queries
node.roles: [data_hot]
node.attr.tier_function: cpu_serve

Both node types share the data_hot role because they participate in the same logical tier. The custom attribute node.attr.tier_function lets us control which nodes receive new writes versus migrated shards.

Index template: route new writes to GPU nodes

PUT _index_template/vectors-template
{
  "index_patterns": ["vectors-*"],
  "template": {
    "settings": {
      "index.routing.allocation.require.tier_function": "gpu_ingest",
      "index.lifecycle.name": "vectors-lifecycle",
      "index.lifecycle.rollover_alias": "vectors-active"
    },
    "mappings": {
      "properties": {
        "embedding": {
          "type": "dense_vector",
          "dims": 1024,
          "index": true,
          "index_options": {
            "type": "int8_hnsw"
          }
        }
      }
    }
  }
}

New indices matching vectors-* are allocated exclusively to nodes with tier_function: gpu_ingest. Writes flow to GPU ingest nodes, and cuVS builds the HNSW graph on the GPU.

Critical: you must explicitly set `index_options.type` to `int8_hnsw` (or `hnsw`) for GPU acceleration. In Elasticsearch 9.1 and later, `dense_vector` fields with 384 or more dimensions default to bbq_hnsw, which cuVS does not support. If you create vector fields outside this template without specifying int8_hnsw, the GPU plugin will fall back to CPU for indexing. The template above sets this correctly, but every vector field in the cluster that should benefit from cuVS needs the explicit setting.

ILM policy: roll over and migrate to CPU serving nodes

PUT _ilm/policy/vectors-lifecycle
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "7d",
            "max_primary_shard_size": "50gb"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "0ms",
        "actions": {
          "migrate": {
            "enabled": false
          },
          "allocate": {
            "require": {
              "tier_function": "cpu_serve"
            }
          },
          "set_priority": {
            "priority": 50
          }
        }
      },
      "cold": {
        "min_age": "180d",
        "actions": {
          "migrate": {
            "enabled": false
          },
          "allocate": {
            "require": {
              "tier_function": "cpu_serve"
            }
          }
        }
      },
      "delete": {
        "min_age": "730d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

A few things to note in this policy.

The `"migrate": {"enabled": false}` in the warm and cold phases is important. The warm-phase migrate action would normally attempt to set _tier_preference to data_warm, but since no nodes in this architecture have the data_warm role, shards could become unallocatable. Disabling migrate and using an explicit allocate action with our custom tier_function attribute gives us precise control over where shards land.

The `min_age: "0ms"` on the warm phase means migration happens immediately after rollover, not after a time delay. This is intentional: we want shards off the GPU ingest nodes within minutes of rollover to keep the GPU ingest tier lean.

Alternative approach: if the naming confusion between "warm" (ILM phase name) and "hot serving" (what the CPU nodes actually do) is a problem for your team, you can instead assign GPU ingest nodes node.roles: [data_hot] and CPU hot-serving nodes node.roles: [data_warm], then let ILM's standard tier migration handle the move without custom attributes. The tradeoff is simpler ILM but potentially confusing semantics: your heaviest query-serving nodes are labeled "warm."

Why 64 GB system RAM is enough for GPU ingest nodes

Under Pattern B, a GPU ingest node only holds the following (approximate, based on standard Elasticsearch JVM sizing guidance and observed CUDA runtime overhead):

This matches the published cuVS benchmark hardware: AWS g6.4xlarge with 64 GB RAM. The GPU ingest node does not need to hold the accumulated HNSW graph for the entire corpus, only the currently-writing index, which rolls over on a short cadence.

CPU hot-serving nodes, by contrast, hold the accumulated corpus in page cache and need 128–192 GB depending on vectors-per-node.

Sizing summary (Pattern B, 300M vectors, int8_hnsw, HA)

Total ERU (Elastic Resource Unit) under ECK: ~32.

How data flows through Pattern B, end to end

In Pattern B, vectors flow from the client to a GPU ingest node, where cuVS builds the HNSW segment on the GPU before ILM migrates the finished shard to a CPU hot-serving node. The sequence diagram below shows the round-trip for one batch of vectors, followed by the rollover that moves the finished shard to the serving tier:

In step form:

1. A client sends a _bulk request with vectors encoded as base64 strings (recommended for throughput).
2. The coordinating node routes the request to the GPU ingest node that owns the active write shard.
3. The GPU ingest node's JVM parses the request, queues vectors in the write buffer.
4. At flush threshold, the cuVS plugin batches vectors and copies them from system RAM to GPU VRAM via PCIe.
5. The GPU constructs a CAGRA graph in VRAM and converts it to HNSW format.
6. The HNSW data is copied back to system RAM via PCIe.
7. Lucene writes the segment file from system RAM to local NVMe.
8. The segment is memory-mapped into OS page cache (system RAM) and is now queryable.
9. After the configured rollover threshold (time or size), ILM rolls the index over and creates a new write index on the GPU ingest tier.
10. The rolled index's allocation requirement changes from gpu_ingest to cpu_serve, and Elasticsearch's shard allocator migrates the shards to CPU hot-serving nodes over the network.
11. CPU hot-serving nodes memory-map the received segments into their page cache and begin serving queries.
12. The GPU ingest node's page cache is released as shards leave, freeing it for the next write cycle.

At no point does query traffic touch the GPU ingest nodes. At no point does the GPU need to hold more than one batch of vectors in VRAM. The GPU is busy during steps 4–6 and idle otherwise. System RAM is in the path on both sides of the GPU: as the staging area for PCIe transfers, and as the persistent page cache for serving.

Adding a BBQ cold tier with requantization

Pattern B extends to a BBQ cold tier for long-retention corpora. BBQ delivers roughly 8x compression over int8, shrinking the cold tier's RAM and disk footprint dramatically. The ILM policy adds a phase that reindexes the int8_hnsw data into a new index configured with bbq_hnsw:

"cold": {
  "min_age": "180d",
  "actions": {
    "allocate": {
      "require": {
        "tier_function": "cpu_serve"
      }
    }
  }
}

In practice, the requantization happens via a separate reindex job triggered alongside the cold-phase transition. BBQ delivers roughly 8x compression over int8 (and ~32x over float32), so the cold tier's RAM and disk footprint shrinks dramatically.

Note that BBQ is a CPU-only quantization path. bbq_hnsw is not supported by cuVS in Elasticsearch as of 9.5. This is why the GPU ingest tier builds with int8_hnsw and the BBQ conversion happens later, on CPU warm or cold nodes. There is no GPU dependency on the cold path.

Which pattern should you choose?

For most production clusters with five or more data nodes, Pattern B is the better default. Pattern A is the right choice for proof-of-concept deployments and small clusters where operational simplicity matters more than cost.

For most production deployments, Pattern B is the better default. The operational complexity is modest (the ILM policy and allocation attributes shown in this post are the entire implementation), and the savings in GPU-node RAM and ERU are material.

Getting started with NVIDIA cuVS GPU vector indexing

GPU-accelerated vector indexing with NVIDIA cuVS is available in technical preview for Elasticsearch 9.3 (self-managed Enterprise) and is targeted for general availability in Elasticsearch 9.5.

Requirements:

• Elasticsearch 9.3+ with Enterprise subscription
• NVIDIA Ampere architecture GPU or newer (compute capability ≥ 8.0), minimum 8 GB VRAM
• CUDA 12.x and cuVS runtime libraries (refer to the Elastic support matrix for exact supported versions)
• Java 22 or higher on the GPU node
• Linux x86_64
• Fast local NVMe (network-attached storage is not recommended)

For teams weighing this against FAISS or a dedicated vector database like Milvus or Pinecone, the operational pitch is the same single platform, same ATO, same ops model, with GPU-accelerated ingest layered onto an existing Elasticsearch cluster. The broader Elastic and NVIDIA partnership context is in Elastic and NVIDIA together unlock next generation enterprise AI search.

Start with Pattern A on a single GPU node to validate throughput on your corpus. Once you have confidence in the numbers, move to Pattern B with the ILM policy above, scale the CPU hot-serving tier to match your query SLA, and let the GPU ingest tier do the one thing it is built for: building HNSW graphs at 12x the speed of CPU.

For the cuVS benchmark methodology and results, see [Up to 12x faster vector indexing in Elasticsearch with NVIDIA cuVS]. For the broader Elastic and NVIDIA partnership, see [Elastic and NVIDIA together unlock next generation enterprise AI search].

Elastic Cloud Serverless already removes the headache of managing infrastructure and version upgrades. CPS takes that a step further. We've replaced complex network peering and manual certificate management with a simple linking model. Now, you can just treat your Elastic Cloud Serverless projects as simple namespaces for your data. Whether you're dealing with strict data residency laws, isolating tenant data, or just trying to avoid the massive network egress fees that come from duplicating logs, CPS lets you search your data exactly where it lives with a single query.

In this post, we’ll walk through how CPS works, how to control your searches using project tags, and how this new model differs from traditional cross-cluster search (CCS).

How to link projects for cross-project search

To get started with cross-project search, link projects in the Elastic Cloud Console or API. Linking is simple and unidirectional: you choose an origin project, then connect the projects it should search. Those links can span regions, cloud providers, and project types, so your data can stay where it belongs without giving up a unified search experience.

Once the link is created, it usually takes effect within about a minute. If you already have Kibana open, refresh to see the new cross-project search capabilities.

How cross-project search queries all linked projects by default

Once projects are linked, cross-project search turns separate projects into a single logical search surface. If your logs live across multiple projects, a query like FROM logs* searches the origin project and any linked project that has matching data. You do not need to name each remote target up front.

That is a major improvement over cross-cluster search. In CCS, reaching both local and remote data often means writing something like FROM logs*,*:logs*. For users, that means less query complexity. For teams, it moves us closer to a true single pane of glass across distributed data.

For more information on this, see the CPS search model docs.

If you're interested in learning about technical details on how we built this, see How cross-project search (CPS) works in Elasticsearch Serverless.

Control of searches via Project Routing

Searching across every linked project by default is convenient and useful for many workflows, but not every search should go everywhere. Cross-project search introduces project routing, a way to limit a query to a specific subset of projects.

It works through project tags defined in Elastic Cloud. Every project has built-in attributes such as its alias, cloud provider, and region. You can also add your own tags to reflect how your organization thinks about its estate, such as environment:prod, environment:test, a business unit, or a customer name. Elasticsearch can then use that metadata to decide which linked projects should participate in a search.

All Elasticsearch endpoints that support cross-project search accept a project_routing parameter. In the technical preview, routing is limited to using project alias. For example, setting project_routing to _alias:my-linked-project sends the query only to that linked project, while _alias:_origin keeps the query on the origin project. Over time, this model opens the door to much richer routing, where search scope can follow the logical structure of your organization instead of the physical layout of your infrastructure.

See the project routing documents for examples and more details about how they work.

Kibana Space-level default project routing

As an example of where more precision is needed for your search routing, searching all linked projects might trigger a flood of false positives in your Kibana rules or confusing results in your existing dashboards. To fix this, you can set a Space-level default project scope in Kibana. This acts as a safe preset for that specific Space—meaning all dashboards, Discover sessions, and alerting rules automatically respect it. Analysts can still override the scope manually during an investigation if they need a broader view.

This matters for teams sharing a central project, such as MSPs, MSSPs, and centers of excellence: you can assign each team their own Kibana space and restrict it to only query their specific customer projects, guaranteeing tenant-specific experiences. Analysts can still override the scope manually during an investigation if they need a broader view.

You can configure this Space default before or after you actually link your projects in the Cloud UI. But because CPS immediately turns on the "search all" behavior the second a link is made, setting your Kibana defaults first ensures your existing detection rules don't suddenly run against a massive global dataset and overwhelm your team.

Using tags in searches

In addition to using tags for project routing, you can also use tags in your ES|QL and _search queries. This can be useful to identify where each record or row in a result set came from, or to sort, filter or aggregate by those tags.

For example, if you want to see which project every row in an ES|QL response came from, you can add the _project._alias tag to the ES|QL query:

FROM logs* METADATA _project._alias | ...

and this allows you to use _project._alias in other parts of the query including KEEP clauses in order to see it in the final result:

FROM logs* METADATA _project._alias | ... | KEEP @timestamp, message, _project._alias

For more examples of using tags in queries see this doc that describes how to use them both in Search APIs and in ES|QL.

If you're interested in learning about technical details on how we added tags to Search and ES|QL queries, see Faster cross-project search in Elasticsearch Serverless with project tags and routing.

How cross-project search handles origin and linked projects equally

If you have used CCS, you might be aware that the local cluster is treated differently from remote clusters in a few ways.

• Errors from the local cluster are handled differently than errors from remote clusters. In particular, CCS uses the skip_unavailable setting to control how errors from remote clusters behave, but that setting does not exist for the local cluster.
• The local cluster has no "cluster alias", so the index expression *:logs* searches all the remote projects, but skips the local cluster. To search both, you have to use the index expression logs*,*:logs*.

In CPS, we have changed both of these behaviors to put the origin project and linked projects on a more even footing.

First, the skip_unavailable setting is not used in Elastic Cloud Serverless. Instead, you control whether you want partial results on a search via the allow_partial_search_results parameter in _search or _async_search or the allow_partial_results parameter in ES|QL.

Second, in Elastic Cloud Serverless, the origin project has a project alias. It is defined in Elastic cloud like all project tags. Thus, in CPS, all of the queries below are equivalent - they target all projects with a "logs" index:

POST logs/_search

POST *:logs/_search


POST logs/search 
{
  "project_routing": "_alias:*"
}

Note: there is an important difference between the qualified index expression *:logs and the unqualified expression logs in terms of how error handling around missing indices works. For details see Unqualified and qualified search expressions in the public documentation.

Access control and security model for cross-project search

Elastic has created a new cloud-based security model, Universal Identity and Access Management (UIAM), that enables a key principle for cross-project search: the projects and data you can access do not depend on where you access it from.

Whether you initiate a search from your primary observability project or an ad-hoc analytics project, your access to the linked data remains consistent, since the access rights were defined in a centralized location. The cloud-based authentication and authorization model uses the cloud UIAM service to ensure your access permissions are uniform regardless of the origin project.

Try Cross-project search

Ultimately, Elastic Cloud Serverless and CPS together reduce operational friction and give you additional options for organizing data based on logical considerations rather than physical or operational considerations. Cross-project search allows your users to focus purely on the logical organization of their data, delivering a unified search experience without the physical complexities of the past.

This final post asks: What changes when the input comes from an AI agent instead?

The answer is that the architecture doesn't change, but the stakes do. Every property of the governed control plane that matters for human-authored queries matters more when the upstream decision-maker is a large language model (LLM). Determinism, auditability, conflict resolution, and constraint enforcement become critical guardrails rather than operational conveniences, because the system producing the input is probabilistic by nature.

The agentic search problem

The most common approach to AI-driven search is straightforward: Give the LLM the database schema, provide business rules in the prompt, and let the agent generate the query directly.

For an ecommerce chatbot, this means injecting the Elasticsearch index mapping, field types, category taxonomies, pricing logic, and business constraints into the agent's context window, and then asking the LLM to translate natural language into valid Elasticsearch Query DSL. The LLM becomes the query author.

This approach works in demos. It fails in production for four reasons.

Context bloat

An enterprise ecommerce index mapping is not a trivial document. Field definitions, nested objects, multi-field configurations, and analyzer settings can run to thousands of tokens before any business logic is added. On top of the mapping, the agent needs category taxonomies (which in enterprise ecommerce can contain tens of thousands of values), pricing rules, brand hierarchies, eligibility constraints, and campaign logic.

The result is a context window dominated by structural metadata rather than the user's actual intent. This increases latency, increases token cost, and degrades the LLM's ability to follow instructions as the context grows. This is a well-documented phenomenon, sometimes called context rot: As the prompt gets longer, the model's attention to any particular instruction weakens.

Probabilistic hallucination

LLMs generate queries based on patterns in their training data and the context provided. When asked to produce Elasticsearch Query DSL, the model can hallucinate field names that don't exist, construct syntactically invalid query clauses, misapply filter types to the wrong field types, or produce queries that are syntactically valid but semantically wrong, returning results that don't match the user's intent.

Google Cloud's BIRD benchmark for Text-to-SQL illustrates the ceiling of this approach. Google's state-of-the-art single-model result achieved between 70% and 80% accuracy, meaning that nearly one in four generated queries was incorrect. This is for SQL, which is far more standardized than Elasticsearch Query DSL. The error rate for LLM-generated Elasticsearch queries in a real production environment, with complex mappings and business-specific semantics, would likely be higher.

For a revenue-critical ecommerce system, a one in four query error rate isn’t a tuning problem to be solved iteratively. It’s an architectural limitation of the approach.

The security gap

When the LLM has access to the database schema and acts as the query author, the system is vulnerable to indirect prompt injection. A user interacting with an ecommerce chatbot can craft inputs designed to manipulate the agent into generating unintended queries.

This isn’t a theoretical risk. Prompt injection is one of the most actively researched attack surfaces in deployed LLM systems. The fundamental issue is that when the agent authors the query, there’s no structural boundary between user intent and query execution. The LLM is simultaneously interpreting the user's request and constructing the database operation. Any manipulation of the first directly affects the second.

High-cardinality scaling failure

Certain ecommerce fields have extreme cardinality. A product catalog might have 17,000 category values, thousands of brand names, and hundreds of attribute combinations. Standard agentic workflows require injecting these values into the context so the LLM can select the correct one when constructing a query.

This creates an impossible trade-off: Either inject all possible values (consuming enormous context and degrading performance), inject a subset (and accept that the agent cannot reference values outside that subset), or fall back to ungoverned search. This connects directly to the core problem from Part 1: If the LLM searches for “oranges” and Elasticsearch returns orange soda, the chat experience degrades in the same way a search experience does. The absence of governance means the system cannot enforce the shopper's intended resolution.

Retrieving relevant values dynamically based on the query is a known alternative, but it introduces an additional nondeterministic step where the retrieval itself can miss relevant values. Additionally, this adds latency and complexity to every query.

The architectural alternative: Decoupling intent from execution

The governed control plane described in Parts 1 through 7 offers a fundamentally different approach. Instead of the LLM authoring the final query, the LLM's role is reduced to a single, well-bounded task: extracting a search intent string from the user's natural language input.

The user says: "I'm looking for cheap brown shoes." The agent's job isn’t to generate an Elasticsearch query. It’s to extract and pass along the search intent, (in this case, something like "cheap brown shoes") to the control plane. The control plane then does what it has always done: percolates the intent string against stored policies, composes matching policies through cascading transformations, resolves conflicts deterministically, and produces a governed Elasticsearch query.

The LLM never sees the index mapping. It never knows about field types, category taxonomies, or pricing thresholds. It never constructs a query clause. It operates on the natural language side of an architectural boundary that we call the metadata air gap, a strict separation between the probabilistic component (the LLM) and the structured data layer (schema, policies, and query construction).

What the metadata air gap provides

• Schema blindness. The LLM has no access to the database schema and therefore cannot generate invalid queries, hallucinate field names, or be manipulated into exposing structural information. The schema exists only on the deterministic side of the air gap.
• Minimal context. Instead of thousands of tokens of mapping data, business rules, and category taxonomies, the LLM's prompt contains only a persona and intent extraction instructions. This dramatically reduces token cost, latency, and context rot.
• Deterministic execution. Every query that reaches Elasticsearch is constructed by the control plane using human-vetted policy templates, not generated probabilistically by an LLM. Syntactic validity is guaranteed. Semantic correctness is enforced by the same policy framework that Parts 1 through 6 described.
• Security by architecture. Prompt injection becomes structurally ineffective. Even if a user manipulates the agent into producing an unusual intent string, that string is percolated against stored policies. If no policy matches, no query is generated. The user cannot instruct the agent to construct a query because the agent doesn't construct queries. The control plane does, and the control plane is deterministic.

How the pieces connect

The following walkthrough shows how the governed control plane handles an agent-mediated query.

Step 1: The user speaks to the agent

A shopper interacting with an ecommerce chatbot says: "I'm looking for cheap chocolate, nothing with peanuts."

Step 2: The agent extracts intent

The LLM's role is intent extraction, not query generation. Given a minimal prompt that instructs it to identify the product intent, the agent produces a search intent string: "cheap chocolate without peanuts".

This is a lightweight classification task. The LLM doesn’t need the index mapping, category taxonomy, or pricing rules to perform it. It needs to understand natural language, which is exactly what LLMs are good at.

Step 3: The control plane governs the query

The intent string "cheap chocolate without peanuts" is passed to the control plane, which percolates it against the policy index. Three policies match:

• The "cheap" policy (extracts "cheap", applies a price filter based on the product category).
• The "chocolate" policy (constrains results to chocolate categories).
• The "without" negation policy (extracts the exclusion target and applies a must_not filter)

The control plane applies these policies through the same cascading transformation described in Part 3 and Part 4: priority ordering, per-field conflict resolution, consumed phrase tracking. If a “Christmas campaign” policy is also active, it composes with the product policies exactly as described in Part 3, the agent's involvement doesn't change the governance model at all.

Step 4: The governed query executes

The control plane produces a fully governed Elasticsearch query: a search for “chocolate”, constrained to the appropriate categories, with a price ceiling derived from the “cheap” policy, an exclusion filter for peanut-containing products, and any active campaign boosts applied. If the “chocolate” policy also includes economic optimization weights (Part 7), those are applied as well. Margin boosting is set to 3.0x because “chocolate” is a browsing query where the retailer benefits from promoting higher-margin products. If the shopper has purchase history (Part 6), personalization signals are layered on top. This query is syntactically valid by construction and semantically correct by policy design.

Step 5: Results return through the agent

The product results are returned to the agent, which presents them conversationally to the user. The agent's role in the return path is presentation: formatting results, answering follow-up questions, providing product details. The retrieval itself was governed, deterministic, and explainable.

What the agent is good at (and what it isn't)

This architecture leverages the LLM for what it does well and protects the system from what it does poorly.

LLMs excel at understanding natural language intent. "I'm looking for cheap chocolate, nothing with peanuts" is a natural language understanding task, parsing intent, identifying product references, recognizing negation. LLMs handle this reliably because it's a classification problem, not a generation problem. The output is a short intent string, not a complex structured query.

LLMs struggle with precise structured output under complex constraints. Generating valid Elasticsearch Query DSL requires exact field names, correct clause nesting, appropriate filter types for each field, and consistent application of business rules across thousands of edge cases. These are exactly the properties that a deterministic system enforces trivially and that a probabilistic system enforces unreliably.

The governed control plane puts each component where it belongs: the LLM on the natural language side, the deterministic policy engine on the query construction side, and an architectural boundary between them.

Governance constrains the blast radius

This is the same insight from Part 3, extended to the agentic context. In Part 3, we observed that governance makes semantic retrieval safer by narrowing the candidate set before retrieval begins. A semantic search over 500 products in a governed category is a fundamentally different proposition from a semantic search over 500,000 SKUs.

The same principle applies to agent-mediated queries. Without governance, an agent that misinterprets "cheap chocolate" could generate a query that searches the entire catalog with no price constraint, no category filter, and no exclusions. With governance, even if the agent produces an imperfect intent string, the control plane constrains the query to the policies that match. The worst case is that fewer policies fire, not that an unbounded query hits the product catalog.

Governance narrows the blast radius of probabilistic errors. This is true whether the probabilistic component is a semantic retrieval model or an LLM agent.

LLM-suggested policies: Expanding coverage

Part 2 introduced the idea that an LLM can suggest new policies that enter the same Author → Test → Promote pipeline as human-authored ones. In the agentic context, this becomes a powerful feedback loop.

An LLM can analyze query logs, identify patterns where the control plane has no matching policy (queries that fall through to unmodified retrieval), and suggest new policies to cover those gaps. A merchandiser reviews each suggestion, tests it, and promotes it if it produces the expected behavior. The governance model ensures that no LLM-suggested policy reaches production without human validation.

Over time, this creates a virtuous cycle: The control plane's policy coverage expands, the proportion of queries that require unmodified retrieval shrinks, and the system becomes progressively more governed, with every policy auditable, versioned, and individually reversible.

The broader pattern: Deterministic guardrails for probabilistic systems

The architecture described in this series, a deterministic control plane that sits between a probabilistic input source and a data retrieval system, isn’t specific to ecommerce search. The same pattern applies wherever an AI agent needs to interact with structured data.

An agent querying a SQL database faces the same challenges: context bloat from schema injection, hallucinated column names, prompt injection risks, and high-cardinality value selection. An agent interacting with a ticketing system like Jira, a customer relationship management (CRM) system like Salesforce, or a code repository like GitHub faces analogous problems. In every case, the core architectural question is the same: Should the LLM author the query, or should the LLM extract intent and pass it to a deterministic layer that authors the query?

The governed control plane provides a repeatable answer to that question. Policies are data. Intent extraction is the LLM's job. Query construction is the control plane's job. The metadata air gap keeps them separated. And the governance framework (priority ordering, conflict resolution, cascading transformations, auditability) ensures that the deterministic layer is operationally manageable as the number of policies grows.

Conclusion

The ecommerce search governance patterns described in this series (policies as data, the Author → Test → Promote workflow, cascading transformations, per-field conflict resolution, percolator-based reverse matching, and multi-tier fallback) were designed for a world where a merchandiser authors policies and a shopper types queries. But the architecture can enable much more than its initial use case.

When the input source is an AI agent rather than a human shopper, the governed control plane becomes the critical safety layer between a probabilistic system and a production data store. It provides the deterministic guarantees (syntactic validity, semantic correctness, auditability, and security) that enterprise systems require and that LLMs cannot provide on their own.

The deterministic control plane doesn’t replace the AI agent. It makes the AI agent safe to deploy.

Put governed ecommerce search into practice

The governed control plane architecture described in this series, from the policy-as-data paradigm to the percolator-based lookup to personalization, economic optimization, and the agentic air gap, was designed and built by Elastic Services Engineering. Every pattern described across this series comes from a working system built and validated against enterprise-scale product catalogs.

If your team is building AI-powered search experiences and needs deterministic guardrails for agent-mediated queries, or if you want to implement a governed, business-editable search architecture on Elasticsearch, Elastic Professional Services can accelerate your implementation. Contact Elastic Professional Services.

Join the discussion

Have questions about search governance, retrieval strategies, or ecommerce search architecture? Join the broader Elastic community conversation.

If you operate Elastic Cloud on Kubernetes (ECK), this release is about reducing the friction in the things you do every day.

Easier to operate, easier to understand

ECK 3.4 is a release focused on reducing what you have to think about when you run The Elastic Stack on Kubernetes. Each headline change picks a multi-step task and turns it into a single declarative answer:

• Simplified zone awareness. Telling ECK that a cluster should be spread across availability zones is now a single field on the NodeSet. The operator handles the topology, the scheduling, and the Elasticsearch-side awareness configuration on your behalf. Your manifests reflect what you mean, not how it's wired.
• Restart a cluster the same way you do everything else. Triggering a rolling restart is now an annotation on the Elasticsearch resource. It's declarative, fits GitOps, and leaves an audit trail. No force-edit on an unrelated field to get a rollout.
• mTLS is automatically configured by the operator. Wiring mutual TLS between Kibana and Elasticsearch by hand requires managing CAs, per-component client certificates, mounts, rotation, and configurations on both ends. ECK 3.4 takes care of all of that: flip a flag on Elasticsearch, point Kibana at it, and the operator manages the rest.

This release is to make day-to-day ECK operations boring, in the best sense: fewer fields to remember, fewer side trips to keep in sync, and simpler-to-understand manifests.

Simplified zone awareness

Make an Elasticsearch cluster highly available across availability zones by setting one field on the NodeSet. ECK 3.4 handles the topology spread, the pod scheduling, and the Elasticsearch-side awareness configuration for you.

Before, you had to wire all of this by hand across four separate objects: an annotation on the Elasticsearch resource for downward node labels, awareness attributes in the NodeSet config, a fieldRef env var in the pod template to surface the zone, and a matching topologySpreadConstraints block plus a nodeAffinity rule pinning the cluster to specific zones. Roughly forty lines of YAML, easy to misconfigure.

In ECK 3.4, the same zone-aware cluster is four lines:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: my-cluster
spec:
  version: 9.4.0
  nodeSets:
  - name: default
    count: 3
    zoneAwareness: {}

To pin to a specific set of zones, name them, and ECK adds the matching required node affinity rules:

spec:
  nodeSets:
  - name: hot
    count: 3
    zoneAwareness:
      zones: ["us-east-1a", "us-east-1b", "us-east-1c"]

If you do need to customize maxSkew or whenUnsatisfiable, providing a matching topology spread constraint with the same topologyKey in podTemplate still wins. Your override stays an override.

One note for upgrades: enabling zoneAwareness on an existing NodeSet changes the StatefulSet pod template (new topology spread constraints, ZONE env var, node affinity, node.attr.zone), which triggers a one-time rolling restart of the affected NodeSet. Plan accordingly.

To learn more about simplified zones management, you can read this page at Elastic Docs.

Declarative rolling restarts

Restarting an Elasticsearch cluster without changing its spec is now a first-class workflow in 3.4. Two new annotations on the Elasticsearch resource do the work:

• eck.k8s.elastic.co/restart-trigger: set or change this value (a timestamp is the conventional choice) to start a rolling restart. Changing the value triggers another restart later; removing the annotation does not.
• eck.k8s.elastic.co/restart-allocation-delay: optional duration string (e.g. "20m") passed to the Elasticsearch node shutdown API as the allocation delay during the restart, so you can hold off on rebalancing while a pod recycles.

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: my-cluster
  annotations:
    eck.k8s.elastic.co/restart-trigger: "2026-04-30T10:00:00Z"
    eck.k8s.elastic.co/restart-allocation-delay: "20m"
spec:
  version: 9.4.0

Under the hood, ECK propagates the trigger value to pod annotations, which changes the StatefulSet template hash and feeds every pod through the existing rolling-upgrade path (node shutdown API, predicates, one-pod-at-a-time deletion). There's no new restart mechanism to learn, and the status messages and observability you already have on rolling upgrades carry over.

For GitOps users, this means a Flux/ArgoCD pipeline can request a restart by patching one annotation: no spec drift, no diff churn, no force-edit on an unrelated field.

Managed mTLS for Kibana ↔ Elasticsearch

Mutual TLS orchestration between Kibana and Elasticsearch arrives with this release. The Elasticsearch CRD accepts a single new field, spec.http.tls.client.authentication: true, that tells the cluster to require client certificates on its HTTPs interface. ECK does the rest: it builds a trust bundle from any secret labeled eck.k8s.elastic.co/client-certificate: true, mounts it into the Elasticsearch pods, sets xpack.security.http.ssl.client_authentication: required, and issues an operator-side client certificate so it can keep talking to the cluster throughout the rollout.

This makes enabling and configuring mTLS for the stack (Elasticsearch and Kibana only, in this release) a much simpler task.

Enabling mTLS on Elasticsearch:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: secure-cluster
spec:
  version: 9.4.0
  http:
    tls:
      client:
        authentication: true # <---- This is all you need
  nodeSets:
  - name: default
    count: 3

On the client side, Kibana's association controller now detects the client-authentication-required annotation on the referenced Elasticsearch and automatically generates a client certificate for Kibana — no extra config needed. If you want to bring your own cert (cert-manager, an internal PKI), point at the secret you've already provisioned:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  version: 9.4.0
  count: 1
  elasticsearchRef:
    name: secure-cluster
    clientCertificateSecretName: my-custom-client-cert

ECK rotates the certificate, mounts the secret into the Kibana pod, and wires elasticsearch.ssl.certificate and elasticsearch.ssl.key. Cleanup of mTLS resources is deferred until all pods have rolled, so connectivity holds throughout the transition.

Kibana is the first Stack component to get this first-class treatment in 3.4. Support for APM Server, Beats, Fleet Server, Elastic Agent, Logstash, Maps, and Enterprise Search ships in the near future. In the meantime, a new recipe walks through manual mTLS for those components using cert-manager.

Other notable improvements

This release includes other improvements worth highlighting. Here is a list with their related pull requests.

• Native Go FIPS 140-3 in the FIPS-enabled operator (separate image). The FIPS-flavored ECK image (docker.elastic.co/eck/eck-operator-fips:3.4.0, plus a UBI variant eck-operator-ubi-fips:3.4.0) now ships with native Go FIPS 140-3 support, pinned at the certified GOFIPS140=v1.0.0 module and enforced at runtime. The standard eck-operator image is unchanged. For Elasticsearch 9.4.0 or later, the operator also generates and mounts a FIPS-compliant keystore password automatically when xpack.security.fips_mode.enabled: true is set (#9263, #9287).
• Reliability fixes worth calling out:
  • Stale CAs in the certificate chain are now detected and trigger reissuance (#9197).
  • Remote-CA secret generation failures are non-blocking (#9271).
  • The NetworkPolicy namespace selector label is fixed for soft multi-tenancy setups (#9153).
  • The Elasticsearch controller skips its default PVC if a volume of the same name already exists (#9199).
  • The DaemonSet reconciler handles stale cache the same way the Deployment reconciler does (#9256).

Getting started

If you're already running ECK, upgrade to 3.4.0 with Helm:

helm upgrade elastic-operator elastic/eck-operator -n elastic-system

Or apply the latest operator manifest directly:

kubectl apply -f https://download.elastic.co/downloads/eck/3.4.0/crds.yaml
kubectl apply -f https://download.elastic.co/downloads/eck/3.4.0/operator.yaml

If you're new to ECK, start with the quickstart guide to get an Elasticsearch cluster running on Kubernetes in minutes.

For the full list of changes, see the ECK 3.4.0 release notes on GitHub.

To start using Elastic Cloud today, log in to the Elastic Cloud console or sign up for a free trial.

Frequently asked questions

How do I make an Elasticsearch cluster zone-aware in ECK without writing topology spread constraints?

Set spec.nodeSets[].zoneAwareness: {} on the Elasticsearch resource. ECK derives the topology, attaches node.attr.zone, sets maxSkew=1 topology spread constraints, and injects the downward labels for you. Provide zones: [...] if you want to pin to a specific set of availability zones. Enabling this on an existing NodeSet causes a one-time rolling restart.

Can I trigger a rolling restart of an Elasticsearch cluster on Kubernetes without editing the spec?

Yes. ECK 3.4 introduces two annotations on the Elasticsearch resource: eck.k8s.elastic.co/restart-trigger (set or change the value, e.g. a timestamp, to start a rolling restart) and eck.k8s.elastic.co/restart-allocation-delay (optional duration string passed to the Elasticsearch node shutdown API). Removing the trigger annotation does not start a new restart.

How do I enable mutual TLS between Kibana and Elasticsearch on Kubernetes?

With ECK 3.4, set spec.http.tls.client.authentication: true on the Elasticsearch CRD and reference it from Kibana via elasticsearchRef. ECK auto-generates a client certificate for Kibana, builds a trust bundle from any secret labeled eck.k8s.elastic.co/client-certificate: true, and configures xpack.security.http.ssl.client_authentication: required for you. mTLS for Kibana ↔ Elasticsearch is a technical preview in 3.4.

Does ECK 3.4 mTLS support cover all Stack components like Beats and Fleet?

Not yet. Kibana is the first Stack component to get first-class mTLS support in 3.4 — the operator auto-generates its client certificate. Support for APM Server, Beats, Fleet Server, Elastic Agent, Logstash, Maps, and Enterprise Search ships in the next release. A new recipe walks through manual mTLS for those components using cert-manager in the meantime.

Does ECK support FIPS 140-3?

Yes, in a separate operator image. ECK 3.4 publishes a FIPS-flavored build (docker.elastic.co/eck/eck-operator-fips:3.4.0, plus a UBI variant) with native Go FIPS 140-3 support. The standard eck-operator image is unchanged. For Elasticsearch 9.4.0 or later, ECK also generates and mounts a FIPS-compliant keystore password automatically when xpack.security.fips_mode.enabled: true is set.

This blog introduces Serverless project tags and details how to use them in the context of cross-project search to target specific projects in queries and improve the query performance.

Project tags background

Each Serverless project has a set of key-value pairs associated with it, allowing you to categorize and organize your projects. These are called project tags. There are two types of tags:

• Prebuilt tags: Predefined tags that exist for every project and are generated automatically; for example, _alias, _id, _type, corresponding to the project alias, project ID, and project type. Predefined tag names always begin with an underscore.
• Custom tags: User-defined tags; they can have any name that begins with a letter and can contain lowercase letters, numbers, underscores, and hyphens. For example, you could have the tag env:qa for your QA environment projects and env:production for your production environment.

In the Elastic Cloud console, you can manage project tags in the “Manage project” screen:

And programmatically, you can see the tags by using the GET _project/tags Elasticsearch API:

{
 "origin": {
   "f58ef00d1538476f884c137bf7d304ff": {
     "_alias": "my-origin-project-1",
     "_csp": "aws",
     "_id": "f58ef00d1538476f884c137bf7d304ff",
     "_organization": "500590167",
     "_region": "aws-eu-west-1",
     "_type": "security"
   }
 },
 "linked_projects": {
   "a4f8c2fa86824395a845e4a055e1ce83": {
     "_alias": "my-linked-1",
     "_csp": "aws",
     "_id": "a4f8c2fa86824395a845e4a055e1ce83",
     "_organization": "500590167",
     "_region": "aws-eu-west-1",
     "_type": "observability",
     "env": "qa",
     "test": "abcd"
   }
 }
}

In your queries, you can use project tags to limit the scope of the search to a subset of your linked projects; for example, only observability projects or only projects belonging to the billing department. This is called project routing; for each search, you can provide a filtering expression that confines the search only to the projects which have the tags matching the expression. This is a very efficient way of reducing the search space, because it doesn’t require accessing any projects that don’t have the matching data.

In addition, you can use the project tags in queries just like any index fields are used: for matches, aggregations, sorting, as part of the search fields output, and more. When used in this way, project tags look exactly like any mapped field, but their values aren’t stored in the index; they’re loaded dynamically from the project configuration. This adds flexibility in using project tag information in searches and aggregations.

When used in this manner, project tags are always prefixed with _project.; for example, project tag _alias becomes _project._alias, and project tag department becomes _project.department.

Project routing

Project routing restricts cross-project search to a subset of linked projects by applying a filter expression, which uses a subset of Lucene query syntax, before the query processing begins; for example, _alias:my_search_project. This is a very efficient way of limiting the query, since the filter is applied before any of the operations are executed, and the linked projects that don’t match the filter won’t even be contacted in this query. However, this way of filtering is limited because you can only use it to exclude the whole project.

This filtering is applied to every index expression, so for example, when running GET logs/_search without the routing expression, it would look for logs index on the origin project and every linked project; however, with routing expression _alias:my-o11y-project, only the project with the alias my-o11y-project will be used.

For convenience, you can store frequently used routing expressions as named project routing expressions and reuse them between different queries by specifying just the expression name: @routing-expression.

In the Technical Preview release, project routing is limited to only filtering by _alias tag matching; for example, _alias:my-security-*. In future releases, we plan to support all tags and most of the Lucene filter syntax.

Query DSL

Project routing can be specified for search using request body field:

GET logs/_search 
{
  "project_routing": "_alias:my_search_project"
}

Elasticsearch Query Language (ES|QL)

You can specify project routing as a statement preceding your main query:

SET project_routing = "_alias:my-project-alias" ;
FROM my_index
| LIMIT 10

Alternatively, you could use a project_routing query parameter in an API call, as above, but within ES|QL, the SET syntax is preferred.

Precedence rule: If both the SET project_routing syntax is used and a project_routing query parameter is provided via the API, the SET syntax takes precedence.

Project tags as fields

Project tags can be used as fields in queries (for fetching, matching, aggregating, and sorting) by referencing them with the _project. prefix. This is implemented by creating a special dynamic mapping type and attaching it to the field named _project. By itself, this field doesn’t have any data and cannot be used directly, but it has subfields with names matching project tag names; for example, _project._alias or _project.env. These fields return constant keyword values, taken directly from the project tags map stored in memory.

Query DSL

In Query DSL, you can use the project tag fields just as you do with regular fields, wherever a field name is accepted; for example:

Fetching:

{
  "fields": ["count", "_project._id", "_project._alias", "_project.env"]
}

Wildcard patterns work, too:

{
  "fields": ["count", "_project.my-tag-*"]
}

By default, however, project tags aren’t in the output fields (including when using * as fields); you always need to include them explicitly with the _project. prefix. You don’t need to add project tags to the output to use them for matching or aggregations; these functions are independent of each other.

Matching:

"query": {
"match": {
"_project._alias": "my-project"
}
}

For this simple example, project routing may be a more efficient way; however, more complex matching can be used, too:

{
  "query": {
    "bool": {
      "should": [
        { "term": { "_project._alias": "my-project" } },
        { "term": { "_project.env": "qa" } }
      ],
      "minimum_should_match": 1
    }
  }
}

Unlike project routing, the full set of Elasticsearch matching expressions can be used here, even in the Technical Preview release. Keep in mind that this comes at a performance cost: All indices named in the index expression will still be resolved, and all linked projects will be contacted, even if the match expression ultimately excludes all their data.

You can also use aggregation on project tags:

{
	"size": 0,
	"aggs": {
		"by_project": {
			"terms": {
				"field": "_project._alias"
			},
			"aggs": {
				"total_count": {
					"sum": {
						"field": "int_count"
					}
				}
			}
		}
	}
}

And sort by a project tag:

{
	"size": 4,
	"sort": [
		{
			"_project._alias": {
				"order": "asc"
			}
		}
	],
	"fields": [
		"count",
		"_project._id",
		"_project._alias"
	]
}

ES|QL

To use project tags as fields in your query results, you must explicitly include them using the METADATA keyword in your FROM clause:

FROM my_index METADATA _project._alias
| LIMIT 5

Wildcards can be used to include multiple tags:

FROM my_index METADATA _project.*
| LIMIT 5

Once included, these tag fields behave like any other field, usable by all ES|QL commands; for example:

FROM my_index METADATA _project._alias
| WHERE _project._alias == "production-data"
| STATS count() by status

In ES|QL, there are a few important differences between using project_routing and defining filters based on project tags in a WHERE command.

Index resolution:

• With project_routing="...", indices are only resolved on the specified projects, so you won’t see fields that are only defined in the mappings of the excluded projects.
• With a simple filter, for example, WHERE _project._alias LIKE "...", the index resolution will be routed to all the projects, so you’ll also see columns that are only defined in projects that don’t match the alias pattern.

Query routing:

• With project_routing="...", the query will be routed only to the involved projects.
• With WHERE _project._alias LIKE "...", the query will be routed to all the projects, so you’ll pay at least for the cost of query coordination.

Query execution:

• With project_routing="...", the query will be executed only on the nodes that match the expression.
• With WHERE _project._alias LIKE "...", the query will be routed to all the nodes, but ES|QL will perform a second optimization phase and will replace project tags with constants, performing constant folding that will transform the original filter into a WHERE true for projects that match the condition and with WHERE false for projects that don’t match the condition. In this case, the engine will recognize that the query will return no results, so in practice there will be no execution. In conclusion, you’ll pay for the cost of coordination and for the cost of local replanning, but the actual execution will be a no-op.

Flexibility:

• In the Technical Preview release, project_routing only allows filtering based on project aliases. Future releases will support more complex syntax and all tags.
• The ES|QL language supports project tags in any command or expression where a field name is allowed, including filters, even with very complex expressions.

Conclusion

Both project routing and project tag fields give you control over which projects participate in a cross-project search, but they serve different purposes.

Use project routing when performance matters most. Whether you set project_routing in a Query DSL request body or use SET project_routing in ES|QL, the effect is the same: Non-matching projects are excluded before any query work begins; no index resolution, no coordination overhead, no wasted compute. If you know exactly which projects to target, this is always the most efficient path.

Use project tag fields when you need flexibility. Project tags behave like regular fields in both Query DSL and ES|QL. You can filter, aggregate, sort, and include them in output. This opens up scenarios that routing alone cannot handle, such as combining tags with Boolean logic, grouping results by project, or filtering on custom tags like “env” or “department”. The trade-off is that all linked projects are still contacted, even if the filter ultimately excludes their data.

Combine both for the best of both worlds. Start with project routing to narrow down to the relevant projects, and then use project tag fields within your query for finer-grained logic. This gives you the performance benefits of early exclusion with the expressiveness of full query support.

For a high-level tour of metrics analytics across ES|QL and Discover, see Explore and Analyze Metrics with Ease in Elastic Observability. This post zooms in on the mechanics.

Here is the mental model in five bullets:

• FROM treats every document as an independent row. That is right for events, but metric aggregations often need the time series that each row belongs to.
• TS adds that time series context: it groups and aggregates data points by time series before any other aggregation runs, and enables functions like RATE, AVG_OVER_TIME, and LAST_OVER_TIME.
• A TS | STATS query normally has two aggregation phases. The inner phase reduces samples inside each time series; the outer phase groups and combines those per-series results.
• The default inner aggregation is LAST_OVER_TIME, which is why TS metrics | STATS AVG(cpu_usage) and FROM metrics | STATS AVG(cpu_usage) can return different numbers.
• Use TS to query a time series data stream (TSDS). Use FROM for events and raw document inspection.

What is a time series, really?

A time series is a sequence of (timestamp, value) data points identified by the metric name and a unique set of dimension values.

For example, request_count reported every 30 seconds by host h1 in data center dc1 is one time series. The same metric on host h2 in dc1 is a different time series.

In a time series data stream, every metric document carries an internal _tsid field that uniquely identifies a time series. Samples that share a _tsid belong to the same time series and are stored sequentially, sorted by timestamp.

That storage layout enables efficient per-series aggregations. It also explains why TS only works on time series data streams. Other index modes have no notion of a time series, so the per-series operations TS relies on have no such identifier to attach to. FROM does not support those operations, which is what the next section is about.

Why FROM leaves metrics on the table

Consider a counter named request_count collected every 30 seconds from three hosts.

A counter is a cumulative metric: each sample is the running total since the process started reporting it. For request_count, a value of 1,000 means "this time series has observed 1,000 requests so far", not "1,000 requests happened since the previous sample". Counters reset to zero on process restart, so a sample of 4 right after 1,004 is a fresh count, not negative traffic. The ES|QL RATE function computes the per-second change within a time series and handles resets without glitches.

You want to calculate the total request rate across all hosts, bucketed by 5 minutes.

If you are used to writing ES|QL over event data, you might start with this query:

FROM metrics-*
| WHERE TRANGE(1h)
| STATS SUM(request_count) BY BUCKET(@timestamp, 5m)

The chart it produces looks plausible at first: a line that goes up over time. But the number on the y-axis is the sum of every cumulative counter value reported in the bucket. Each host contributes its own running total, repeatedly, once per sample. Because the query uses SUM on those cumulative values, the result is not a rate, it is not the number of requests in the bucket, and it grows without bound even if the application stops receiving requests.

request_count is a monotonically increasing counter, so its raw values represent "how many requests have ever happened on this host", not how many happened in the bucket. The right computation is "how much did this counter increase per second on each host, then sum across hosts." FROM cannot express that operation directly. It can group rows by fields, but it has no built-in notion of "the same time series over time" and no way to ask for the change of a counter within each time series. It also cannot use sliding-window time series functions such as RATE(request_count, 5m), which we will come back to below.

TS was introduced for this purpose, providing a succinct syntax to express time series aggregations:

TS metrics-*
| WHERE TRANGE(1h)
| STATS SUM(RATE(request_count)) BY TBUCKET(5m)

RATE(request_count) runs per time series and produces a per-second rate that handles counter resets correctly. SUM then adds those rates across hosts.

Two aggregation phases: inner and outer

Every TS | STATS query has two distinct aggregation phases.

Let's make that concrete with a query that calculates the request rate per data center:

TS metrics-*
| WHERE TRANGE(1h)
| STATS SUM(RATE(request_count)) BY datacenter, TBUCKET(5m)

The diagram below shows how TS evaluates this query. It first reduces samples inside each time series, then groups and combines those per-series values into one result per datacenter and time bucket.

The phases are:

Inner (within a time series). Runs separately for each time series. It collapses many (timestamp, value) data points within a bucket into a single value per time series per bucket by applying the inner aggregation function, such as RATE in the example above. Functions: RATE, AVG_OVER_TIME, MAX_OVER_TIME, LAST_OVER_TIME, STDDEV_OVER_TIME, and so on. The full list is on the time series aggregation functions page.

Outer (across time series, the "grouping" phase). Combines the per-series values into a single value per group per bucket. Functions: SUM, AVG, MAX, MIN, percentiles, and the rest of the regular ES|QL aggregates.

In SUM(RATE(request_count)) BY datacenter, TBUCKET(5m):

• RATE(request_count) is the inner aggregation. It runs per time series.
• SUM(...) is the outer aggregation. It combines time series within the same datacenter and bucket.
• TBUCKET(5m) defines the bucket boundaries (equivalent to BUCKET(@timestamp, 5m)).

The outer aggregation is optional. If you only need the per-time-series result, use the time series aggregation function directly:

TS metrics-*
| WHERE TRANGE(1h)
| STATS request_rate = RATE(request_count) BY TBUCKET(5m)

That query keeps the per-series rate for each bucket instead of wrapping it in SUM, AVG, or another aggregate across time series.

The default inner aggregation: LAST_OVER_TIME

TS has to reduce raw samples inside each time series before it can run the outer aggregation. That means every metric field in a TS | STATS aggregation needs an inner aggregation, even when the query does not spell one out.

Consider a metric named cpu_usage. It is a gauge: a metric that captures a value at a point in time and can move up and down freely. A sample of 0.42 means "this host is at 42% CPU at this time". For a gauge, the natural "value in this bucket" is the most recent sample.

That is what ES|QL fills in for you. If you write TS metrics | STATS AVG(cpu_usage) BY host.name, TBUCKET(5m), the implicit inner aggregation is LAST_OVER_TIME(cpu_usage) and the query is equivalent to:

TS metrics
| WHERE TRANGE(1h)
| STATS AVG(LAST_OVER_TIME(cpu_usage)) BY host.name, TBUCKET(5m)

For each time series, LAST_OVER_TIME picks the latest sample in the bucket. Then AVG averages across time series.

It is also why the same-looking query against FROM and TS can return different numbers. FROM averages every individual document. TS averages one value per time series per bucket. If your hosts publish at slightly different rates, those averages diverge. For example, in a five-minute bucket, a host that publishes every second contributes 300 documents while a host that publishes every two minutes contributes only two or three. With FROM | STATS AVG(cpu_usage), the chatty host dominates the average. With TS, each time series is reduced to one bucket value first, so the outer average gives each host one value to contribute.

If you want the average value during the bucket instead of the latest value, make the inner aggregation explicit:

TS metrics-*
| WHERE TRANGE(1h)
| STATS AVG(AVG_OVER_TIME(cpu_usage)) BY host.name, TBUCKET(5m)

AVG_OVER_TIME averages all CPU utilization samples within each time series. The outer AVG then averages those per-series values across matching hosts. That makes the result sample-weighted within each time series, then equally weighted across time series. Use this when you care about how the value behaved during the bucket, not just where it ended up.

The same rule applies to peaks and troughs. For a peak CPU chart, use MAX(MAX_OVER_TIME(cpu_usage)), not just MAX(cpu_usage). The inner MAX_OVER_TIME finds the peak within each time series; the outer MAX finds the peak across matching time series.

Counters work the other way around. Their sample value is a running total, so the latest sample on its own is rarely meaningful. For a counter, the inner aggregation you almost always want is RATE for a per-second rate, or INCREASE for the total change in the bucket. Falling back on the default LAST_OVER_TIME gives you the most recent cumulative value, which is the trap the FROM query in the previous section walked into.

Pick the inner function deliberately. The outer function is the easy part.

When to use TS, when to use FROM

A practical rule of thumb:

• Use TS for metric aggregations against a time series data stream. It is the source command designed for that data, and it applies per-series semantics by default.
• Use FROM for events: logs, traces, audit records, transactions. Each row is independent. There is no time series context.

FROM still works on TSDS indices and is occasionally useful, for example when you want to inspect raw metric documents without per-series grouping. For dashboards, alerts, and any kind of charting, TS is the right default.

If you first need to discover which metrics or time series exist in the data, use METRICS_INFO or TS_INFO after TS and before STATS. See ES|QL METRICS_INFO and TS_INFO: Catalog your time series data for a deeper walkthrough.

Post-process TS results with ES|QL

The first STATS command is the boundary between time series processing and regular ES|QL processing. Before that first STATS, TS needs to keep the data grouped by _tsid, so commands that change row order or shape are not allowed. After that first STATS, the output is a regular ES|QL table. You can sort it, limit it, join lookup data, enrich it, or compute derived columns.

For example, this query calculates average CPU per host and bucket, finds the maximum bucketed average for each host, and returns the ratio:

TS metrics-*
| WHERE TRANGE(1h)
| STATS avg_cpu = AVG(AVG_OVER_TIME(cpu_usage)) BY host.name, time_bucket = TBUCKET(5m)
| INLINE STATS max_avg_cpu = MAX(avg_cpu) BY host.name
| EVAL cpu_ratio = avg_cpu / max_avg_cpu
| KEEP host.name, time_bucket, cpu_ratio
| SORT host.name, time_bucket DESC

Sliding windows for the inner aggregation

Time series aggregation functions accept a second argument: the window size for the inner phase.

TS metrics-*
| WHERE TRANGE(1h)
| STATS AVG(RATE(app.requests, 5m)) BY TBUCKET(1m)

This computes the rate over a 5-minute sliding window, but reports a value every minute. It is useful when you want a smoother chart at fine bucket sizes.

The window is the ES|QL counterpart to a PromQL range vector selector: RATE(app.requests, 5m) serves the same purpose as rate(app_requests[5m]).

Gotchas worth knowing

A few things in TS can seem surprising, especially when coming from the events-based FROM mental model. None of these are bugs; most are direct consequences of the per-series model. Here is what to watch for.

COUNT(*) is rejected. Say you want to know how many samples were collected per service in each bucket. The instinct from FROM is COUNT(*), but TS rejects it: there is no plain "row" once data is grouped by time series, so a row count has no defined meaning. Pick what you actually want to count:

• Number of samples per service: STATS samples = SUM(COUNT_OVER_TIME(cpu_usage)) BY service.name, TBUCKET(5m). The inner COUNT_OVER_TIME counts samples per time series; the outer SUM adds them across the time series in the group.
• Number of distinct hosts reporting per service: STATS hosts = COUNT_DISTINCT(host.name) BY service.name, TBUCKET(5m). This counts unique label values across time series.

You cannot sort, limit, lookup join, or enrich before STATS. TS metrics | SORT @timestamp | STATS ... will fail. The grouping by _tsid must happen first, before anything else can run. Filter with WHERE if you need to narrow the scope. After the first STATS, the output is regular ES|QL and you can pipe it through any command, as shown in the previous section.

Gauge vs counter mapping. Time series functions are sensitive to the metric type set in the field mapping. RATE only works on counters; *_OVER_TIME functions are intended for gauges. If you build TSDS mappings by hand, pay special attention to this part.

This can be a source of friction for Prometheus users. Prometheus metric type metadata is not always available in the data Elasticsearch receives, so the metric type may have to be inferred from naming conventions (_total for counters, and so on). Those heuristics are imperfect, and a misclassified metric is rejected by the function that should accept it. The deeper mechanics, including how Prometheus Remote Write maps metric types into TSDS, are covered in How Prometheus Remote Write Ingestion Works in Elasticsearch.

Explicit converter functions (gauge-to-counter and counter-to-gauge) are on the roadmap to make these cases easier to recover from at query time.

Kibana charts go empty when you zoom in too far. In Kibana, TBUCKET adapts to the date picker, so zooming in shrinks the bucket size. When the bucket size drops below the data's collection interval, every other bucket has no sample, RATE and the rest return null, and the chart silently goes blank. Elastic is evaluating mitigations such as a runtime warning when the bucket size is too small, a configurable minimum bucket size, or automatic widening of the window or bucket size.

Wrap up

For metric queries, start with TS unless you specifically need raw documents. Then choose the inner aggregation based on what the value should mean inside each time series: RATE for counters, LAST_OVER_TIME for current gauge values, and explicit *_OVER_TIME functions for peaks, averages, minimum values, or distributions.

Once the per-series value is right, the outer aggregation is the familiar part: group and reduce those time series into the chart, alert, or table you need.

For the full reference, see the TS command docs and the list of time series aggregation functions.

One line: Speed that scales with your data

On billions of documents, analytical queries push against a real efficiency-precision trade-off. We’ve been hard at work pushing back. Our native columnar support is one of the best there are. ES|QL itself is a fast, purpose-built analytical engine, getting ever smarter at aggregation execution. And Elasticsearch ships a steady stream of efficiency innovations, like Block k-dimensional (BKD) tree pruning, with more landing all the time.

But even with all of that, native approximate queries really shine through. Starting in Elasticsearch 9.4, ES|QL supports approximate query execution. All you have to do is add one line to your queries: Prepend SET approximation = true. Now Elasticsearch will automatically sample a subset of your data, run the aggregation on that sample, extrapolate the results, and report confidence intervals. All transparently.

SET approximation = true;
FROM logs-*
| STATS count = COUNT(*) BY time = BUCKET(@timestamp, 5 MINUTE)
| SORT time

Your existing query stays unchanged. The SET directive tells Elasticsearch to handle the sampling, extrapolation, and statistical validation for you. No query rewriting, no manual sampling math, no guessing at sampling probabilities.

SET approximation = true is a forward-compatible directive. Today, it speeds up the most heavily used aggregations. As we expand support to more capabilities, your existing queries benefit automatically. Queries that aren’t yet approximated run exactly without errors; a warning header explains why.

How much faster?

On the ClickBench benchmark, well-behaved analytical queries ran on average 23x faster with confidence intervals enabled. Individual queries hit ~100x. Disabling confidence interval computation, the highest-leverage queries land near ~300x.

The advantage grows as your datasets grow. Approximate-mode cost is capped by the configured sample size, while exact-execution cost scales with row count. Doubling your index doubles exact-query time but barely changes approximate-query time for the same accuracy! This is a beautiful property of the underlying math, not an engineering trick, and it’s why approximation gets more valuable as you scale.

Speedup also depends on query shape, grouping cardinality, and sample size. See the FAQ for the full set of factors and tuning tips. Read “Fast approximate ES|QL” in two parts ([Part 1], [Part 2]), straight from the creators of the feature.

What you get back

The response includes your original aggregated values, automatically scaled to represent the full dataset; a COUNT on a 1% sample comes back as the estimated total, not the sample count. Column names and types are preserved (backwards-compatible). Plus two additional columns per approximated value:

• Confidence interval: A range that bounds the true value at the configured confidence level (default 90%). For example, a count of 268,473 with interval [264,444–273,179] means you can be 90% confident the true count falls in that range.
• Certified flag: A Boolean indicating whether the confidence interval for that value meets formal statistical guarantees. When certified is true, the data distribution allows us to rely on the results at face value. When false, the approximation is still often close but we can’t claim the same formal guarantees, typically because the distribution may be highly skewed or involve too few documents in a group. Think of it as the difference between "statistically proven" and "best estimate."

This is a deliberate design choice: Consumers that don't care about the confidence metadata can choose to not compute them at all (see “Granular control when you need it”) or ignore the extra columns and use the results exactly as before. Consumers that do care (like AI agents that read the results programmatically) get everything they need without a second query.

Use cases for approximate queries in ES|QL: Where this matters

AI agents and agentic workflows

Approximate queries don’t just speed up agent queries; they enable a scan-then-enhance investigation pattern that wasn’t practical at scale before. An agent can sweep billions of documents in sub-second time, identify candidates, and zoom in for exact answers, all inside a single reasoning loop. The certified flag turns approximation into a decision signal: Proceed at face value when it’s true, escalate to an exact query when it’s false and the step needs a tight guarantee. As ES|QL becomes the foundation for agentic analytics in Elastic, approximation is the speed layer that makes investigation possible at this scale.

Dashboards and charts on large datasets

Dashboards that aggregate weeks or months of data can become sluggish as data volumes grow. With SET approximation = true, the same dashboard loads faster. In the future, Kibana will inject the setting transparently, so users won't need to know it's happening; they’ll just see faster charts.

Log pattern analysis in ES|QL at scale

CATEGORIZE, GROK, and regex-heavy conditions are among the most compute-intensive parts of ES|QL because they require nontrivial compute per document. With approximate execution enabled, these large-scale pattern and exploration workflows become practical on very large indices.

Exploratory analysis and hypothesis testing

When you're exploring data to form hypotheses, for example, "Which services have the highest error rates this week?", you rarely need exact counts. You need shapes, relative magnitudes, and outliers. Approximate mode gives you those at interactive speed, and the confidence intervals tell you when to switch back to exact mode for the final answer.

How approximate queries work in ES|QL, without the math

The speedup is real engineering, not a query-planner trick. Sampling happens at the Lucene layer: Elasticsearch reads only the documents in the sample, so I/O and compute savings are proportional to the sampling rate. The aggregation runs on the sample, and the result is automatically scaled to represent the full dataset.

Confidence intervals are computed by a bootstrap procedure over multiple sub-partitions of the sample: statistically rigorous, not a heuristic or a guess. This is what backs the certified flag: When the methodology’s assumptions are met, the intervals carry formal guarantees.

Granular control when you need it

The defaults are designed to work well out of the box, but you can tune them:

SET approximation = {"rows": 500000, "confidence_level": 0.95};
FROM logs-*
| STATS count = COUNT(*), avg_duration = AVG(duration) BY service.name

• rows: How many documents to sample (default: 100,000 for ungrouped queries, 1,000,000 for grouped). More rows means higher accuracy and longer runtime.
• confidence_level: The confidence level for intervals. Defaults to: 0.9. Set it to a higher level for an increased probability that the value is within the confidence interval.
• Skip confidence intervals for maximum speed: Set confidence_level to null, and Elasticsearch returns just the point estimates, adding another 2–5x speed on top of approximate execution. This is how the highest-leverage queries land near 300x.

What's next

SET approximation = true is a forward-compatible directive. As we add support for FORK, JOIN, chained STATS, and additional aggregations, your existing queries automatically benefit.

Future work also includes tighter integration with Kibana so dashboards and Discover can enable approximation automatically and improved handling of highly skewed grouping fields.

Additionally, we’ll make approximate queries natively accessible to agents, so they can opt into fast execution as part of their analytics tools and reasoning loop.

Get started

Approximate queries are available in Elasticsearch 9.4 as a technical preview on the Enterprise subscription tier. Add SET approximation = true; to the beginning of your query, and see the difference. Check the ES|QL SET command reference for configuration options.

FAQ

What is approximate query execution in Elasticsearch?

Approximate query execution is a mode where Elasticsearch samples a subset of your data, runs the aggregation on the sample, and extrapolates the result to represent the full dataset. You get back the estimated value plus a confidence interval showing how much to trust it. It's controlled by a single SET directive prepended to your existing ES|QL query; no query rewriting required.

How do I speed up ES|QL aggregations without reducing my data retention?

Just add SET approximation = true to your query. Approximate execution samples at query time, not at index time. Your data stays fully indexed, fully retained, and queryable both exactly and approximately. Elasticsearch handles sampling and extrapolation on the fly. Drop the directive any time you want exact results; nothing about the underlying data changes.

How much faster are approximate queries?

On the ClickBench benchmark, aggregation-heavy ES|QL queries that are well-suited to sampling typically run 10–40x faster with confidence intervals enabled, with individual queries reaching 100x or more. Disabling confidence interval computation (SET approximation = {"confidence_level": null}) adds another 2–5x on top, so the highest-leverage queries hit nearly 300x. The advantage grows with dataset size: Sampling cost is capped by the configured sample size, while exact execution cost scales with the row count, so the bigger your index, the bigger the win for the same precision.

How accurate are approximate queries? Can I trust the results?

Each approximated value comes back with two signals: a confidence interval (a range bounding the true value at a configurable confidence level) and a certified Boolean flag. When certified is true, the confidence interval carries formal statistical guarantees. When false, the result is still often close, but the data distribution didn't meet the assumptions required for a formal guarantee. Accuracy depends on data characteristics and query shape, not on document count, so speedup gains increase as your dataset grows.

What does the speedup depend on?

Five main factors:

• Dataset size. Larger datasets produce larger speedups, for the reason described above (exact scans grow with N; sampled scans don’t).
• Query shape. Queries that scan a lot to compute relatively little (large STATS, especially MEDIAN and PERCENTILE) benefit most. Queries that are already cheap (small WHERE filters matching few rows, or simple counts that hit indexed summary statistics) see little speedup.
• Grouping cardinality and distribution. Well-distributed BY fields with healthy per-group sample counts benefit cleanly. Very sparse or highly skewed grouping (for example, a near-unique field or a long tail of rare values) can erode the gain because rare groups end up with too few sampled documents.
• Confidence interval computation. Computing intervals adds overhead. Set confidence_level to null, and you trade interval reporting for an additional 2–5x speedup.
• Sample size. The defaults (100k for ungrouped STATS, 1M for STATS … BY) work well for most queries. Increasing rows improves accuracy on high-cardinality grouping at the cost of some speedup; decreasing it does the reverse.

Can I use approximate queries for log analysis and pattern detection?

Yes. CATEGORIZE, GROK, and regex-heavy conditions are among the most compute-intensive operations in ES|QL because they require per-document processing. With SET approximation = true, these operations run on a sampled subset instead of the full index, making large-scale log pattern analysis and exploration fast on very large datasets.

Do I have to rewrite my ES|QL queries to use approximate mode?

No. Prepend SET approximation = true to your existing query. The aggregation expressions, column names, and output types stay the same. The response adds two columns per approximated value (the confidence interval and the certified flag), but existing consumers that don't use those columns see no breaking change.

What aggregations does approximate mode support in 9.4?

COUNT, SUM, AVG, WEIGHTED_AVG, MEDIAN, PERCENTILE (except extremes), MEDIAN_ABSOLUTE_DEVIATION, and STD_DEV (with caveats for highly skewed distributions). More coverage on the way.

Will I get the same result twice for the same query?

Not exactly. Approximate execution randomly samples documents at query time, so successive runs of the same query return slightly different point estimates and confidence intervals. The variation between runs is small relative to the confidence interval each run reports. If you need bit-for-bit reproducibility, run the exact query. For dashboards, depending on the use case, the variation can typically be smaller than the visual resolution of the chart.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In most ecommerce deployments, economic signals, like profit margin and product popularity, are either ignored in search ranking or applied as static, global weights. A fixed margin boost might push high-margin products up across every query, which works for "chocolate" (where shoppers are open to suggestion) but backfires for "baby formula" (where shoppers want the trusted, popular brand).

The governed control plane makes it possible to treat economic optimization as a per-query decision, expressed as policy data and managed through the same admin UI as every other governance mechanism. A merchandiser can say "for chocolate queries, prioritize margin" and "for baby formula queries, prioritize popularity", without writing code, without deploying changes, and with full auditability.

For the mathematical foundation of margin and popularity boosting in Elasticsearch, including the logarithmic scaling formula and factor tuning explanation, see Boosting e-commerce search by profit and popularity with the function score query in Elasticsearch.

Two business signals: Margin and popularity

Every product document in our product catalog carries two numeric fields:

• margin: The product's profit margin as a percentage (0 to 200 in our dataset).
• popularity: A relative sales volume metric (0 to 10,000 in our dataset), such as weekly average units sold.

These fields represent two fundamentally different business objectives. Margin optimization pushes profit per transaction. Popularity optimization pushes conversion probability since products that many shoppers buy are products that the current shopper is likely to buy.

The baseline: Global boosting with business signals

Before introducing per-query policy overrides, the system applies a default boost for both margin and popularity. These are implemented using Elasticsearch's field_value_factor with logarithmic scaling inside a function_score query as described in Boosting e-commerce search by profit and popularity with the function score query in Elasticsearch.

The design has three properties worth noting:

• Calibrated range. Each signal's factor is calibrated so that it contributes at most approximately +1.0 to the boost multiplier at the top of its range. Combined with a baseline weight of 1, the final multiplier ranges from 1.0 (a product with zero margin and zero popularity) to approximately 3.0 (maximum margin plus maximum popularity). A product with strong business signals scores roughly 3x higher than an identical product with none, regardless of the BM25 score magnitude.
• Logarithmic scaling. The ln1p modifier grows fast at small values (rewarding incremental gains) but flattens at high values (preventing runaway scores from a single dominant product). This also makes the system resilient to data distribution changes: If the maximum popularity in a dataset shifts significantly, the boost curve stretches rather than breaking.
• Multiplicative, not additive. The business-signal boost is applied multiplicatively against BM25 (boost_mode: "multiply") rather than added to it. BM25 scores vary dramatically across queries, so an additive boost would have inconsistent impact depending on query specificity. Multiplicative scaling guarantees a consistent percentage uplift regardless of the absolute BM25 magnitude.

Per-query boosting overrides through policies

The default weights (1.0 for both margin and popularity) apply to every query. But the governed control plane makes it possible to override these weights on a per-query basis through the same policy engine described in Part 3 and Part 4.

Each policy document has two optional fields: margin_boost_weight and popularity_boost_weight. When a policy matches a query and includes weight overrides, those values flow through to the function_score construction, replacing the defaults.

Why per-query control matters

Consider two queries and why they demand different economic optimization strategies.

Margin boosting: Chocolate

A shopper searching for "chocolate" is browsing. They'll be satisfied by many chocolate-related products. The retailer's store-brand chocolate truffles at 60% margin might be just as appealing as the name-brand bar at 15% margin. Aggressive margin boosting pays for itself if the shopper doesn't care which chocolate they purchase and buys one of the margin-boosted hits.

Chocolate results without margin boosting

To isolate the effect of per-query margin boosting, we first disable margin boosting entirely for this query (margin boost weight: 0). Without any margin signal, the ranking is driven by text relevance. In our dataset, the first hit has a margin of 10 and the next one has a margin of 84 (out of a max of 200) as follows:

Setting a margin boost on queries for “chocolate”

A merchandiser who decides that “chocolate” queries should prioritize margin makes that change in the admin UI, tests it against representative queries, and promotes it to production. The change takes effect on the next query. No engineering ticket, no deployment, no code change. The following "chocolate" policy sets margin_boost_weight: 3.0, which ensures that searches for chocolate will aggressively promote high-margin items.

Chocolate results with margin boosting

With the above margin boost policy enabled, the higher-margin chocolates with a margin of 197 and 184 are boosted to the top of the results as follows (remember that the maximum margin in our dataset is 200):

Popularity boosting: Baby formula

A parent searching for baby formula is not experimenting. They want the product that other parents trust. Pushing a high-margin store-brand formula above the established brand that thousands of parents are buying would feel wrong and erode trust. Popularity is the right signal here because it functions as social proof for a high-stakes purchase.

Baby formula results without a popularity boost

To isolate the effect of per-query popularity boosting, we first disable popularity boosting entirely for this query (popularity_boost_weight: 0). Without any popularity signal, the ranking is driven by text relevance. In this example, the top hit has a popularity of 50 on a scale that goes up to 10,000.

Setting a popularity boost on queries for “baby formula”

A "baby formula" policy sets popularity_boost_weight: 5.0 and margin_boost_weight: 0; formula searches prioritize what's popular, completely ignoring margin.

Baby formula results with popularity boosting

If we enable the above rule, then the most popular baby formula (Lactogen 2 with a popularity of 9979) will be boosted to the top of the results, as shown below.

Disabling business signals: Clearance

Not every query benefits from economic boosting. A shopper searching for "clearance" is looking for deals; margin and popularity are both irrelevant to that intent. A high-margin product is the opposite of what the shopper wants, and a popular product may not be on clearance at all.

A "clearance" policy sets margin_boost_weight: 0 and popularity_boost_weight: 0, which disables both business signals entirely. Results are ranked on pure text relevance with no economic influence. This completes the design space: Policies can amplify either signal independently, rebalance them, or turn them off altogether.

How overrides flow through the control plane

When the percolator returns matching policies, the control plane checks for margin_boost_weight and popularity_boost_weight fields on the highest-priority matching policy. If present, those values replace the defaults in the RewriteState. If no matching policy includes weight overrides, the default values (1.0 for both) are used.

The weights then flow through to the function_score construction when the final Elasticsearch query is assembled. The structure of the function_score doesn't change; only the weight values on the margin and popularity functions.

Weight overrides participate in the same governance model as every other policy mechanism. They’re subject to priority ordering: A Christmas campaign policy with margin_boost_weight: 0.5 will override a product-category policy with margin_boost_weight: 3.0 if the campaign policy has higher priority. The cascading transformation model from Part 3 applies: Economic optimization parameters are just another field in the policy's execution plan.

Interaction with other policies

Per-query weight overrides compose naturally with the constraint enforcement, conflict resolution, and personalization mechanisms described in earlier parts of this series.

Consider a search for "cheap chocolate" during a Christmas campaign, with a shopper who has a purchase history and belongs to a vegan cohort. The control plane processes this query through the full governance stack:

1. The "cheap" policy extracts the price constraint and removes "cheap" from the query.
2. The "chocolate" policy sets margin_boost_weight: 3.0 and constrains results to chocolate categories.
3. The “Christmas campaign” policy (higher priority) overrides the category constraint with seasonal categories and adjusts the price ceiling.
4. The “vegan cohort” policy applies a soft boost to vegan-certified products.
5. The margin and popularity boosts are applied with the governed weights (margin at 3.0× from the “chocolate” policy, popularity at the default 1.0×).
6. The shopper's purchase history boosts are applied as the outermost scoring layer.

Every layer stacks multiplicatively. The economic optimization weights are governed by the same policy framework that controls category constraints, campaign overrides, and cohort-specific boosts. A merchandiser can tune all of these through the admin UI, all without code changes.

This example also illustrates where economic optimization sits in the scoring stack. The layers nest in a deliberate order: the base query (keyword or semantic match), then governance constraints (hard filters and soft boosts from Part 3 and Part 4), then business-signal boosts (margin and popularity with governed weights), and then purchase history personalization (Part 6). Each layer wraps the previous one, and the effects compound multiplicatively. Governance controls what appears. Economic optimization influences what ranks highest from the retailer's perspective. Personalization adjusts ranking further from the shopper's perspective.

Tuning guidance

The factor values in the baseline function_score are calibrated for the demo dataset's field ranges. A production deployment with substantially different ranges for margin or popularity should recalibrate the factors so that each signal contributes a consistent maximum boost. The logarithmic scaling provides built-in resilience to outliers and distribution shifts, but the factors are worth reviewing whenever the underlying data changes significantly. For the calibration methodology, see Boosting e-commerce search by profit and popularity with the function score query in Elasticsearch.

From economic optimization to agentic AI

The governed control plane now handles intent classification, constraint enforcement, conflict resolution, personalization, and economic optimization, all expressed as policy data, all managed through a business-editable admin UI, and all composable through a deterministic transformation framework.

The final post in this series asks what happens when the input to this system isn’t a search string typed by a human shopper, but an intent string extracted by an AI agent, and why the deterministic properties of the governed control plane become even more critical when the upstream decision-maker is probabilistic.

Put governed ecommerce search into practice

The per-query economic optimization described in this post (policy-governed margin and popularity weights composing with governance constraints, personalization, and campaign overrides) was designed and built by Elastic Services Engineering as part of our repeatable ecommerce search accelerators. Contact Elastic Professional Services.

Join the discussion

Have questions about search governance, retrieval strategies, or ecommerce search architecture? Join the broader Elastic community conversation.

What’s hard about filtering partition indices

With partition indices, all searches are done in two phases:

• Find the nearest centroids.
• Find all the nearest vectors within the nearest centroids’ clusters.

For DiskBBQ, the centroids are quantized and possibly indexed in their own structure. The cluster contents (we’ll call them postings lists from now on) are laid out in an effort to make scoring the vectors fast. Postings are stored in blocks of 32, with each block in doc_id order. Doc IDs are delta-encoded to minimize disk usage with low decode overhead. Vector values are also block-encoded, separating dimensions from quantized corrections to maximize single instruction, multiple data (SIMD) throughput using our optimized kernels.

Vector Cluster (Posting List) layout
  | metadata |
  | doc_deltas[32] | vec_quant[32] | vec_quant_corrections[32] |
  | doc_deltas[32] | vec_quant[32] | vec_quant_corrections[32] |
  | ... |
  | doc_deltas[T]  | vec_quant[T]  | vec_quant_corrections[T]  |   (T <= 32)

Once we reach a postings list, the layout is optimized for fast scoring and filtering. For example, if an index sort is provided, blocks of vectors that match a filter will be stored and scored together within a list. This unlocks scoring contiguous blocks of vectors at a time, taking full advantage of the underlying CPU throughput.

That said, we don’t know if a cluster matches a given filter until we actually check its doc_ids. Once verified, we can be sure to only score against the relevant vectors. In restrictive-filter cases, we can inspect a centroid and still find that none of its vectors match the filter. To compensate, we keep scoring and exploring centroids until we get a representative group of vectors scored.

This meant wasted work for restrictive filters. We score centroids, not knowing if they have vectors relevant to the filter or not; prepare to score the postings list, only to find that none of the blocks apply. The wasted compute adds up:

• Unnecessarily scoring the centroid.
• Loading a filtered-out postings list because it’s close to the query vector.
• Decoding and checking the document IDs in the list, only to find out none match.
• Continue the search, potentially hitting another completely filtered-out centroid, until we visit enough to get the desired recall.

Here’s an example showing the old flow. Check all the centroids, see what matches, move on to centroids that have matching vectors. Rinse and repeat.

How do we get to the right centroids quickly?

The simplest solution would be to skip centroids that contain no valid vectors. But we don’t want to index additional information for all potential filter fields and values. A user can provide many variations of complex or simple filters. This is a strength of Elasticsearch, and we don’t want to hamper that.

Instead, we simply store the mapping of doc_id -> centroid_ord. This gives us an immediate view of all docs and their centroid membership. Allowing us to iterate any provided filter in document order, quickly determining the relevant centroids. Of course, iterating every document to check if it passes a filter is not free. We only apply this eager logic if the average number of documents per cluster that match the filter is 1.25. Yes, this is a “magic number”; however, it's empirically based. Assuming the filter is random, we’re validating at least one matching vector per centroid with some overhead. We may refine this in the future, but early experimentation found this number to be a sweet spot for most users.

Here’s the new way. Detecting we have a restrictive filter, go straight to the filtered centroids.

Benchmark, benchmark, benchmark

Here’s a macro-benchmark with a random filter. The filter selectivity is purposefully extreme to show the significant improvement on hyper-restrictive filters. Here we see almost an order-of-magnitude improvement. Where before, when filters got very restrictive, there would be a horrible elbow. Now latency remains consistent and will in fact improve as filters get more restrictive.

A further validation is our nightly runs of so-vector with rally showing the improvement. You can try this yourself by specifying bbq_disk in the vector_index_type in the rally configuration.

What's next?

This is in Elastic Serverless now and will be in stack release 9.4.0. We aren’t done improving vector search in Elasticsearch. This is just another step in our journey to bring you simple, efficient, practical, and fast vector search. Thank you so much for using the code that we write. We ❤️ you.

They’re ECS-aligned, ready for Discover and out-of-the-box dashboards once you ship the log, no custom schema project. Below: why we built this, how it differs from slow logs, what each line contains, and how to turn it on.

Why we built this (you asked, a lot!)

Coordinator-level query logging has been a very popular request; we listened and delivered! The same pain kept showing up: You want the response duration for Service Level Objectives (SLOs) and dashboards. You want to know the execution time of queries executed in your cluster, and you want to be able to see the full query.

If using cross-cluster search, a search that fans out across clusters looks like one operation from the app or Kibana, but operationally it’s a chain of work: coordination, remote execution, merges, timeouts, and partial results. When something is slow or flaky, teams need to know not only how long the request took but also which clusters contributed and whether the outcome was success, partial, or a hard failure.

What you get: One log stream, one entry per query! Every entry has the coordinator duration (the very same took time that actually matches your search API response), success or failure, and the full query text. Elastic Common Schema–compliant (ECS) JSON, optional duration threshold and user/audit fields, plus X-Opaque-Id that lets you trace a hot query back to the saved object it originates from, and the trace ID so you can correlate with Kibana or your own tooling.

What’s more: Logs follow a stable, ECS-aligned schema, which means you don’t need to design your own ingestion pipelines or field mappings. This consistency enables out-of-the-box dashboards and analytics that work immediately once logs are shipped.

Slow logs vs. query logs: The 30-second version

Slow logs have been the go-to tool for years. They tell you which search operation is slow, but they emit one line per shard that took part, where each line reflects that shard’s piece of the work. This means that they don’t provide a single row that says how long the query execution took, from the client’s perspective. Query logs do exactly that: one line per query, with the end-to-end (wall clock) duration that lines up with the took time in the search API response. This makes them much better suited for understanding workload patterns and identifying problematic queries quickly.

Slow and query logs also differ in when they fire and what they cover. Slow logs only write when a shard’s slice breaches a duration threshold; that is, you’re optimized for “show me unusually slow shard work.” Query logs can record every query (or only those above a configurable threshold you set at the cluster level), so you can tune volume for analytics versus troubleshooting. Slow logs only support DSL queries, while query logs cover ES|QL, DSL, SQL, and EQL, which matches how you reason about “what ran on my cluster” in a modern stack. Both provide the same support in terms of correlation with headers, traces, and audit information (when you turn on user context).

The table below summarizes the main differences between the historical slow logs and the new query logs features.

What you get in each log line

Every line is one JSON object (one request) in a dedicated file (for example, *_querylog.json under your Elasticsearch log directory). Below is what you can do with the data:

Did it succeed, how long did it take, and what broke? Outcome (whether the request was successful or not), duration (took / took_millis, in line with the API), and a clear failure or timeout when something goes wrong. That’s the core signal for alerting, SLOs, and dashboards: “Are we green? If not, what’s the error?” You also get how many rows or hits came back (result_count), so you can separate “slow but empty” from “slow and huge.”

What actually ran? Query type (esql, dsl, sql, eql) plus the full query text. That answers “Which dashboard rule, saved search, or client pattern is hammering us?” Mix it with duration and outcome to find the worst offenders to fix or throttle.

Who asked for it, and how do I trace it end to end? X-Opaque-Id and trace ID tie a line back to Kibana or your own headers. Task and optional parent task IDs help follow work that was enqueued or chained (async or nested operations).

Cross-cluster search: Who participated, and did anyone misbehave? When cross-cluster search (CCS) is in play, the log can carry remote cluster aliases, per-cluster duration, and status (successful, failed, partial, skipped). You can see at a glance whether a slow search was local or a specific remote dragging the response. DSL can also record that a search was served from a remote alias; ES|QL exposes the richer cluster map; EQL logs a lighter view (for example, which remotes and how many) when remotes are involved.

Security (optional). With elasticsearch.querylog.include.user, you get the usual identity and realm fields (plus effective user when run-as applies), and API key metadata when applicable. Pair with query text and duration for governance and capacity conversations that use names, not only IPs.

There’s more available than we covered here, including additional execution details, shard-level outcomes, and optional profiling information depending on the query type. For every field path and setting, see the Elasticsearch documentation on query logs.

Where the logs live (and how to use them)

Logs land in your Elasticsearch log directory as *_querylog.json (for example, mycluster_querylog.json) on the coordinating node. Ship them with the querylog fileset in the Filebeat Elasticsearch module, so you can then inspect them in Discover (filter by event.dataset: elasticsearch.querylog). On Elastic Cloud, you need to enable Logs on your deployment, and the query logs are shipped as soon as you enable them.

Two workflows. If you need a one-off look to find out who’s hammering the cluster, what the query mix is, or a quick audit, just turn logging on, set a duration threshold so you only log what matters (for example, ≥ 1 s or ≥ 5 min), and then turn it off when you’re done. If you want ongoing query analytics, simply enable logging, point Filebeat at the log, and open a dashboard on the monitoring cluster. Two very simple steps, enable + ship, and you’re done. One request per line, one duration per request, no custom pipeline.

The dashboard below builds upon the new query logs and is provided out of the box. On the top row, you can find the P95/P99 query latencies (with an optional “acceptable latency” bar), the query type breakdown, the success and failure ratio, the user and system queries ratio, and (for DSL) hits versus aggregations. Underneath that, the latency over time (avg, p50, p95, p99, max) with a reference line so you can spot regressions, query volume over time (stacked by type), and tables for top indices, top users, and top error types. Filtering for cluster, user, or index lets you zoom into exactly what you want to focus on.

Heads up. Logging of queries is asynchronous, so it doesn’t block the query execution. Use the duration threshold to cap volume. Also worth noting that at very high queries per second (QPS), we may drop some lines rather than slow your cluster down. For analytics, shipping to a separate monitoring cluster keeps the cluster you’re debugging from taking the extra load.

Some configuration and code samples

Query logging is off by default. Flip it on in elasticsearch.yml or via the cluster settings API. Here’s how.

Enable query logging

In elasticsearch.yml:

elasticsearch.querylog.enabled: true

Or dynamically via the cluster settings API:

PUT _cluster/settings
{
  "persistent": {
    "elasticsearch.querylog.enabled": "true"
  }
}

Only log queries above a duration threshold

If you don’t want to log every health check or tiny request, simply set a threshold so only queries that run at least this long get an entry. Duration is in time units:

PUT _cluster/settings
{
  "persistent": {
    "elasticsearch.querylog.enabled": "true",
    "elasticsearch.querylog.threshold": "1s"
  }
}

Include user/audit information

If you use the Security plugin and want to see who ran each query:

PUT _cluster/settings
{
  "persistent": {
    "elasticsearch.querylog.enabled": "true",
    "elasticsearch.querylog.include.user": "true"
  }
}

Log DSL searches that hit only system indices

By default, searches that target only system indices aren’t logged. To include them, enable query logging and set:

PUT _cluster/settings
{
  "persistent": {
    "elasticsearch.querylog.enabled": "true",
    "elasticsearch.querylog.include.system_indices": "true"
  }
}

Example log entries

One line = one JSON object = one request with the same shape for ES|QL, DSL, SQL, EQL. Below: a successful DSL search and a failed EQL query with timestamp, duration, query type, and full query. On success, you get result count and shard stats, on failure an error block. User-inclusion and X-Opaque-Id show up when you’ve enabled them.

Success (DSL search):

{
  "@timestamp": "2026-03-04T19:40:34.736Z",
  "log": {
    "level": "INFO",
    "logger": "elasticsearch.querylog"
  },
  "event": {
    "duration": 1000000,
    "outcome": "success"
  },
  "elasticsearch": {
    "querylog": {
      "type": "dsl",
      "query": "{\"size\":10,\"query\":{\"match_all\":{\"boost\":1.0}}}",
      "indices": ["query_log_test_index"],
      "result_count": 3,
      "search": { "total_count": 3 },
      "shards": { "successful": 1 },
      "took": 1000000,
      "took_millis": 1
    },
    "node": { "name": "node-1" },
    "cluster": { "name": "my-es-cluster" }
  },
  "http": {
    "request": {
      "headers": { "x_opaque_id": "opaque-1772653234" }
    }
  },
  "user": {
    "name": "elastic",
    "realm": "reserved"
  }
}

Failure (EQL query):

{
  "@timestamp": "2026-03-04T19:40:35.271Z",
  "log": {
    "level": "INFO",
    "logger": "elasticsearch.querylog"
  },
  "event": {
    "duration": 1326334,
    "outcome": "failure"
  },
  "elasticsearch": {
    "querylog": {
      "type": "eql",
      "query": "any where true",
      "indices": ["nonexistent_index_xyz"],
      "result_count": 0,
      "took_millis": 1
    },
    "node": { "name": "node-1" },
    "cluster": { "name": "my-es-cluster" }
  },
  "error": {
    "type": "org.elasticsearch.index.IndexNotFoundException",
    "message": "no such index [Unknown index [nonexistent_index_xyz]]"
  }
}

Wrapping up

Elasticsearch query logs provide you with one single coordinator-level log for every query (ES|QL, DSL, SQL, EQL). One line per request, coordinator duration, full query, optional user and X-Opaque-Id. Enable it, set a duration threshold and user-inclusion if you want them, and you’re done. Logs live in your log dir (*_querylog.json), and when shipped with Filebeat, you can find them in Discover under the elasticsearch.querylog dataset.

Head to the Elasticsearch documentation on query logs for the full list of configuration settings, and field references. Slow or broken queries can also be found in AutoOps, which leverages the X-Opaque-Id to tie a long-running search back to its origin, such as a dashboard, a saved search, or an alerting rule.

Finally, it’s also worth noting that this new query log is an evolution of the ES|QL-only query log that we released in 9.2. We recommend adopting the new query log since it not only supports ES|QL queries, but also all your other queries.

Now, go see what’s actually running in your cluster.

Add JSON_EXTRACT for surgical extraction from raw JSON strings: flattened fields, embedded payloads, OTel resource attributes (60+ semantic conventions that standard mappings don't index individually). Together, these turn schema from a gate into a spectrum: indexed fields for speed, _source fallback for everything else. See also our companion posts on ES|QL views and ES|QL subqueries.

The mapping cliff edge

Elasticsearch mappings define how fields are indexed. When a mapping is complete, queries are fast; they hit inverted indices, doc values, and all the performance structures Elasticsearch builds at index time. But when a field was never mapped (missed during onboarding, added by a new integration, or simply not anticipated), it's invisible. Queries that reference it fail. The traditional fix: Update the mapping, reindex the data. For a multi-terabyte index, that means hours of reprocessing and doubled storage during the reindex window.

ES|QL schema-on-read changes the contract. The mapping is no longer a cliff edge where "mapped" means queryable and "unmapped" means invisible. Instead, it's a spectrum:

• Mapped fields → fast path. Queries hit indexed structures. This is still the preferred mode for production workloads.
• Unmapped fields with SET unmapped_fields="load" → queryable from _source at query time. Slower than the indexed path (no inverted index to accelerate filters), but the data is accessible in seconds instead of not at all: immediately, retroactively, against historical indices.

You can discover and query first and then decide later whether a field is worth mapping for performance.

Unmapped field access: The strategic abstraction

SET directives are query-level settings that appear at the top of an ES|QL query, before the FROM clause. They configure how the query engine behaves for that specific query without affecting anything else. The unmapped_fields directive controls what happens when a query references a field that doesn't exist in the mapping.

Consider an OTel logs index where resource.cost_center was never mapped. Without SET unmapped_fields, referencing it produces an error:

-- This fails: "Unknown column [resource.cost_center]"
FROM otel-logs-*
| STATS errors = COUNT(*) BY service.name, resource.cost_center

Add one line and the query works:

SET unmapped_fields="load";
FROM otel-logs-*
| WHERE log.level IN ("error", "warn")
| STATS errors = COUNT(*), latest = MAX(@timestamp)
    BY service.name, resource.cost_center
| SORT errors DESC

The field is loaded from _source at query time. No mapping change. No reindex. Works against data that was ingested weeks or months ago.

nullify is useful when you want queries to succeed even if some indices in a wildcard pattern don't have a particular field mapped. load is the mode you reach for when you actually need the data.

Partially mapped fields

The power of unmapped_fields really shows with wildcard index patterns where a field exists in some indices but not others. Suppose resource.team was mapped in recent indices but not in older ones:

SET unmapped_fields="load";
FROM otel-logs-*
| WHERE log.level IN ("error", "warn")
| STATS errors = COUNT(*), latest = MAX(@timestamp)
    BY service.name, resource.team
| SORT errors DESC

For indices where resource.team is mapped, values come from the fast indexed path. For indices where it isn't, values are loaded from _source. The query returns a unified result across the entire time range; no reindexing of historical data required.

JSON_EXTRACT: The lower-level tool

SET unmapped_fields is the right answer for fields that exist in _source under predictable names. But some data requires more surgical extraction, reaching into a JSON string stored in a text field or navigating nested objects inside a flattened field type where dot notation can't reach subkeys.

JSON_EXTRACT is the lower-level escape hatch. It takes a source field and a JSON path expression and returns the value at that path:

Extracting from string fields

Payment services often store structured error details as a JSON string in a response body field. JSON_EXTRACT reaches into that string at query time:

FROM svc-payments-*
| WHERE transaction.status IN ("failed", "timeout")
| EVAL error_code = JSON_EXTRACT(response_body, "$.error_code"),
       reason     = JSON_EXTRACT(response_body, "$.reason"),
       retryable  = JSON_EXTRACT(response_body, "$.retry")
| STATS failure_count = COUNT(*) BY error_code, reason, retryable
| SORT failure_count DESC

No ingest pipeline needed to extract error_code from the response body. The schema can evolve without reindexing.

Extracting from _source: Flattened fields and OTel data

For fields mapped as flattened, such as resource.attributes in many OTel indices, which can contain 20–30 nested keys per document, the entire JSON object is stored as an opaque token. Elasticsearch indexes the leaf values as flat keywords for filtering, but doesn't decompose them into separate mapped fields, so dot notation has nothing to resolve against. JSON_EXTRACT on _source follows the actual nesting in the stored document:

FROM svc-auth-* METADATA _source
| EVAL svc_version = JSON_EXTRACT(_source, "$.resource.attributes['service.version']"),
       env         = JSON_EXTRACT(_source, "$.resource.attributes['deployment.environment']"),
       host        = JSON_EXTRACT(_source, "$.resource.attributes['host.name']")
| STATS login_failures = COUNT(*) BY svc_version, env, host
| SORT login_failures DESC

Use dot notation for the nested structure and then bracket notation for leaf keys that contain dots (like service.version as a single key name). This is a common pattern for OpenTelemetry data.

What to use when

SET unmapped_fields is the abstraction most users should reach for first. JSON_EXTRACT is the tool for cases that need direct JSON manipulation or for data patterns like flattened fields that aren't natively handled yet.

How this compares

"Schema on read" is Splunk's founding narrative; the idea that you index everything as raw text and decide the schema at search time. ES|QL takes a fundamentally different position: You get both. Here's how that plays out in practice.

Splunk SPL extracts fields at search time using search commands like spath, rex, search-time field extractions, and calculated fields. The benefit is flexibility; you never need to declare a schema up front. The cost: field-value searches scan raw event data. Splunk's TSIDX files index metadata (host, source, sourcetype) and index-time fields, but user-defined search-time fields hit raw events on every query. A query that touches a billion events scans a billion events. Splunk compensates with summary indexing and data models, but those are manual, preconfigured accelerations that trade flexibility back for performance; the same trade-off Elasticsearch mappings make, just with extra steps. ES|QL's SET unmapped_fields gives you the same "query fields you never declared" flexibility, but mapped fields still hit Elasticsearch's indexed structures at full speed. You pay the _source scan cost only for the specific fields that aren't mapped, not for every field in the query.

Elasticsearch Query DSL has runtime fields and OpenSearch has derived fields — both extract from _source at query time without reindexing. But both require per-field configuration: Runtime fields need a Painless script and type declaration per field, either at the index mapping level or in each query; derived fields require index-level or cluster-level configuration. You have to know the field name and define the extraction logic before you can query it. SET unmapped_fields="load" is a per-query directive that skips all of that; one line, and every unmapped field in the index becomes queryable. No per-field definitions, no index settings changes, no scripts.

ClickHouse requires a strict schema for standard columns; adding one means ALTER TABLE. However, ClickHouse's JSON type (GA in 25.3) automatically creates typed dynamic subcolumns for every path encountered at insert time, with no per-field declaration needed. The limitation is retroactive access; data already ingested into a String column requires JSONExtract* functions for field access, similar to ES|QL's JSON_EXTRACT, and cannot be retroactively migrated to a JSON column without a data pipeline change. There's no equivalent to SET unmapped_fields that makes arbitrary historical fields queryable without touching the schema or re-ingesting.

The key difference: Splunk makes you choose between flexibility and performance at the platform level. ES|QL lets you choose per field, per query. Mapped fields are fast. Unmapped fields are accessible. You don't have to pick one model for your entire dataset.

Current constraints

SET unmapped_fields="load" is incompatible with subqueries and views in the current release; use nullify mode instead when composing queries. See the SET documentation for details.

What's next

Schema-on-read is a strategy, not just two features. The direction is to make more data queryable at query time without requiring a perfect schema at ingest time:

• Native flattened field support is next; dot notation directly into flattened fields without the JSON_EXTRACT on _source workaround. This eliminates the most common reason users need JSON_EXTRACT today and is planned.
• Lifting the load mode restriction for subqueries and views will let you combine schema-on-read with the composition primitives from our views and subqueries posts.

The long-term goal: SET unmapped_fields becomes the primary way users handle schema evolution, with JSON_EXTRACT reserved for truly surgical JSON manipulation.

Try it

Unmapped field access and JSON_EXTRACT are available as Tech Preview features. Try them in Kibana Dev Tools or Discover. We'd love your feedback; file a GitHub issue with the ES|QL label.

ES|QL unmapped field access and JSON_EXTRACT are Tech Preview features. Tech Preview features are subject to change and are not covered by the support SLA of GA features. The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

This post explains how the query, discovery, and metadata endpoints build on the earlier ingest and query work to form that API surface. Companion posts go deeper on individual pieces:

• Native PromQL support in ES|QL covers how PromQL queries are translated into ES|QL execution plans.
• Ship Prometheus Metrics to Elasticsearch with Remote Write covers ingestion setup.
• How Prometheus Remote Write Ingestion Works in Elasticsearch covers the remote write internals.

This is still a work in progress. The sections below call out what is supported today and which parts are still evolving.

The API surface

Today, the Prometheus-compatible API surface falls into three groups.

Query endpoints

The query endpoints let Prometheus-compatible clients evaluate PromQL expressions:

• GET /_prometheus/api/v1/query_range evaluates a PromQL expression over a time window (matrix results).
• GET /_prometheus/api/v1/query evaluates at a single point in time (vector results). Currently implemented as a short range query that returns the last sample.

Only GET is supported for query endpoints today. Some clients default to POST, so you may need to configure them to use GET. The Prometheus POST convention uses application/x-www-form-urlencoded bodies, which Elasticsearch's HTTP layer rejects as a CSRF safeguard before the request ever reaches the handler.

For the full PromQL coverage status, see the companion post on PromQL in ES|QL.

Metadata endpoints

The metadata endpoints serve the discovery information that clients need for autocomplete, variable dropdowns, and metric browsing.

The series, labels, and label values endpoints all accept match[] selectors and a time range (start/end). The match[] parameter takes a Prometheus series selector like http_requests_total{job="api"} and restricts the response to time series that match. This keeps responses fast and relevant on clusters with large numbers of metrics. For example:

GET /_prometheus/api/v1/series?match[]=http_requests_total{job="api"}

GET /_prometheus/api/v1/labels?match[]=http_requests_total

GET /_prometheus/api/v1/label/instance/values?match[]=http_requests_total{job="api"}

The first returns all series for http_requests_total where job="api", with their full label sets. The second returns only the label names that exist on http_requests_total series. The third returns only the instance values that appear on matching series.

GET /_prometheus/api/v1/metadata is different: it returns type and unit for each metric, optionally filtered by name via a metric parameter.

GET /_prometheus/api/v1/metadata?metric=http_requests_total

It does not accept match[] selectors or a time range. In Prometheus, metadata is collected from active scrape targets (the HELP, TYPE, and UNIT lines they expose), so the response does not involve a data scan. Elasticsearch does not have a dedicated metadata store like that, so the current implementation discovers metric metadata by visiting time series data from the last 24 hours. This keeps the query fast without requiring a full index scan. That 24-hour lookback is fixed today: the Prometheus metadata API does not expose start or end parameters that Elasticsearch could use to make it user-adjustable.

How the metadata endpoints work under the hood, including the TS_INFO and METRICS_INFO commands that power them, is covered below.

Index pre-filtering

All query and metadata endpoints accept an optional {index} path segment after /_prometheus/:

GET /_prometheus/metrics-prod-*/api/v1/query_range?query=up&start=...&end=...

This restricts which Elasticsearch indices the query runs against before any expression evaluation begins. On clusters with many data streams across teams or environments, this avoids scanning unrelated indices and can significantly reduce query latency. You can configure separate data sources per index pattern to give teams scoped access to their own metrics.

A note about Remote Write

For ingestion, Elasticsearch also exposes the standard Prometheus Remote Write endpoint:

• POST /_prometheus/api/v1/write ingests time series via the Prometheus Remote Write v1 protocol. v2 is not yet supported.

Remote Write writes into Elasticsearch's existing time series data streams (TSDS), not a separate Prometheus-specific storage layer. Prometheus labels become TSDS dimensions, and metric names become fields in the index mapping. The remote write architecture post covers the full mapping in detail, including how metric types are inferred and how labels are stored with a labels. prefix.

How it works

Under the hood, all endpoints work the same way: parse the incoming HTTP parameters, build an ES|QL query plan, execute it against time series data streams, and convert the columnar result back into the JSON format Prometheus clients expect.

TS_INFO and METRICS_INFO

The metadata endpoints need to answer questions like "what labels exist?" or "what metric types are defined?" across potentially millions of time series, without scanning every data point.

Internally, the Prometheus metadata endpoints answer those questions by building ES|QL plans around two new processing commands: METRICS_INFO and TS_INFO. You do not need to use these commands directly to use the Prometheus API, but they are the core execution primitives behind the metadata responses. Both work by visiting only one document per time series to extract its metadata, rather than scanning all samples. This means their cost scales with the number of distinct time series, not the number of data points.

METRICS_INFO returns one row per distinct metric with its name, type, unit, and associated dimension fields. TS_INFO is more granular: one row per (metric, time series) combination, including the actual dimension values as a JSON object.

TS metrics-*
| METRICS_INFO
| SORT metric_name

A dedicated blog post on TS_INFO and METRICS_INFO is coming soon, covering the two-phase execution model, how they scale, and how to use them directly in ES|QL queries beyond the Prometheus API.

How the metadata endpoints use them

Each metadata endpoint constructs an ES|QL plan with one of these commands at its core.

/api/v1/labels and /api/v1/series use TS_INFO, since they need per-time-series detail (which labels exist, which dimension values identify each series). /api/v1/metadata and /api/v1/label/__name__/values use METRICS_INFO, since they only need per-metric information (metric names, types, units).

/api/v1/label/{name}/values for regular labels (anything other than __name__) does not use either command. Regular labels like job or instance are actual dimension fields in the index, so the endpoint can query them directly with a group-by aggregation. When match[] selectors are provided, they are translated into a WHERE clause that filters the time series before the aggregation runs.

The __name__ label needs a different strategy because it is not always present as a dimension field. Prometheus Remote Write does store labels.__name__, but metrics ingested through other paths (OpenTelemetry, the bulk API) do not have it. The metric name is encoded in the field name itself (e.g., metrics.http_requests_total). You could look at the index mappings to enumerate field names, but mappings alone do not tell you which metric has which dimensions, and they cannot be filtered by label values from a match[] selector. METRICS_INFO can do both: it enumerates metric names across indices while respecting upstream WHERE filters.

In all cases, the API layer handles the translation back to Prometheus conventions: stripping the labels. and metrics. storage prefixes and synthesizing __name__ for non-Prometheus metrics that lack it.

In conclusion

The result: any Prometheus-compatible client can query and explore Elasticsearch metrics through endpoints it already understands. Remote Write metrics, OpenTelemetry metrics, and metrics indexed through other paths all show up through the same API, backed by the same TSDS indices.

All the Prometheus APIs mentioned here are available as tech preview in Elasticsearch Serverless today. For self-managed clusters and Elastic Cloud Hosted deployments, available as tech preview in Elasticsearch 9.4, with the exception of GET /_prometheus/api/v1/metadata. To experiment locally, use start-local.

This post introduces two personalization mechanisms that extend the governed control plane without changing its architecture. Both mechanisms stack multiplicatively with the governance layer from Parts 1 through 5: Policies still fire, constraints are still enforced, conflicts are still resolved, and personalization signals are composed into the same governed query, ensuring that the results Elasticsearch returns are already personalized.

The first mechanism boosts products the individual shopper has purchased before. The second activates cohort-specific policies based on the shopper's profile. Together, they demonstrate that personalization is not a separate system bolted alongside search or applied as post-retrieval processing; it’s a natural extension of the policy-driven control plane.

For a deep dive into the mathematics of the personalization techniques used in this post, see Personalizing search in Elasticsearch without ML post-processing and Cohort-aware ranking in Elasticsearch.

To see a live demonstration of how purchase history can be used to boost search results for returning customers, watch the video: Explainable Personalization: Boosting Search with Purchase History.

Individual purchase history boosting

The simplest form of personalization is also one of the most effective: If a shopper has bought a product before, boost it when they search for something related. A shopper who regularly buys a particular brand of chocolate chip cookies should see those cookies ranked higher when they search for "cookies", not because a model predicted a preference, but because there’s direct behavioral evidence.

How it works

When a search request includes a user identifier, as would be the case for a user that has an open session, the control plane runs two Elasticsearch queries in parallel using a thread pool:

1. The percolator query against the policy index (the same governance lookup described in Parts 3 and 4).
2. A purchase history query against a user_purchases index, filtered to the specific user by term(user_id) and then matching the current search string against that user's product titles.

These run concurrently (neither waits for the other), so the personalization lookup adds no meaningful latency to the governance pipeline.

The purchase history query uses Elasticsearch's text analysis (stemming, tokenization) when matching the current search string against stored product titles. This means a search for "cookies" will match a past purchase of "brownie cookies" through standard text analysis, without requiring exact string matching.

Computing boost weights

Not all past purchases deserve the same boost. The weight accounts for two intuitive factors: how often the shopper bought the product, and how recently. A product purchased 15 times last week is a much stronger signal than a product purchased once six months ago. The weighting uses logarithmic scaling on frequency (so a single heavily purchased item doesn't overwhelm everything else) and exponential decay on recency (so older purchases fade naturally over time).

For the mathematical details of the boost formula, see Personalizing search in Elasticsearch without ML post-processing.

How it becomes a query

The purchase history boosts are composed into the query as the outermost scoring layer, wrapping the governance policy filters and boosts from Parts 3 and 4 and any business-signal boosts, such as margin and popularity (which we’ll explore in Part 7). This means a product that’s removed by a governance policy won’t reappear because of a purchase history boost. Governance controls the result set; personalization adjusts ordering within it. Products without any purchase history are not penalized. Their governed ranking is preserved, though products with relevant purchase history will rank above them, all else being equal.

Why query Elasticsearch on every search?

The purchase history is queried from Elasticsearch on every search rather than cached in the application layer. This is a deliberate design choice. Because the query matches the current search string against product titles using Elasticsearch's text analysis pipeline, the system benefits from the same stemming, tokenization, and language handling that powers the product search itself. A cached in-memory lookup would require reimplementing that analysis or accepting cruder matching.

To see why this ordering matters, consider a shopper who previously purchased orange juice and now searches for "oranges”. The purchase history query matches "orange juice" against the search term "oranges" through text analysis and computes a boost for that product. But the governance layer has already constrained "oranges" to the produce category, filtering out orange juice entirely. The purchase history boost for orange juice is present in the query, but it has no effect because there’s no matching document in the governed result set for it to act on. The shopper sees fresh oranges, ranked by relevance and personalization. The governance guardrail holds.

The performance cost is minimal: The purchase history index is small (a user's purchase history is typically dozens to hundreds of documents, not millions), and the query runs in parallel with the percolator lookup, so it doesn't extend the critical path.

Example query for “spring water” without user history

If a non-logged-in user or a user that has never purchased “spring water” searches, they might see results similar to the following:

Example user purchase history

On the other hand, a user called Carol has a shopping history that contains the following products:

Example search for “spring water” with the above purchase history

If Carol searches for “spring water”, she’ll see personalized results that reflect what she has purchased in the past. Looking at the purchase history above, she purchased “Carbonated Spring Water” (the green bottle) above, about 40 times, and most recently two days ago. If she searches for “spring water”, then that product is boosted up, as we know that she likes it. Notice that in the non-personalized results, the Rubicon spring water was the first hit instead.

Cohort-aware policy activation

Individual purchase history works well for returning customers with established behavior. But many shoppers are new, anonymous, or browsing outside their usual patterns. For these shoppers, cohort membership provides a different kind of personalization, one based on who the shopper is, not what they've done.

A vegan shopper searching for "chocolate" should see vegan chocolate ranked higher. A halal-observant shopper searching for "snacks" should see halal-certified options prominently. A health-conscious shopper searching for "yogurt" should see probiotic options boosted.

Cohorts as policies, not product tags

Products already carry their normal attributes, including fields like dietary_restrictions: ["vegan"] or dietary_restrictions: ["halal"]. The question is where the logic lives that connects a shopper's cohort to those product attributes.

The naive approach would be to hard-code that mapping in the application layer or in the search template: If the user is vegan, add a boost on dietary_restrictions: "vegan". But this is the same application-layer spaghetti described in Part 1, and it creates the same operational friction: Adding a new cohort or changing what a cohort means requires a code change.

The governed control plane keeps the cohort logic in the policy engine instead. A cohort policy bridges two things: a shopper's cohort membership (for example, “vegan”) and a product attribute (for example, dietary_restrictions: “vegan”). The policy defines the connection: When a shopper in the vegan cohort searches, boost products where dietary_restrictions includes “vegan”.

Because cohort logic lives in the policy engine rather than application code, this means:

• Adding a new cohort can be done by creating a new policy; no product reindexing required.
• Cohort policies use the full rule engine: They can add filters, apply soft boosts, expand synonyms, change retrieval strategy, or any other action a policy can take.
• Cohort behavior is managed through the same admin UI as all other policies: A merchandiser can create, test, and promote cohort policies through the Author → Test → Promote workflow described in Part 2.

Example vegan cohort policy

A merchandiser creates a cohort policy with the following characteristics:

• Cohorts: ["vegan"].
• Match criteria: Matches any query (or a specific product category).

Action: Soft boost on dietary_restrictions: "vegan" with a boost weight of 2.

How cohort activation works

Each policy document has a cohorts field. Universal policies that apply to all shoppers regardless of cohort can leave this field blank, and these will internally be assigned a value of "_all" by the control plane. Cohort-specific policies store their target cohort names, such as ["vegan", "kosher", “sweet_tooth”].

When a search request includes a user profile, the control plane constructs a simple terms filter for the percolator query:

{ "terms": { "cohorts": ["_all", "vegan", "health_conscious"] } }

This single filter includes all universal policies plus the user's cohort-specific policies. The _all sentinel makes this a clean inclusion filter: No must_not or exists queries are needed to handle the case where a policy has no cohort restriction.

The percolator then evaluates policy matches as usual. The only difference is that the candidate policy set has been narrowed to those relevant to this shopper's cohorts. Everything downstream (cascading transformations, per-field conflict resolution, consumed phrase tracking) operates identically to the non-personalized flow described in Parts 3 and 4.

Non-vegan (standard) user results when searching for “chocolate”

When a non-vegan user searches for chocolate, there’s no vegan cohort boost applied to their results. They would often see non-vegan chocolates in the top hits, as follows:

Vegan cohort policy results when searching for “chocolate”

When a vegan-cohort shopper searches for "chocolate", this policy is included in the percolator candidate set. It matches, and the control plane applies a soft boost to vegan-certified chocolates. The boost is multiplicative: Vegan chocolates rank higher, but non-vegan chocolates are not fully excluded because the above filter is defined as a soft boost, which we described in detail in Part 3 of this series.

However, if the shopper explicitly searches for "Hershey milk chocolate", the vegan boost still applies but may be outweighed by the stronger text relevance of Hershey milk chocolate products.

A shopper outside the vegan cohort searching for the same query never sees the “vegan cohort” policy; it’s not in their candidate set. The governance layer is identical; only the active policy set differs.

Cohorts with purchase history

A vegan shopper with extensive purchase history gets vegan-cohort-specific policy activation as well as purchase history boosts. For new or anonymous shoppers, implied cohort membership alone provides meaningful personalization without requiring any behavioral data (for example, perhaps an anonymous user has only searched for vegan products, and so we classify them as a member of the vegan cohort). A shopper who self-identifies as halal-observant during account creation immediately receives halal-tailored results on their first search.

How personalization layers compose

The nesting order of function_score layers matters. From innermost to outermost:

1. Base query: The keyword or semantic match with named queries (fulltext_match, title_phrase_match).
2. Governance policy layer: Hard filters as bool.filter clauses, soft boosts as function_score functions (Parts 3 and 4).
3. Business-signal boosts: Margin and popularity boosting (which we’ll explore in Part 7).
4. Purchase history boosts: The outermost function_score layer.

This ordering ensures that governance controls the result set (what appears), business signals adjust ranking within that set (what appears first from the retailer's perspective), and purchase history adjusts ranking further based on individual behavior (what appears first from the shopper's perspective). Each layer wraps the previous one multiplicatively, so the effects compound rather than conflict.

What this means operationally

Personalization through the governed control plane preserves every operational property described in Parts 1 and 2:

• Zero-deploy changes. Cohort policies are created, tested, and promoted through the admin UI. Adding a new dietary cohort or adjusting a boost weight requires no code changes and no engineering involvement.
• Auditability. Every cohort policy is a discrete, versioned document. When a merchandiser asks, "Why are vegan products ranking higher for this user?", the answer is a specific policy with a specific priority, visible in the debug panel alongside every other policy that fired for that query.
• Conflict resolution. Cohort policies participate in the same per-field conflict resolution described in Part 3. If a cohort policy's category boost conflicts with a campaign policy's category override, the conflict is resolved deterministically by the same priority and strategy framework, no special handling needed.
• Measurability. Because cohort policies are discrete and individually toggleable, their impact on conversion, click-through, and add-to-cart rates can be measured independently, just like any other policy in the system.

What's next in this series

The next post explores another dimension of the governed control plane: how margin and popularity boosting can be tuned per query through policies, turning economic optimization into a governance decision rather than a static configuration.

See Part 7: Query-governed economic optimization: Per-query margin and popularity boosting

Put governed ecommerce search into practice

The personalization patterns described in this post (individual purchase history boosting and cohort-aware policy activation) were designed and built by Elastic Services Engineering as part of our repeatable ecommerce search accelerator. Both mechanisms integrate with the governed control plane architecture described throughout this series. Contact Elastic Professional Services.

Join the discussion

Have questions about search governance, retrieval strategies, or ecommerce search architecture? Join the broader Elastic community conversation.

You can now create high-performance semantic embeddings for text, images, videos, and audio recordings, spanning nearly 100 languages, and use them for classification, clustering, semantic similarity measurement, and indexing for retrieval. If your data lives in PDFs, recordings, and video alongside text, you no longer need separate pipelines for each.

The jina-embeddings-v5-omni family is the most compact embedding model currently on the market with support for pictures, speech, print, and video. It offers:

• jina-embeddings-v5-text’’s frontier-class text embeddings for retrieval, analysis, and AI agent applications.
• Best-in-size-class embeddings for visual semantic similarity, visual understanding, and image retrieval. jina-embeddings-v5-omni-small has the best performance on image benchmarks of any model in the 1 billion (10⁹) parameters and is superior to our own previous jina-clip-v2. Only a few models with three to 30 times as many parameters can beat it.
• State-of-the-art embeddings for multilingual visual understanding and retrieval, beating models as much as 20 times larger.
• Best-in-size-class audio embeddings, with only models that have double or more the number of parameters performing better on standard benchmarks.
• Support for video, especially for locating objects and events in footage.

This has applications in all areas of information retrieval, document processing, and data analysis. jina-embeddings-v5-omni opens up access to information locked up in different media silos and makes it accessible for retrieval, analysis, and use by AI agents. Information in audio and video recordings, PDF, printed page scans, and infographics are on an equal footing with digitized texts in your data ecosystem.

Just like jina-embeddings-v5-text, these models come in two sizes: small and nano. Both models extend their corresponding text equivalent with additional modules supporting audio and visual input. Users can select modules at load time. In addition, task-specific extensions for semantic similarity, classification, clustering, and information retrieval are implemented as compact low-rank adapters (LoRAs) and are all loaded, so users can select them at inference time.

Both models are very compact. jina-embeddings-v5-omni-small can run on conventional GPU-equipped servers, and jina-embeddings-v5-omni-nano is small enough to run on commodity hardware. This represents a large potential savings in compute costs and makes possible licensed local installation and edge processing, reducing latency and increasing your control of your own data.

The v5-omni suite uses innovative model design and machine learning techniques to compose new embedding models from previously trained ones without having to retrain them. We use encoders from pretrained, language-aligned, embedding models for audio and video media as input preprocessors for our existing jina-embeddings-v5-text model suite. The resulting models generate embeddings for images and sound recordings that are semantically compatible with the embeddings it generates for texts.

The v5-omni models produce text embeddings that are identical to jina-embeddings-v5-text (that is, jina-embeddings-v5-omni-small with jina-embeddings-v5-text-small; and jina-embeddings-v5-omni-nano with jina-embeddings-v5-text-nano) so you can extend existing text retrieval repositories to multimedia applications without rebuilding your indices.

The integrated encoders are all derived from open-weight sources. For images and video, we’ve used encoders from Qwen3.5 models:

• For jina-embeddings-v5-omni-nano, the fine-tuned SigLIP2 Base encoder from Qwen3.5-0.8B.
• For jina-embeddings-v5-omni-small, the fine-tuned SigLIP2 So400m encoder from Qwen3.5-2B.
• For audio support, we’ve added the encoder from Whisper-large-v3, extracted from Qwen2.5-Omni-7B, to both the small and nano versions.

We’ve connected these media-specific encoders to the text-processing backbone with trained cross-modal projectors. These projectors translate their native outputs to input embeddings compatible with jina-embeddings-v5-text. The only newly trained parts of the jina-embeddings-v5-omni models are the weights in those projectors.

This architecture means we only need to train the cross-model projectors, roughly 5.5 million parameters for jina-embeddings-v5-omni-small and under 3.5 million for jina-embeddings-v5-omni-nano, for each of the four LoRA adapters. This approach minimizes the additional training needed to connect different embedding models, leveraging the specialized training of each to produce an extremely compact, high-performance, modular embedding suite.

Selected model properties

Input/output

* See Using jina-embeddings-v5-omni below for more on how non-text media is tokenized.

Size

* See Using jina-embeddings-v5-omni below for more on how non-text media is tokenized.

Task-specific training

The jina-embeddings-v5-omni family supports the same task-specific LoRA adapters as jina-embeddings-v5-text:

Output embeddings depend on the selected task category. For example, you shouldn’t use retrieval-oriented embeddings for clustering or semantic similarity embeddings for classification.

Multimedia, multimodal, multilingual, multifunctional

To show what jina-embeddings-v5-omni can do, let’s take the famous opening passages of two novels and measure their semantic similarity:

A Tale of Two Cities (Charles Dickens)

It was the best of times, it was the worst of times, it was the
age of wisdom, it was the age of foolishness, 
it was the epoch of belief, it was the epoch of incredulity,
it was the season of Light, it was the season of Darkness,
it was the spring of hope, it was the winter of despair,
we had everything before us, we had nothing before us,
we were all going direct to Heaven, we were all going
direct the other way—in short, the period was so far like
the present period, that some of its noisiest authorities
insisted on its being received, for good or for evil, in 
the superlative degree of comparison only.

Pride and Prejudice (Jane Austen)

It is a truth universally acknowledged, that a 
single man in possession of a good fortune must
be in want of a wife. However little known the
feelings or views of such a man may be on his first
entering a neighbourhood, this truth is so well
fixed in the minds of the surrounding families,
that he is considered as the rightful property of
some one or other of their daughters.

Using jina-embeddings-v5-omni-small, with its semantic similarity adapter, these texts have a similarity of 0.5329.

That number doesn’t mean much without something to compare it with, so let’s compare these two texts to their French translations using the same model and adapter:

Semantic similarity scores for texts across languages

The two texts show much greater similarity to their translations than to other texts in the same language or a different one. This reflects the very high performance multilingual semantic embeddings of jina-embeddings-v5-text-small, included unchanged in jina-embeddings-v5-omni-small.

Adding multimedia support to jina-embeddings-v5-omni means we can extend this experiment to whole other types of data. For example, we fetched scans of the first pages of both novels from old print editions:

Figure 2: A Tale of Two Cities, undated 19th-century edition, and Pride and Prejudice, 1903 Macmillan edition.

Let’s compare both texts to the scans, again using the semantic similarity adapter:

Semantic similarity scores between texts and images

You see that semantic similarity scores strongly favor texts that match image contents.

We can also compare the texts to a screenshot of a social media post and a meme that reference those texts, using the same setup:

Figure 3: An Elon Musk tweet referencing A Tale of Two Cities, and a meme referencing the famous opening of Pride and Prejudice.

Semantic similarity scores between texts and images

We can do the same for speech. We obtained recordings of readings of both texts, in English and in French:

• A Tale of Two Cities (English audio from Librivox).
• A Tale of Two Cities (French audio generated by OmniVoice AI).
• Pride and Prejudice (English audio from Librivox).
• Pride and Prejudice (French audio generated by OmniVoice AI).

Semantic similarity scores between texts and audio across languages

This multilingual and multimedia ability extends to information retrieval.

The retrieval adapters for the jina-embeddings-v5-omni models implement asymmetric retrieval. This means they embed queries differently from the way they embed retrieval target documents, so cross-modal queries are always in some direction, with queries in one media and documents in another, giving different scores from when they’re reversed.

The tables below show the retrieval scores for text, audio, and page scan images for A Tale of Two Cities and Pride and Prejudice, when the text from A Tale of Two Cities (in English) is encoded as the query:

Text to text

Text to image

Text to audio

Users can also run the query the other way around, doing audio-to-text and image-to-text retrieval.

Below are the scores using the English audio of A Tale of Two Cities as a query and various texts as documents:

Image to text

And the scores using a scan of page one of A Tale of Two Cities (in English) as a query:

Audio to text

Video search

The jina-embeddings-v5-omni‘s capabilities for video indexing and search bring new capabilities to Elasticsearch databases, but it’s subject to many of the same warnings that apply to texts. Generating a single embedding for a long film is like embedding a very long novel: Detailed information will be swamped, and the resulting embedding will be a good match for many very spurious queries.

If you embed the whole text of Lord of the Rings (~500,000 words), it’s likely to be a good match for most queries, no matter what you’re looking for. Similarly, if you index a two-hour Hollywood film, you’ll get a lot of spurious matching and totally missed details. jina-embeddings-v5-omni is optimal with short clips.

For this example, we downloaded the trailer to the 1961 film Breakfast At Tiffany’s, which is just 158 seconds long and in the public domain. You can see the trailer on the Internet Archive.

Figure 4: The theatrical poster for Breakfast at Tiffany’s.

We used PySceneDetect to split the trailer into 28 individual scenes, with lengths varying from 1.877 seconds (45 frames) to 18.393 seconds (441 frames). Scene detection is imperfect, but it provides an adequate mechanism for splitting video into bite-sized chucks for retrieval. Then we generated document embeddings for each of the 28 segments, using jina-embeddings-v5-omni-small, so we could test the effectiveness of text queries at finding specific elements in the video.

For example, querying for “cat” returned the following clips as the top three results. The one scene with a cat in it is at the top, with a score of 0.1634:

Watch clip one.

The next highest match, with a score of 0.1237, is much lower:

Watch clip 2.

You can also query for actions. If you query with the string “kiss”, the top four matches all contain kisses:

Watch clip 3. Its score is 0.2864.

Scores: For the second match (0.2494), third match (0.2099), and fourth match (0.2068), respectively

And you can search for text displayed in videos, like for “Buddy Ebsen”, which only appears once. jina-embeddings-v5-omni-small readily identifies it as the best match with a score of 0.3885, considerably higher than the next best match:

Buddy Ebsen clip.

Visual document retrieval

Jina AI multimodal embedding models are top performers in visual document processing and state-of-the-art in multilingual visual document processing. This means handling image data that contains text, figures, and structured information. Important data is often in the form of print scans, PDF files, diagrams, technical drawings, screenshots, pictures, infographics, and the like. These kinds of images are often mechanically composed or computer generated. They can’t usually be reduced to text without loss of meaning and are poorly suited to computational vision models designed for photography of natural scenes.

jina-embeddings-v5-omni’s embeddings encompass information about the things in the image, the text printed on them, and the relationships between the two. Visual document retrieval makes it possible to index rich images that contain both things and relevant text and to do so across languages.

As an example, let’s use four product images from various ecommerce websites:

Now, let’s see how well jina-embeddings-v5-omni-small scores these four images for the query “ramen noodles”:

It readily finds the Japanese match.

Now, let’s try a query for “マカロニチーズ” (Japanese for macaroni and cheese):

It finds the correct match with the same ease as an English query.

jina-embeddings-v5-omni also excels at interpreting information-rich images, like charts. To see this in action, look at these two bar charts:

Two charts, Chart 1 to the left, about the global burden of disease, and Chart 2 to the right, about the lifespans of dog breeds.

Let’s see how well they match two potential text questions, each relevant to one but not both charts, using jina-embeddings-v5-omni-small for retrieval:

You can also reverse the search, using images as queries to find texts. The table below shows target documents extracted from the abstracts of topically related scientific papers and their retrieval scores, using the chart images as queries:

Features

Truncatable embeddings

We trained the backbone jina-embeddings-v5-text models underpinning jina-embeddings-v5-omni with Matryoshka Representation Learning, so you can truncate both text and multimedia embeddings from these models.

By default, jina-embeddings-v5-omni-small generates embeddings with 1024 dimensions, taking 2KB to store at 16-bit precision. jina-embeddings-v5-omni-nano’s embeddings have 768 dimensions, taking up about 1.5KB. You can reduce the size of these embeddings down to 32 dimensions (64 bytes) at some cost to accuracy but with a large gain in processing speed and reduced resource costs. In general, reducing embedding sizes by half lowers accuracy by about 2%, down to 128 dimensions, below which accuracy falls much faster.

Truncatable embeddings allow users to decide the optimal trade-off between accuracy, speed, and cost, given their own use cases.

Quantization

The jina-embeddings-v5-omni family also inherits robust performance under quantization from its jina-embeddings-v5-text backbone. This further increases speed and reduces computing and storage costs by storing less precise numbers. We’ve trained them to work with Elasticsearch’s Better Binary Quantization (BBQ) to provide near-identical performance to unquantized embeddings. On the Massive Text Embedding Benchmark (MTEB) retrieval benchmark suite, binarization reduces performance by less than 3% compared to full 16-bit values, while saving 93% of the space and dramatically increasing processing and retrieval speeds.

Cross-language performance

jina-embeddings-v5-text’s extensive multilingual training carries over to jina-embeddings-v5-omni, with nearly 100 languages in jina-embeddings-v5-text-small’s pretraining and 15 major global languages in jina-embeddings-v5-text-nano’s. For audio media, the Whisper-large-v3 model has roughly 100 languages in its training, and the Qwen-modified SigLip2 vision models integrated in jina-embeddings-v5-omni-small and -nano were trained with data from 201 distinct languages and dialects.

Benchmark performance

Text

jina-embeddings-v5-omni models are identical to jina-embeddings-v5-text models when used just for text. They’re the top performers on the MMTEB benchmark suite in their respective size categories for semantic text embeddings.

Figure 5: jina-embeddings-v5-omni’s size and performance on text benchmarks, compared to competing models. The cited size is without loading extensions for other media.

Visual semantic similarity

On standard visual semantic similarity benchmarks, jina-embeddings-v5-omni delivers the best scores of any model near its size. jina-embeddings-v5-omni models show by far the best performance for public open-weight models of comparable size. jina-embeddings-v5-omni-small is only beaten by a model three times its size on visual semantic similarity tasks, and jina-embeddings-v5-omni-nano is beaten only by jina-embeddings-v5-omni-small and by models 10 to 25 times larger.

Figure 6: Visual semantic similarity benchmark mean scores for jina-embeddings-v5-omni-small, jina-embeddings-v5-omni-nano, and comparable models, as well as their sizes including vision extensions.

Visual document retrieval

jina-embeddings-v5-omni-small is competitive with three and seven billion parameter models while remaining under one billion parameters. jina-embeddings-v5-omni-nano similarly stands out for its size, beating models ten to sixty times larger.

Figure 7: Mean ViDoRe visual document retrieval scores on six benchmarks: DocVQA, InfoVQA, ShiftProj, SynAI, Tabfquad, and TatDQA.

Audio retrieval

On the standard MAEB (Massive Audio Embedding Benchmark) audio retrieval benchmarks, both jina-embeddings-v5-omni-small and jina-embeddings-v5-omni-nano rank among the top performers. Only very large models – more than three times the size of jina-embeddings-v5-omni-small – beat its score.

Figure 8: Mean score for various models on the MAEB audio retrieval benchmarks.

Although LAION’s larger_clap_general model does improve on jina-embeddings-v5-omni-nano ‘s score while having fewer parameters, it’s an audio-only model with none of the additional multimodal features of the v5-omni suite.

Video

On video, jina-embeddings-v5-omni-small excels at finding the place in a video that matches a text query. The Charades-STA and MomentSeeker tests are the standard benchmarks for this task, and you can see from the charts below that jina-embeddings-v5-omni-small is the top scorer among comparable open-weight models despite a far smaller size.

Figure 9: Charades-STA scores for various models, along with their sizes.

Figure 10: MomentSeeker scores for various models, along with their sizes.

We also compared jina-embeddings-v5-omni-small to ByteDance's Seed 1.6, a closed-weight model with undisclosed parameter count. Our model beats Seed 1.6 by a large margin on the Charades-STA benchmark and nearly equals it on MomentSeeker.

Strengths and limitations

jina-embeddings-v5-omni models expand users’ ability to index, search, and analyze digitized information in a number of ways, particularly:

• Multilingual speech retrieval from text queries.
• PDF, scans, and visual document search.
• Video temporal grounding, that is, identifying parts of videos that match natural language text descriptions.
• Audio genre classification, including musical genres.
• Image classification based on scene information and object identification.

Performance is more limited in some other areas. It may be possible to use jina-embeddings-v5-omni to do these tasks, but we haven’t trained for them and results may be poor.

We’re actively working at improving our technology in these areas:

• Finding specific videos from natural language descriptions.
• Image-to-image semantic similarity and retrieval.
• Intent classification in speech, like recognizing verbal commands.
• Processing mixed media inputs, that is, images and accompanying text, or audio, images, and texts combined.

Using jina-embeddings-v5-omni

This model suite supports input via three entry points: text, audio, and images and video together. jina-embeddings-v5-omni runs within a framework that converts a broad array of standard formats and does other preprocessing.

We process images using the same NaFlex approach provided in the initial SigLip2 release: If the input is smaller than 262,144 pixels (equivalent to 512x512), it’s upscaled until it's larger than that minimum; and if larger than 3,072,000 pixels, then it’s downscaled until it’s smaller than that maximum. The conversion process ensures that both the height and width of the image is a multiple of 14 pixels, with as little aspect ratio distortion as possible to accomplish that goal. The result is split into patches of 28x28 pixels, so the total number of patches is however many 28x28 squares are needed to cover the image. Each patch is treated like a single token at inference time, and each image input is accompanied by special start and end tokens to delimit a single image.

The jina-embeddings-v5-omni models modify video resolution in the same way that images are modified (see above), and we extract up to 32 frames from the video. If the video has more than 32 frames (which is likely, since standard formats are usually at least 24 frames per second), we evenly space the frames we extract. Then, for every two frames, the video preprocessor generates one set of tokens equal to the number of 28x28 squares needed to cover the video.

Figure 11: jina-embeddings-v5-omni extracts 32 equally spaced frames from the video. If you have a long video, this means a lot will be lost.

For more details on video preprocessing, see the SigLip2 technical documentation.

Audio tokenization follows the approach built into Qwen-2.5-Omni: Sound files are cut into 30-second segments; if longer than 30 seconds, resampled to 16kHz, transformed into a 128-channel mel-spectrogram. Each 40ms is treated as a single token, so every 30-second segment is handled as 750 tokens, one token per 40ms of audio, plus special start and end tokens to delimit a single sample.

For more details on audio preprocessing, see the Qwen-2.5-Omni Technical Report.

Availability

Both jina-embeddings-v5-omni-small and jina-embeddings-v5-omni-nano are available on the Elastic Inference Service (EIS), via the Jina API, and for local installation via download (small and nano). Model weights are distributed freely to try out on a non-commercial license. Contact Elastic sales for commercial use.

Getting started

To use jina-embeddings-v5-omni for text, you can integrate using the semantic_text field just like with jina-embeddings-v5-text. Just set the inference_id to .jina-embeddings-v5-omni-small or .jina-embeddings-v5-omni-nano. See the Reference Guide for instructions.

To embed other media with jina-embeddings-v5-omni, you need to use the inference API. For example:

POST _inference/embedding/.jina-embeddings-v5-omni-small
{
  "input": [
    {
      "content": { 
        "type": "image", 
        "format": "base64", 
        "value": "data:image/jpeg;base64,..." 
      } 
    }, 
    { 
      "content": { 
        "type": "text", 
        "value": "Some text to create an embedding" 
      } 
    } 
  ] 
}

For jina-embeddings-v5-omni-nano, change the POST URI to _inference/embedding/.jina-embeddings-v5-omni-nano.

To encode documents in other media, or generate embeddings for classification or clustering, you need to create an inference endpoint with the jinaai service.

For queries, use the query builder as in the example below. Replace the inference_id value with .jina-embeddings-v5-omni-nano to use the nano model instead of small.

POST my-index/_search
{
  "knn": {
    "field": "dense-vector-field",
    "k": 10,
    "num_candidates": 100,
    "query_vector_builder": {
      "embedding": {
        "inference_id": ".jina-embeddings-v5-omni-small",
        "input": {
          "type": "image",
          "format": "base64",
          "value": "data:image/jpeg;base64,..."
        }
      }
    }
  }
}

See the query builder documentation for more information.

To use BBQ with jina-embeddings-v5-omni, follow the instructions for BBQ indexing.

More information

For more information about jina-embeddings-v5-omni, see the model’s technical report and page on the Jina AI website. The jina-embeddings-v5-omni collection page on Hugging Face also contains technical information and instructions for downloading and running these models locally. The jina-embeddings-v5-omni models can be downloaded under a CC-BY-NC-4.0 license, so you’re free to try them out, but for commercial use, please contact Elastic sales.

Just send your OTel metrics to the Elasticsearch OTLP/HTTP endpoint and they're stored using the new exponential_histogram type and queryable immediately. Already have historical data stored in the classic histogram type? A simple ::exponential_histogram cast in your ES|QL queries handles the migration transparently. Already using downsampling? Both histogram field types are now fully supported.

Histogram metrics

When dealing with metrics (in OpenTelemetry or Prometheus, for instance), counters and gauges are the most common metric types. Gauges allow you to monitor values that rise or fall (e.g., CPU utilization). Counters allow you to, well, count things, such as the total number of HTTP requests your service is handling. Counters normally just increase in value, with a few exceptions when they reset, like when a server reboots.

In the case of counters, you can additionally collect a counter measuring the total sum of your HTTP response times, which allows you to derive the average response time by dividing that sum by the total number of requests. However, average response times provide limited insights into the collected data and the system behavior. The best insights are gained by analyzing the collected metric distribution, e.g., through median and percentile calculations. This is where counters fall short.

In the past, workarounds have been applied: For example, classic Prometheus-style histograms attempt to capture the distribution using a set of counters. By defining fixed buckets (e.g., one for response times in the range [0s, 1s), one for [1s, 4s), and so on) and associating a counter with each, we can at least estimate percentiles broadly. However, the key problem here is that we have to know the distribution of our data up front to properly define these buckets.

To that end, the OpenTelemetry community has come up with a better solution: exponential histograms. Exponential histograms assign collected values to buckets, just like classic Prometheus-style histograms. The key differentiator is that these buckets vary dynamically based on the collected values. The name "exponential" comes from the fact that the bucket sizes increase exponentially: we use small buckets for small values and wider buckets for larger values. You can find an excellent introduction in the OpenTelemetry exponential histograms introduction.

Note that in addition to classic histograms, Prometheus also added native histograms, which directly map to OTel exponential histograms. Native histograms have their own PromQL syntax. We are actively working on adding support for that syntax to the Elasticsearch PromQL implementation, so that you can directly query exponential histograms using PromQL.

Demo setup

Let's start by collecting some histogram metrics to show how they can be stored and analyzed in Elasticsearch using ES|QL.

We'll focus on a Java JVM metric: garbage collection durations. OpenTelemetry defines the jvm.gc.duration, which is a histogram-typed metric. The OpenTelemetry Java agent natively supports collecting this metric.

We'll spin up a JVM running a Renaissance benchmark to put it under stress. We'll start that JVM with the vanilla OpenTelemetry Java agent attached and have it send the metrics directly to Elasticsearch.

You can find the ready-to-run Docker-compose file here. You'll just need to insert your Elasticsearch OTLP/HTTP endpoint and API key in the docker-compose.yml:

OTEL_EXPORTER_OTLP_ENDPOINT: https:///_otlp
OTEL_EXPORTER_OTLP_HEADERS: "Authorization=ApiKey "

Note that you don't have to use this demo setup. We even encourage you to try it with your own application. Here are the other important OpenTelemetry agent settings the demo already includes, which you should include too if you're bringing your own app:

OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE: delta
OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION: BASE2_EXPONENTIAL_BUCKET_HISTOGRAM
OTEL_INSTRUMENTATION_RUNTIME_TELEMETRY_ENABLED: "true"

Let's step through them:

• Temporality preference: OpenTelemetry supports both cumulative and delta-based histograms. Cumulative means that the histogram is only cleared after an application restart, while delta clears it after each export. At the time of writing, Elasticsearch only supports delta temporality for histograms. We are actively working on supporting cumulative histograms as well.
• Default Histogram Aggregation: By default, OpenTelemetry exports histograms in the Prometheus-style fixed bucket format. Since we want to reap the benefits of exponential histograms, we tell the agent to use them instead.
• Runtime Telemetry enabled: This tells the agent to actually collect the detailed JVM metrics, which include jvm.gc.duration.

Now we are ready to go! We'll let the application run in the background and switch over to Kibana to analyze the GC metric.

Querying with ES|QL

Now let's open up Kibana and navigate to "Discover". There we'll switch to ES|QL mode, and start querying the collected data:

TS metrics-* | STATS COUNT(jvm.gc.duration)

As a response, we now see the metric panel shown below. If you don't see any data, make sure to double-check the Kibana time range filter.

This number represents the total number of garbage collection operations that happened in our test application during the selected time frame.

Similarly, we can query the total time spent on those garbage collection operations:

TS metrics-* | STATS SUM(jvm.gc.duration)

So we have roughly 270k garbage collections, which in total took 713 seconds. Given these two numbers, we can now compute the average if we are still fluent in primary school-level math. Even if not, you can just let ES|QL do that for you:

TS metrics-* | STATS AVG(jvm.gc.duration)

Now we know that the average garbage collection operation took about 3 milliseconds. However, Java experts might know that there are different kinds of garbage collections happening, which can have significantly different pause times. Fortunately the OpenTelemetry metric comes with attributes, which allow us to slice the data accordingly:

TS metrics-* | STATS AVG(jvm.gc.duration) BY jvm.gc.action

As expected, major garbage collections take a lot more time per collection than minor ones, at least on average. So far, we have done nothing you couldn't also achieve by just using counters. Let's now use histograms to understand the actual distribution of the GC latency. We'll look at the data over time (by grouping using TBUCKET) and focus on the major garbage collections:

TS metrics-* 
| WHERE jvm.gc.action == "end of major GC"
| STATS MAX(jvm.gc.duration),
        PERCENTILE(jvm.gc.duration, 99),
        MEDIAN(jvm.gc.duration),
        MIN(jvm.gc.duration)
 BY TBUCKET(100)

The graph now shows us the minimum, maximum, median and 99th percentile for major garbage collections. Note that we aren't bound to only querying the median and the 99th percentile. We can query any percentile we'd like to see, as these are estimated at query time from the raw exponential histograms.

A note on backwards compatibility

So far, we have seen how you can use the new shiny toy in Elasticsearch and ES|QL: exponential histograms. However, since this has just reached general availability (GA) in the 9.4 release, what about your historical data?

Before exponential histograms were added, Elasticsearch was already capable of storing OpenTelemetry histograms in the histogram field type. To do so, we converted them to a different data structure supported by the histogram field type: T-Digest. T-Digest provides good accuracy for extreme percentiles (e.g., 99th percentile) at the cost of accuracy for percentiles in the middle of the distribution, such as the median. In contrast, exponential histograms provide a guaranteed upper bound on the relative error for every percentile. As conversions always introduce errors, we are happy to now have native support for exponential histograms, allowing you to collect and analyze your metrics end-to-end without unnecessary conversions.

But still, what should you do if you have historical data and still want to query it? Thanks to ES|QL union types, the answer is actually easy: You just have to add a ::exponential_histogram suffix to the histogram metrics in your queries:

TS metrics-* | STATS AVG(jvm.gc.duration::exponential_histogram)

When this query encounters histogram fields, it will attempt to convert them to exponential histograms. When operating on exponential_histogram fields, the ::exponential_histogram cast has no effect. Note that this also works with mixed data sets: if your backing indices use both types, the query will just do the right thing.

So if you are building queries or dashboards that you expect to run on pre-9.4 ingested data, we recommend that you simply add: ::exponential_histogram casts.

Wrapping up

Native support for OpenTelemetry exponential histograms in Elasticsearch gives you better metric fidelity and more flexible analysis in ES|QL. In this blog post, we have shown you how to easily ingest and analyze your histogram metrics with ES|QL using various aggregations and the impact exponential histograms have.

Exponential histograms are generally available in Elasticsearch basic starting with the 9.4.0 release. They will be available in Elastic Cloud Serverless a few weeks after the 9.4.0 release, once mOTLP (the managed observability OTLP intake) switches to use the Elasticsearch OTLP endpoint. We'll update this blog post and add a note on the Elastic Cloud Serverless release notes when that happens.

Place controls anywhere in the dashboard

Most dashboarding tools lock filters into a fixed bar at the top of the page. Kibana 9.4 breaks that constraint: controls are now regular panels you can drag anywhere to place them where you need them. Pin them to the top so they stay visible while you scroll, or place them inside a collapsible section where they automatically scope their filters to that section only — so a single dashboard can serve multiple use cases without one filter resetting all the panels.

Fewer clicks to a readable chart

Building a time series in Kibana Dashboards used to take 14 clicks. Now, it takes three. The new defaults give you a line chart with a bottom legend and cleaner x-axis labels — the way you'd configure them by hand. A right-hand legend still makes sense past ten series, but for fewer, the bottom list avoids the empty margin a side legend leaves behind. For top-value breakdowns, the default jumps from three series to nine — matching how many distinct colors the eye can parse at once — and the colors themselves are darker and more separated, so series stand out at a glance

Before (bar chart by default, legend to the right, redundant x-axis label):

Now (line chart by default, list legend at the bottom, more contrasted color palette for lines readability):

A calmer look for dense dashboards

Dense dashboards in most tools still look like a wall of boxes. We redesigned Kibana's dashboard visuals to feel flatter and more cohesive — more like a single page than a grid of disconnected panels. See some of these changes highlighted in the picture below compared to what the same dashboard looks like in 9.3.

Other improvements

Collapse filter pills

Another common filtering approach is using filter pills at the top of the dashboard. When you stack many pills, they used to consume vertical space; they now scroll automatically after a threshold, and you can collapse the row to reclaim the space.

Closing the gap for ES|QL panels

Elasticsearch Query Language (ES|QL) panels are closer to parity with the rest of dashboards: Drilldowns between dashboards work for ES|QL visualizations, and click-to-filter from the chart keeps improving. Upcoming releases aim to close remaining gaps, such as annotations and saving ES|QL visualizations to the library.

Panel size visibility

Before 9.4, it was hard to match panel height and width because there was no readout for panel size. Panel sizing is now shown while you resize, so you can align dimensions across panels for a more harmonious layout.

Easier section reordering

Collapsible sections picked up usability fixes, too: You can reorder sections while they’re expanded, and you can drag a section from its header without hunting for the small drag handle.

ES|QL support for Vega visualizations

Vega lets you build custom visualizations beyond what built-in chart types — radar charts, chord diagrams, or anything the grammar supports. The hardest part has always been the data query: writing verbose Elasticsearch Query DSL with nested aggregations and format paths just to feed data into the chart. Vega now accepts ES|QL as a data source, so you can get the data with a single readable query. Less time wiring up the data, more time on the visualization itself.

What’s next

This is just the start. Upcoming releases will keep pushing on smarter defaults and closing the gap between ES|QL and data-view visualizations so the two feel interchangeable.

Got a pain point or a feature request? Hit Submit feedback in the top menu — we're listening.

How to try it

If you use Elastic Cloud Serverless, you may already be on these changes. Otherwise, upgrade to 9.4 and then create a dashboard or open an existing one. Many updates apply automatically to new visualizations, while layout and panel options appear in edit mode. If you aren’t on Elastic Cloud yet, start a trial and explore the latest Kibana Dashboards there.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elasticsearch has supported storing metrics in time-series data streams (TSDS) since version 8.7. This offering mainly focused on storage gains as explained in an earlier blog post. Still, performance was not on par with specialized systems for storing and querying metrics, in terms of storage efficiency, indexing throughput and query latency.

In the past year, we revisited the storage layer, optimized ingestion for OTel metrics and extended the ES|QL compute engine with vectorized processing for time series data. These efforts led to substantial performance wins across the board, compared to earlier versions of TSDS:

1. Up to 6.6x improvement in storage efficiency, reaching 3.75 bytes per data point in OTel metrics
2. Up to 50% improvement in indexing throughput for OTel data
3. Up to 160x improvement in query latency, including blazing fast counter rate evaluation and window support in time series aggregations

Elasticsearch has thus become a leading columnar metrics engine, matching or exceeding the competition (like Prometheus, Mimir, and ClickHouse) in indexing throughput and exceeding it by up to 2.5x in storage efficiency and 30x in query performance. All while maintaining the ability to store logs and other data and fully use the rich querying capabilities of ES|QL (e.g. inline stats, lookup join) — which other PromQL-based systems lack. Elasticsearch can thus serve as a unified storage and query engine for all user data, with no compromises for metrics and observability applications.

How TSDS is organized

TSDS has the following properties that help improve the performance of time-series codecs and produce correct results when aggregating data points per time series:

• The metric name and the dimension names and values are used to calculate the _tsid, a unique identifier per time series.
• TSDS get sorted by [_tsid ascending, timestamp descending] order. Each time series is thus stored in sequence on disk, with newer data points appearing first. Since the _tsid is calculated over dimension values, the latter are also clustered on disk.
• Shard routing is based on _tsid, with each _tsid value appearing in one shard only.
• Backing indices are time-bound, with no overlap over time between them.

The rest of this post explains how we use these properties to improve storage, indexing, and query performance.

Storage optimizations

TSDS already achieved a very competitive storage footprint, reaching 0.9 bytes per data point, when it is possible to combine many metrics in a single doc, sharing the same dimension values. However, when most data points have a unique set of dimensions (which is typical for OTel or Prometheus metrics), docs end up containing a single data point. In this setup, storage required 25 bytes per data point, with dedicated metrics stores requiring less than 10 bytes per data point.

To further reduce the storage footprint, we applied a series of optimizations over the past year:

Replace inverted indices and BKD trees with doc value skippers

Elasticsearch creates inverted indices (for text values) or BKD trees (for numeric values) by default for all non-metric fields, i.e. for @timestamp and dimensions. These indices improve performance for queries including filters on these fields, but have significant impact to storage — effectively doubling the footprint for each field. More so, they are also processed during segment merging, increasing the cpu, memory and storage overhead and slowing down the system — especially in high ingest throughput scenarios, as is often the case with metrics.

Lucene has been extended with doc value skippers, a form of hierarchical sparse indices that store the minimum and maximum value of blocks of documents. Range queries can check these min and max values and ignore blocks that don't fall into the requested range. Skippers work particularly well on sorted fields. Since TSDS are sorted by [_tsid, timestamp desc], dimension values get also clustered on disk. It's therefore possible to replace indices on @timestamp and dimension fields with doc value skippers that amplify the columnar layout — each field stored in its own files, with no duplicate tracking of each doc for indexing purposes.

Doc value skippers have negligible storage overhead — replacing indices with them led to a reduction of 10 bytes out of the initial 25 bytes per data point in OTel. Moreover, they work very well in practice when queries include filters on time ranges or dimension values (including prefixes and regex) — there was no noticeable regression in query performance in our benchmarks when they replaced separate indices. Doc value skippers are enabled for TSDS by default since version 9.3.

Enable synthetic IDs

The _id metadata field was another big contributor to the storage footprint. TSDS has already been extended to trim the doc values once they were no longer needed for replication, but the inverted index was kept around to efficiently support the id-based APIs (Get, Delete, Update).

The ID value for TSDS is synthesized by combining the _tsid and @timestamp values that uniquely identify each data point. Since these fields are configured with doc value skippers, it's possible to replace the inverted index on _id with (a) retrieval of the _tsid and @timestamp value from the _id value, and (b) checks for matches using doc value skippers respectively. Care has to be taken to avoid expensive checks for duplicate IDs during metric ingestion, with segment-level bloom-filters keeping the overhead at bay.

Supporting synthetic IDs in metrics is a first for Elasticsearch. It led to a reduction of 5 bytes out of the initial 25 bytes per data point for OTel metrics, with no loss of functionality. Synthetic IDs are enabled for TSDS by default in version 9.4. We plan to extend their uses in logs and other applications after further evaluation.

Trim sequence numbers

Sequence numbers are used as part of replication, but also to provide strong consistency semantics on doc modification operations through Optimistic Concurrency Control (OCC). While such semantics are applicable to certain scenarios, they don't fit in metrics where concurrent updates are very rare, with no practical need for guarding against concurrent operations on data points with matching ids. We therefore decided to disable the use of sequence numbers in all APIs, along with OCC support, for TSDS, in version 9.4. This leads to a substantial storage reduction of 4 bytes out of the initial 25 bytes per data point for OTel data, as there's no inverted index and sequence numbers get trimmed once no longer needed for replication. Update and delete by query operations are still supported, albeit with weaker consistency semantics.

If OCC is still deemed important for a particular metrics application, the old behavior can be restored by setting index.disable_sequence_numbers: false in the index template of the involved TSDS.

Use large numeric codec blocks

TSDS already uses an advanced codec, as explained in an earlier article. The codec works very well in most cases, but has poor performance in case of repeated sequences of keywords and numbers, leading to an inflated storage footprint for dimensions containing IP and MAC addresses. We identified that the existing logic for identifying repeated sequences requires larger codec blocks to work well, especially as the sequence length increases. After experimentation, the numeric block size was increased from 128 to 512 elements in version 9.3, leading to a reduction of 2 bytes out of the initial 25 bytes per data point for an OTel dataset containing IP and MAC addresses as dimensions. We're also working on a more configurable codec layout that will allow more flexibility with block sizes and other parameters, based on field type and cardinality.

Indexing throughput

Elasticsearch has support for bulk ingestion of documents. This entrypoint has long been optimized for leniency, ensuring that all docs get accepted. This flexibility, however, incurs additional processing cost during indexing. Metric applications proved good candidates for using different approaches to reduce this overhead, as explained below.

Introduce OTLP protobuf entrypoint

OTel metrics and Prometheus have established protocols for metrics ingestion, using protocol buffers. In the past, a translation step was required to convert collected protobuf messages to bulk requests that Elasticsearch can consume.

Elasticsearch was recently extended with endpoints accepting messages from OTel metrics collectors and over Prometheus remote write. Parsing and processing these (binary) messages is cheaper, compared to json parsing, while hash operation over dimensions for _tsid calculations get reused and amortized across more data points within a single protobuf message. Furthermore, _tsids get evaluated once per doc in the coordinator nodes and propagated to data nodes for indexing, thus deduplicating an expensive step per indexed doc. These improvements led to up to a 20% speedup in indexing throughput for OTel metrics. The OTLP entrypoint was added in version 9.2 (tech preview) and reached GA in version 9.3. We've added similar entrypoints for Prometheus remote write in version 9.4 (tech preview) and are actively working to cover OTel Logs and Traces.

Reduce indexing CPU with doc value skippers

In addition to a substantial storage footprint, inverted indices require a lot of cpu to build and reconstruct during segment merging. The use of doc value skippers in their place helps also reduce cpu load at ingestion and thus improves indexing throughput by 10%, a welcome bonus on top of the aforementioned storage wins.

Synthetic recovery source

The original source of a document, as provided at index time, is never stored for metrics. Still, Elasticsearch needed to temporarily store it for replication purposes. That changed in version 9.1, where the source gets synthesized on demand for replication purposes. This is known as synthetic recovery source and reduces disk I/O by 50%, with a significant impact to metrics indexing performance. Check out this article for more details.

Query execution

Replacing inverted indices with doc value skippers leads to a pure columnar storage layout for TSDS, with metric and dimension fields stored as Lucene doc values, each field encoded and compressed in their own file. Combined with the introduction of the ES|QL compute engine that uses vectorized execution internally, it became possible to introduce a fully columnar storage and query processing engine for metrics in Elasticsearch. We pushed this idea to the extreme and implemented a columnar metrics processing engine that comfortably outperforms dedicated metrics engines and other columnar stores in query performance.

Time series integration in compute engine

Time series processing is largely based on applying aggregation functions per time series (or _tsid), such as a gauge average or a counter rate. These partial results are then reduced by a secondary function to produce results for the grouping dimensions, e.g. per host and process. Observability dashboards are built on top of this execution model, providing summary views of how metrics evolve over time and allowing for quick deep-dives by filtering on dimension values and time ranges.

To support this execution model, we introduced the TS source command, providing a simple yet powerful syntax for executing such queries that combine an inner aggregation function per time series with an outer aggregation over the partial results of the former. For instance, the following query calculates the hourly sum of rate of search requests per host over the last day:

TS metrics
  | WHERE TRANGE(1d)
  | STATS SUM(RATE(search_requests)) BY TBUCKET(1h), host

To execute this query, the compute engine is aware of how data is stored and applies the inner aggregation function per _tsid value. Since data are sorted by _tsid, time series aggregation functions process metric values as they get fetched, until the _tsid changes or the timestamp belongs to the next time bucket. This leads to vectorized execution of these functions over the fetched columns of metric values, while dimension values are only fetched (once) when the _tsid changes. The evaluation of the secondary aggregation function is also efficient, with partial aggregates stored in arrays of primitive values that get populated when _tsid values change.

The compute engine has inherent support for parallel query evaluation, taking full advantage of the available processing cores. Time series aggregations fully use this feature and process data points in parallel as applicable, reducing response times through improved cpu utilization.

Time series processing in ES|QL was introduced in version 9.2 as tech preview and reaches GA in version 9.4. We expect all metrics applications to adopt it and benefit from the much improved query performance wins.

Zero-copy data decoding and loading

Vectorized processing of time series data delivered immediate performance wins (8x for some queries), compared to aggregations through the /_search API, but the performance was still inferior when compared to competitive metrics stores. Benchmarking and profiling showed that there were too many array copies within the compute engine, between data decoding and evaluation of aggregation functions. To that end, the following optimizations were introduced:

• The codec for TSDS was extended to decode on-disk data directly into primitive arrays inside blocks that the compute engine uses to evaluate time series aggregations. No additional copies required, as the compute engine can bulk-read these blocks and process their arrays, one column at a time.
• Blocks containing a single value N times are represented as constant blocks with these 2 values, as opposed to an array with length N, a form of in-memory run-length encoding. Filtering and aggregation operations were extended to efficiently consume these blocks. This reduced memory pressure and cpu overhead for the _tsid and dimension fields, as their values get clustered due to index sorting.
• Documents with null values for the aggregated metric fields are filtered out at the Lucene level, before they get decoded and copied into blocks.
• All filters and regular expressions on the timestamp and dimension fields get pushed down to Lucene that makes use of doc value skippers to efficiently filter out non-matching docs.

Combined, these optimizations led to query execution speedups exceeding 10x (totaling 80x when combined with the 8x speedup from vectorized execution). They were included since the introduction of the TS source command in version 9.2, and fine-tuned ever since.

Optimized counter rate evaluation

While most time series aggregations can be trivially parallelized and evaluated, rate evaluation of cumulative counters is tricky as it requires processing all data points in order to detect counter resets (e.g. when a host restarts). To address this, the compute engine uses the _tsid prefix to shard time series across threads. Care has been taken to assign in-order ranges of _tsid values to each thread, as opposed to hash-partitioning _tsids, so that each thread can scan on-disk data in order, still making use of efficient decoding and zero-copying into blocks. The performance wins are impressive, with rate evaluation performance far exceeding dedicated metrics stores as we shall see in the next section.

Another interesting problem for cumulative counters is how to properly calculate counter increases for the entire time bucket when there are no data points at the bucket boundary timestamps. Metrics systems often use extrapolation, extending the first and last data points of each time bucket to the boundaries, or calculate the delta between the last data point of adjacent buckets. We posit we can do better, by interpolating between the last data point of each bucket and the first of the next, to get an estimate of the value on each boundary. The delta is then calculated over the interpolated values of the lower and upper boundary of each time bucket.

Sliding window support

Elasticsearch has long supported aggregations bucketed by time, but it was not possible to extend the window of processed data beyond each time bucket. Using windows larger than the time bucket, e.g. a window of 5 minutes for per-minute bucketing, helps smoothen out spikes and observe the underlying trend per time series with reduced noise:

All time-series aggregation functions have been extended with window support, as an optional argument. In case the window is a multiple of the time bucket (e.g. 1h window with TBUCKET(5m)), the compute engine first aggregates data points over intervals matching the time bucket span, and then combines these partial results per window span. This 2-phase approach eliminates repeated scans of data points and makes optimal reuse of intermediate results, improving response times. Window support was introduced as tech preview in version 9.3 and reaches GA in version 9.4.

Efficient datetime rounding

Queries on time-series data commonly include time bucketing. While data points can be trivially assigned to sub-hour time buckets, larger buckets start interfering with issues like time zones, daylight savings, variable days per month etc. Elasticsearch has elaborate logic for datetime rounding that takes these peculiarities into account, but that has relatively high cpu cost when processing time series data.

To mitigate this, the compute engine has been extended to identify cases where simpler logic can be employed to assign data points to time buckets. For instance, it can identify when the buckets are sub-hour or when timezones and daylight savings don't affect a particular query, and switches to simple modulo operations for datetime rounding. This led to a further 30% improvement in response times for certain queries. This change is introduced in version 9.4.

Performance evaluation

To evaluate the performance of our offering and track how it evolves and improves over time, we focused on OTel metrics since (a) Open Telemetry is the industry standard for collecting metrics, with universal adoption by all cloud providers and (b) they lead to a storage layout with 1 metric per doc, a setup that traditionally hurt performance for Elasticsearch.

We rely on Metricsgenreceiver to generate datasets. This tool is inspired by TSBS, producing data simulating the data points collected by the OTel hostmetricreceiver. We used two datasets:

1. A low-cardinality setup, with 100 hosts sending metrics every 1s, containing 14k time series in total
2. A high-cardinality setup, with 10k hosts sending metrics every 10s, containing 1.4M time series in total

We benchmarked on single-node deployments on EC2, using c6i.4xlarge and c8g.8xlarge machines for the low- and high-cardinality datasets respectively.

For competitive comparison, we used Prometheus (v.3.11.1), Mimir (v.3.0.6.) and ClickHouse (v26.3.9.8-lts). Prometheus and Mimir have proper time series processing, e.g. for counter rate, whereas ClickHouse lacks such support and only provides approximate values at best (for instance, it can't track counter resets consistently). We still report response times for ClickHouse to showcase that, once we optimize Elasticsearch for columnar query processing, it can exceed competing columnar engines even when they don't process the data per time series as expected.

We strived to use the default configuration for every system (including Elasticsearch), without tweaking them to optimize performance for the particular workload. This helps understand the user experience when systems are deployed by novice users, without much experience (or time) to tweak before receiving metrics traffic and setting up dashboards. We focused on single-node runs to keep noise low and accommodate all systems (Prometheus doesn't offer a multi-node setup out of the box). Elasticsearch performance provably scales well with the number of nodes; we plan to share scalability results in a future post.

Storage efficiency and indexing throughput

Our efforts to improve storage efficiency paid big dividends. Performance on OTel metrics dropped from 25 to 3.75 bytes per data point, in a year. Such an improvement, on top of an offering already optimized for time series, is really impressive and very rare in the industry.

The competitive picture looks favorable at this point, with Elasticsearch:

• Slightly outperforming Mimir in storage efficiency and indexing throughput
• Outperforming Prometheus by 2.5x in storage efficiency and by a small margin in indexing throughput
• Outperforming ClickHouse by 2x in storage efficiency and by 40% in indexing throughput

Query performance

The novel columnar engine for metrics processing proves very efficient in practice. We used a mix of queries based on gauge averages and counter rates, the most common operations that require different optimization approaches. The queried interval was 4 hours of data, covering all time series per metric.

ClickHouse doesn't support time series aggregations, so the results have limited value and are not directly comparable to Prometheus or Mimir that natively support time series processing. We used the published guidelines to adjust each query to get similar results to the extent possible. The point is to show how our columnar engine compares to generic columnar stores.

Here is a summary of the results:

†ClickHouse lacks native support for time series aggregations and counter reset detection.

Gauge average

We compared performance of evaluating the per-host hourly average of average memory utilization per time series, using the following queries:

# PromQL
avg by (host.name) (avg_over_time(system.memory.utilization[1h]))

# ES|QL
TS metrics-hostmetricsreceiver.otel-default
| STATS AVG(AVG_OVER_TIME(system.memory.utilization)) BY host.name, TBUCKET(1h)

Elasticsearch comfortably outperforms the other systems by up to 30x, in both low and high cardinality datasets.

Counter rate

We next compared performance of evaluating the per-host hourly average of cpu rate, using the following queries:

# PromQL
avg by (host.name) (rate(system.cpu.time[1h]))

# ES|QL
TS metrics-hostmetricsreceiver.otel-default
| STATS AVG(RATE(system.cpu.time)) BY host.name, TBUCKET(1h)

Despite processing data points per time series in order, counter rate performance matches calculating gauge average (the involved time series have 6.6x more docs than the query above). Elasticsearch maintains its wide advantage compared to the other systems and outperforms Mimir and Prometheus by 30x in the low cardinality dataset and by 16x in the high cardinality one.

It's really impressive that, for the high cardinality dataset, Elasticsearch is able to process 4 hours of data for half a million time series in less than 2 seconds, while the other systems take more than 30 seconds, leading to unresponsive dashboards for such queries. ClickHouse is also slower, despite having no logic to detect counter resets and extrapolate/interpolate deltas across buckets.

Prefix filter on host name

We next compared performance of filtering on host names based on their prefix, using the following queries:

# PromQL
avg by (host_name)
  (avg_over_time(system_cpu_load_average_1m{host_name=~"host-.*"}[5m]))

# ES|QL
TS metrics-hostmetricsreceiver.otel-default
| WHERE host.name LIKE "host-*"
| STATS AVG(AVG_OVER_TIME(system.cpu.load_average.1m)) BY host.name, TBUCKET(5m)

Elasticsearch manages to maintain an advantage of up to 5x compared to the other systems, despite replacing the inverted index on host.name with a doc value skipper.

Gauge average with window

We compared the performance of time series aggregations with a window of 90 minutes and time buckets of 30 minutes, using the following queries:

# PromQL
avg by (host.name) (avg_over_time(system.memory.utilization[90m]))&step=30m

# ES|QL
TS metrics-hostmetricsreceiver.otel-default
| STATS AVG(AVG_OVER_TIME(system.memory.utilization, 90m))
    BY host.name, TBUCKET(30m)

Elasticsearch maintains an advantage that reaches 25x for the low cardinality dataset and 8x for the high cardinality one. ClickHouse is outperformed by close to 4x, denoting the efficiency of our approach for windowed query operations.

What's next for Elasticsearch metrics

Elasticsearch has been extended with metrics storage and processing capabilities that outperform Prometheus, Mimir, and ClickHouse. We're making fast progress with supporting PromQL and Prometheus remote write, also available as tech preview in version 9.4. These extensions enable users familiar with Prometheus and relevant systems to switch their applications to Elasticsearch — no need to migrate existing dashboards. Since Prometheus integration reuses the same storage and query engine that has been presented in this article, the same performance wins are also expected for Prometheus. Furthermore, collected metrics can be queried with PromQL and ES|QL, side-by-side or in ES|QL query pipelines, further boosting the analytics capabilities far beyond what was conceivable so far with Prometheus-based systems.

The improvements in storage efficiency, indexing throughput and query performance are already impressive, but we're not done. We'll be introducing more refinements to the codec for time series data, further reducing bytes per data point. Batch processing of ingested metrics will be further improved, reducing synchronization overhead and redundant processing layers that are not needed for well-formatted collected metrics. We're also planning to make wider use of doc value skippers, storing pre-computed aggregates like sum and count per block of values, to shortcut data point loading and processing where applicable, as well as use more cpu-friendly partitioning and grouping operations.

From policy logic to retrieval architecture

Part 3 and Part 4 provided a technical deep dive into the governed control plane and its implementation using the Elasticsearch percolator. Once the logic layer has identified which policies to apply, the system must address the retrieval strategy used to execute the search.

Managing the transition from precision to recall is a critical function of any ecommerce search engine. For example, a basic search implementation often defaults to broad keyword matching. If a shopper searches for "organic Pink Lady apples", this can lead to irrelevant results, such as apple-scented dish soap, apple juice, or organic pink grapefruit, appearing at the top of the list simply because they share a common term. While these items are technically matches, they fail to satisfy the user's intent and typically lead to high bounce rates. However, a "No results" page is equally detrimental to conversion. This conflict is resolved by implementing a three-tier execution model, which uses the governed control plane to orchestrate a principled fallback strategy.

The three-tier execution model

This architecture executes up to three retrieval tiers in a sequence, each with a specific matching logic.

Highest tier: Strict matching

Strict matching is a lexical match that requires that all query terms appear in the product metadata.

• The logic: A search for "organic navel oranges" returns only products containing all three terms.
• Application: This tier provides the highest precision. When a customer types a precise product name, such as "organic navel oranges", they’re typically seeking that exact item rather than an alternative.

Mid-tier: Relaxed matching

If the strict tier fails to return sufficient results, the system expands the search parameters.

• The logic: This tier allows for a subset of terms to lexically match, using Elasticsearch's minimum_should_match logic.
• Application: Relaxed matching maintains lexical grounding. A search for "organic navel oranges" might surface "navel oranges" (missing the "organic" term) or "organic oranges" (missing the "navel" term). These represent intuitive, keyword-based alternatives for the shopper.

Lowest tier: Semantic matching

• The logic: This tier uses vector/semantic embeddings (such as Elastic Learned Sparse EncodeR [ELSER], E5, or Jina) to retrieve conceptually related products, regardless of direct keyword overlap.
• Application: A search for "organic navel oranges" might surface "mandarins" or "clementines”. This serves as the final retrieval tier, intended to provide relevant options when literal keyword matches are unavailable.

To see this multi-tier orchestration in action and how the Engine steps down from lexical to semantic matching, watch the video: Eliminating Zero-Result Pages: PRISM’s Multi-Tier Search Fallback.

Tier orchestration: The "bucket filling" logic

While the governed control plane provides the logic and the queries for each tier, the application layer is responsible for the execution. The application executes these tiers sequentially and excludes lower tiers once the accumulated result count on the first page reaches or exceeds 10 items (or whatever number of results you want to display on the first page). This threshold ensures a full first page of results while prioritizing the most accurate retrieval method.

Scenario 1: High-intent search ("oranges")

The first tier returns 15 hits. Since 15 is more than 10, the current result set is locked to only strict matches (which can be paged through) and subsequent tiers are not executed.

Strict tier:   [##########]##### (>= 10 found: Exact matches)
Relaxed tier:  [          ]      (Tier bypassed)
Semantic tier: [          ]      (Tier bypassed)

Scenario 2: Specific but limited results ("organic blood oranges")

The strict tier finds only four items. Since this is less than 10, the system triggers the relaxed tier, which finds 12 more relevant products. The combined total (16) meets the threshold of 10, so the current result set is locked to the strict and relaxed tiers. Subsequent paging will only surface results from these two tiers (preventing lower-quality semantic hits from appearing on later pages).

Strict tier:   [####      ]       (4 found)
Relaxed tier:  [    ######]###### (>= 6 found)
Semantic tier: [          ]       (Tier bypassed)

Scenario 3: Abstract or intent-based search ("high vitamin C snacks")

Keyword matches are limited (only five hits between tiers 1 and 2). The system triggers the semantic tier to find conceptually relevant items, such as kiwis, guavas, or red peppers, to fill the result set. The result set for this query includes products from all tiers.

Strict tier:   [##        ]             (2 found)
Relaxed tier:  [  ###     ]             (3 found)
Semantic tier: [     #####]######################...

This orchestration optimizes for latency, as the computational cost of the semantic tier is only incurred when the keyword-based tiers are insufficient. Additionally, this allows fast-responding keyword results to be displayed while semantic results are integrated shortly after, maintaining a responsive user interface.

Determining intent via tier activation

The logic used to fill the first page serves a critical secondary purpose: It acts as a diagnostic for user intent. The application uses the logic returned by the governed control plane to determine which tiers remain active for the current result set and paging.

If the strict and relaxed tiers together yield fewer than 10 results, the query is likely exploratory or abstract. In this case, activating the semantic tier is a benefit. Because the query is diagnosed as exploratory, the system allows the shopper to page through the entire depth of the semantic results. This provides access to conceptually related alternatives that lexical matching would have missed, which is appropriate for an abstract search.

Conversely, if the strict tier returns a robust set of results (for example, 30 hits), it confirms that the system has found high-precision matches. The user can page through those 30 hits and will likely find what they’re looking for. In this scenario, there’s no need to provide additional, less relevant exploratory hits. By disabling lower tiers for these high-precision queries, we ensure that a shopper deep diving into specific results isn’t distracted by irrelevant semantic fallback as they paginate through the current result set.

Governance across tiers

A critical component of this architecture is that policies apply globally across all tiers. If a user has a "vegan" preference profile, the governed control plane injects that constraint into the strict, relaxed, and semantic queries. This ensures that even when the system uses semantic fallback to return "mandarins" for an orange search, the results remain compliant with the user's broader dietary preferences or business constraints.

The problem of facet instability

A challenge with multi-tier search is maintaining consistent faceted navigation (sidebar filters). If a search for "chocolate" yields 12 strict results, the sidebar filters might show "dark" and "milk". If a user selects "dark" and the result count drops, a naive system might trigger the semantic tier to fill the page, which could suddenly introduce "red wine" into the filters due to a semantic relationship.

The governed control plane identifies which tiers contributed to the initial search and locks the facets to those tiers. This prevents the sidebar from changing unexpectedly during a filtered session, ensuring a stable user experience.

The pagination challenge: Seamless multi-tier paging

Pagination in a tiered system requires precise state management. As established, the first page determines the scope of the current result set. If the first page required semantic results, the user can page through all available results from all three tiers. On the other hand, if the first page was satisfied by high-intent keyword matches, the semantic tier is not retrieved for that specific result set.

The governed control plane manages this through:

• Tier locking: The response includes an array identifying the contributing tiers. The front end returns this on subsequent requests to keep the tier composition consistent across all pages.
• Dynamic offset calculation: The back end calculates an offset based on the requested page and the total products returned in preceding tiers.Example: If the first page has returned seven strict matches and three relaxed matches, a request for page 2 (starting at index 10) would execute a relaxed tier query with an offset of three.
• ID exclusion for lower tiers: The system retrieves IDs from the higher tiers (which, by definition, will always be fewer than the page size threshold) and explicitly excludes them from lower-tier results using an ID-only query (which avoids the overhead of a full fetch phase for excluded items).

Summary

The multi-tier approach ensures search results are precise when data is available and helpful when it is not. By providing a governed fallback sequence for the application to execute, the architecture maintains high relevance while eliminating "no results" scenarios.

What's next in this series

The next posts in this series extend the governed control plane into new territory. Part 6 explores personalization (using purchase history boosting and cohort-aware policies), and Part 7 demonstrates per-query economic optimization. Stay tuned!

Put governed ecommerce search into practice

The search architecture described in this post, where retrieval tiers, economic weights, and governance constraints compose into a single request, was designed and built by Elastic Services Engineering as part of our repeatable ecommerce search accelerators.

To learn more about applying these patterns to your business, Contact Elastic Professional Services.

The problem: Heterogeneous data, one query

Consider a production incident investigation. Errors are spread across three microservices: an API gateway, a payments service, and an auth service, each with different field names and different conventions. Before subqueries, combining them in a single ES|QL query meant cramming everything into one FROM with CASE chains:

FROM svc-gateway-*, svc-payments-*, svc-auth-* METADATA _index
| WHERE http.response.status_code >= 500
    OR transaction.status IN ("failed", "timeout")
    OR (event.action == "login" AND event.outcome == "failure")
| EVAL
    service = CASE(
      _index LIKE "svc-gateway*", "gateway",
      ... /* one branch per source */),
    error_detail = CASE(
      _index LIKE "svc-gateway*", CONCAT("HTTP ", http.response.status_code::string),
      ... /* one branch per source */)
| KEEP @timestamp, service, error_detail, source.ip
| SORT @timestamp DESC

This is brittle and slow. The disjunctive OR prevents predicate pushdown; every index scans every condition. Every CASE chain grows with every source. Copy it into five dashboards and three alert rules, and you have eight places to update when anything changes.

The fix: Independent pipelines

Subqueries replace the monolithic FROM + CASE pattern. Each data source gets its own complete pipeline:

FROM
  (FROM svc-gateway-*
   | WHERE http.response.status_code >= 500
   | EVAL service = "gateway",
         error_detail = CONCAT("HTTP ", http.response.status_code::string)
   | KEEP @timestamp, service, error_detail, source.ip),
  (FROM svc-payments-*
   | WHERE transaction.status IN ("failed", "timeout")
   | EVAL service = "payments",
         error_detail = transaction.status
   | KEEP @timestamp, service, error_detail, source.ip),
  (FROM svc-auth-*
   | WHERE event.action == "login" AND event.outcome == "failure"
   | EVAL service = "auth",
         error_detail = CONCAT(event.action, " ", event.outcome)
   | KEEP @timestamp, service, error_detail, source.ip)
| SORT @timestamp DESC
| LIMIT 20

The gateway branch only scans for HTTP 500s. The payments branch only looks at transaction statuses. The auth branch only checks login failures. Because each branch has its own WHERE, the optimizer pushes filters independently into each index, restoring the predicate pushdown that a single FROM with OR conditions prevents. Fields that exist in one branch but not another are filled with null.

Adding a fourth service means adding a fourth branch. Existing branches don't change.

Save it as a view

This is where subqueries and logical views combine. Wrap the subquery above in a named view, with one API call:

PUT _query/view/error_triage
{
  "query": "FROM (FROM svc-gateway-* | WHERE ...) , (FROM svc-payments-* | WHERE ...) , (FROM svc-auth-* | WHERE ...)"
}

Now consumers just write FROM error_triage | STATS error_count = COUNT(*) BY service. Three indices, three pipelines, one name. If you have 10 dashboards and five alert rules consuming this pattern, that's 15 copies of the same logic today; with a view, it's one definition and zero consumer-side edits when you add a fourth service. See Elasticsearch ES|QL Views for the full views deep dive.

What you can do inside a branch

Each branch supports the full ES|QL pipeline: WHERE, EVAL, STATS, LOOKUP JOIN, ENRICH, and more. See the subquery documentation for the complete list.

Aggregate different metrics, and then combine

Each branch can compute its own summary before results are merged. This is useful when different indices track the same concept under different field names:

FROM
  (FROM svc-gateway-*
   | STATS avg_latency = AVG(http.response.time_ms) BY hour = BUCKET(@timestamp, 1 hour)
   | EVAL service = "gateway"),
  (FROM svc-payments-*
   | STATS avg_latency = AVG(transaction.duration_ms) BY hour = BUCKET(@timestamp, 1 hour)
   | EVAL service = "payments")
| SORT hour DESC, service

Both branches produce avg_latency and hour, but each computes it from a different source field. The combined result is a single table you can chart or alert on, without normalizing field names at ingest time. This pattern is impossible with a single FROM; you can't compute different aggregations per index without subqueries.

Subqueries vs. FORK

ES|QL also has FORK (now generally available), which creates parallel execution branches from the same input. The distinction:

Different indices → subqueries. Same data, different analyses → FORK.

How this compares

If you're coming from other query languages, here's how ES|QL subqueries stack up at the time of writing:

Splunk SPL/SPL2 has append and multisearch in classic SPL, and SPL2 adds a union command that merges events from multiple datasets (the closest analogue to ES|QL subqueries). Federated Search extends this across remote Splunk deployments (analogous to CCS). The differences are in how the engine handles each branch: ES|QL subqueries give each branch independent predicate pushdown, meaning filters are pushed into each index's shard-level structures separately. SPL2 union merges datasets but optimization across branches is limited to what the search scheduler can parallelize. Wrapping ES|QL subqueries in a view gives you engine-level encapsulation with role-based access control (RBAC); Splunk's equivalent is saved searches and macros, which are text substitution expanded at parse time.

SQL databases have UNION ALL, which is the closest analog. The difference is that SQL UNION ALL typically requires matching column counts and types at parse time. ES|QL subqueries are more forgiving; columns that exist in one branch but not another get null-padded automatically, which matters when your sources have different schemas (the norm in observability data). SQL views solve the reuse problem similarly, but ES|QL views are cluster-level objects, not database-scoped; they work across cross-cluster search boundaries.

Grafana / Datadog / other dashboarding tools handle multisource composition at the visualization layer: Run separate queries, merge in the panel. This works for display but breaks for alerting, downstream queries, and anything that needs a single result set programmatically. ES|QL subqueries push the composition into the engine, so alerts, views, and API consumers all get the same unified result.

Current constraints

In the Tech Preview release, subqueries are non-correlated; branches run independently and can't reference the outer query. They're supported in FROM only (not TS), and FORK can't be used inside or after subqueries. See the subquery documentation for details.

What's next for subqueries

WHERE subqueries — WHERE field IN (FROM other_index | ...) and other correlated forms — will extend the composition model from FROM into filtering. This brings the familiar SQL pattern of nested filtering to ES|QL.

Try it

Subqueries in FROM are available as a Tech Preview. Try them in Kibana Dev Tools or Discover. We'd love your feedback; file a GitHub issue with the ES|QL label.

ES|QL subqueries in FROM are a Tech Preview feature. Tech Preview features are subject to change and are not covered by the support SLA of GA features. The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

A brief quantization primer

Vector search indexes often store millions or billions of embedding vectors, each hundreds or thousands of floats wide. Scalar quantization compresses each float coordinate independently to a small integer, typically 1, 2, or 4 bits, reducing storage by 8-32× and enabling fast integer-arithmetic scoring.
Why does per-coordinate quantization work at all? Because the expected squared error decomposes as a sum of independent per-coordinate terms: $\mathbb{E}[\|x - q(x)\|^2] = \sum_i \mathbb{E}[(x_i - q_i(x_i))^2]$, by linearity of expectation. Each coordinate can be quantized independently regardless of the joint distribution. The remaining question is how to make each per-coordinate quantizer good.

BBQ and Optimized Scalar Quantization

Better Binary Quantization (BBQ), and its underlying algorithm Optimized Scalar Quantization (OSQ), have been part of Elasticsearch for multiple releases. OSQ is an evolution of several techniques to make scalar quantization more accurate, specifically for vector search.

Each vector's components are mapped to a uniform grid over an interval $[a, b]$. The interval is initialized from the vector's statistics (assuming approximately normal residuals), using the exact same optimization objective as TurboQuant but with an additional constraint on centroid positions. They are then refined by coordinate descent to minimize an anisotropic loss $L = (1-\lambda)(x \cdot e)^2/\|x\|^2 + \lambda\|e\|^2$, where $e$ is the quantization error vector. With the production default $\lambda = 0.1$, this deliberately sacrifices some MSE to concentrate accuracy along the query direction. This is the direction that matters for ranking.

Before quantization, the segment (or cluster in the case of Inverted Vector File (IVF) indices) centroid $c$ is subtracted from every vector. This removes the dominant shared component that would otherwise waste the quantizer's dynamic range. Both the symmetric and asymmetric dot-product paths also center the query using the same segment centroid, so the only quantized inner product is between centered residuals. The correction terms, $\langle c, x \rangle$ for each vector and $\|c\|^2$, depend on only a single vector each, and can be precomputed exactly. Centering therefore adds no per-pair cost.

Documents are quantized at 1-bit (32× compression), queries at 4-bit (cheap since there are only a handful per search). The storage constraint binds on documents, not queries, so spending more bits on the query side recovers float-query accuracy while keeping per-document footprint minimal.

A block-diagonal orthogonal preconditioner equalizes coordinate variances and normalizes their distribution before quantization. This is the same goal as a full Hadamard rotation used by TurboQuant, but with no power-of-2 padding overhead.

Because the grid is uniform, quantized dot products decompose into integer dot products with scalar corrections. This enables NEON/SVE and SSE/AVX popcount and multiply-accumulate pipelines: bit-plane decomposition for 1-bit and 2-bit, nibble multiply for 4-bit, and a RaBitQ-style mixed 4×1 kernel that decomposes to four 1-bit kernels.

For a deeper dive into the sparse rotation and what it brings to robustness, see the Robust Optimized Scalar Quantization blog. For a full walkthrough of optimized scalar quantization, see this OSQ deep dive.

TurboQuant

TurboQuant (Google, ICLR 2026) takes a slightly different path to the same starting observation: that concentrated, predictable per-coordinate distributions are easy to quantize well.

Rather than adapting the quantizer per vector, TurboQuant normalizes the vector and applies a shared randomized Hadamard rotation to the entire dataset. This general sort of idea was first proposed and formalized by RaBitQ, which showed that the random rotation yields worst-case bounds over the data, holding for any fixed unit vector, for their quantization scheme. The idea of implementing via a Hadamard rotation was suggested by Weaviate for rotational quantization. After normalization and rotation, each coordinate's distribution almost always converges to $\mathcal{N}(0, 1/d)$ in high dimensions, regardless of the original data. TurboQuant builds on this foundation: with the distribution pinned down, it solves for the optimal Lloyd-Max scalar quantizer, a 1-D $k$-means problem on the known density. The resulting non-uniform centroids bunch up where the density is highest (near zero) and spread out in the tails. This achievies provably near-optimal MSE: within ~2.7× of the information-theoretic lower bound in general, and as tight as 1.45× at 1-bit.

For inner products, MSE-optimal quantizers introduce a multiplicative bias (which is most severe at 1-bit: $2/\pi \approx 0.64$). TurboQuant corrects this with a two-stage design ($Q_\text{prod}$): spend $b-1$ bits on the MSE quantizer, then use the remaining 1 bit for a Quantized Johnson-Lindenstrauss (QJL) sketch of the residual, yielding a provably unbiased inner-product estimator.

The paper's nearest-neighbour experiments were conducted on GPU (NVIDIA A100), where the lookup-table access pattern maps naturally onto shared memory.

The key design divergence: integer arithmetic vs. lookup tables

The difference between uniform and non-uniform centroids may seem minor, but it creates a large computational gap.

OSQ's uniform grid means each quantized coordinate is an integer whose arithmetic meaning is preserved. The dot product of two quantized vectors decomposes into an integer dot product, directly exploitable by SIMD: vpdpbusd on x86, multiply-accumulate and vcnt (popcount) on ARM NEON. The pipeline is branch-free and the data access pattern is sequential.

TurboQuant's non-uniform centroids break this. Each coordinate pair requires looking up a centroid value from a shared codebook, and the access pattern is data-dependent with each index selecting a different table entry. On NEON, which lacks a float gather instruction, this means scalar loads to build each vector register before the Fused Multiply-Add (FMA). Precomputing per-coordinate product tables ($d \times 2^b$ entries, amortized over all documents) doesn't help either: the FMA is relatively cheap on modern cores, so the bottleneck remains the data-dependent gather, not the arithmetic. Our benchmarks confirm this: precomputed ADC tables are no faster (and sometimes slower due to the larger working set) than inline centroid lookup.

Terminology used in the comparisons

The results sections below refer to several OSQ scoring configurations. All use uniform-grid quantization with scalar correction terms to recover the exact dot product up to quantization noise.

Symmetric $n$-bit quantizes both query and document at $n$ bits per coordinate.

Asymmetric keeps the query as a full float vector and quantizes only the document. The dot product is a float-times-integer sum. This is more expensive per pair than symmetric, but avoids any query quantization noise. TurboQuant's scoring is always asymmetric (float query dotted against quantized document via centroid lookup).

1-4 is the production configuration for OSQ: documents at 1-bit (32× compression), queries at 4-bit. This exploits the asymmetry of search: there is one query but millions of documents, so query storage is free but document storage is the binding constraint.

Centered means the segment centroid $c$ has been subtracted from all vectors (and the query) before quantization, with the exact correction recovered from precomputed scalar terms. Centering focuses the quantizer's dynamic range on the information-bearing residual rather than the shared mean.

$\lambda$ controls the anisotropic loss tradeoff: $\lambda = 1$ minimizes pure MSE, $\lambda = 0.1$ (production default) sacrifices some MSE to concentrate accuracy along the query direction, the direction that determines ranking.

How do they compare in practice?

The following results were obtained on an Apple M2 Max. The code to reproduce all these results is available here.

Head-to-head: MSE

On reconstruction MSE, the metric TurboQuant was designed to optimize, TurboQuant outperforms plain OSQ at every bit-width.

Relative MSE ($\|x - \hat{x}\|^2 / \|x\|^2$) on $d = 768$ Gaussian vectors (1,000 vectors, lower is better):

The $\lambda$ columns reveal that OSQ's production setting ($\lambda = 0.1$) deliberately sacrifices MSE for dot-product accuracy. With $\lambda = 1$ (pure MSE), the gap narrows, to just 1.18× at 1-bit.

But where does TurboQuant's remaining MSE advantage actually come from, the Lloyd-Max centroids, or the Hadamard rotation? We can answer this directly by applying the same randomized Hadamard rotation to OSQ (zero-pad 768→1024, random sign flips, Walsh-Hadamard butterfly, quantize in rotated space, invert). Theory predicts the MSE improves by a factor of $d/d' = 768/1024 = 0.75$:

OSQ + Hadamard matches TurboQuant almost exactly at 1-bit (0.306 vs 0.307) and 2-bit (0.092 vs 0.092). TurboQuant's MSE advantage is the rotation, not the centroids. At 3–4 bits the Lloyd-Max placement contributes a modest ~1.1× edge, real but small.

The convergence of the improvement ratio applying the Hadamard transformation to OSQ is itself informative: at 4-bit it hits the theoretical 1.33 exactly, but at 1-bit it's only 1.19. The shortfall quantifies the value of OSQ's data-dependent interval refinement: it already captures ~40% of the dimension expansion and component equalization benefit that Hadamard provides. The coordinate-descent is doing some of the same work as the rotation, adapting to each vector rather than relying on a data-oblivious transform. However, the real advantage, as we discuss below, is this formulation allows us to concentrate accuracy along the query direction.

This raises a natural question: how does OSQ's block-diagonal sparse preconditioner compare to the full Hadamard rotation in practice?

Head-to-head: sparse preconditioner vs Hadamard

OSQ's sparse preconditioner applies a block-diagonal random orthogonal transformation: dimensions are randomly permuted into blocks (64×64 in production), each block is multiplied by an independent random orthogonal matrix. This equalizes coordinate distributions within each block. The Hadamard rotation achieves the same goal globally but requires zero-padding to the next power of 2.

We test on anisotropic Gaussian data ($d = 768$, $\sigma_i$ ramping from 1 to 5 across coordinates), a challenging distribution where some coordinates carry far more variance than others.

Transform latency ($d = 768$, 1,100 vectors, ARM NEON, lower is better):

Hadamard is the fastest non-trivial option thanks to $O(d \log d)$ butterflies vs $O(d \times b)$ for block size $b$, though all block-diagonal variants are fast enough to be negligible in practice with even the 64×64 block at 4.9 μs is tiny compared to typical search latencies. The full dense $d \times d$ rotation is impractical at 244 μs/vec but serves as a theoretical reference. Note that the block-diagonal transform works for arbitrary dimensions: no power-of-2 padding is required, and the effective dimension stays at $d$.

MSE (relative MSE, $\lambda = 1$, anisotropic data, lower is better):

Even 32×32 blocks recover most of the gap from no-transform (0.443) to full rotation (0.362), 93% at 1-bit. Block 64×64 closes the gap further. On isotropic data (not shown), all methods produce identical MSE (~0.362 at 1-bit), confirming there is nothing to equalize when coordinates already have equal variance.

Dot-product accuracy (1-4 centered, raw relative dot-product error; note these are raw RMSE including multiplicative bias, which is appropriate for comparing preconditioner variants against each other since the bias structure is similar, lower is better):

On isotropic data the block-diagonal methods and the full dense rotation produce the same dot-product error as no transform since there is nothing to fix. Hadamard is the outlier, improving from 0.723 to 0.629. But this improvement is not from better preconditioning: the full dense rotation, which is an equally good random orthogonal transform, shows no improvement at all. The difference is the padding. Hadamard operates in 1024 dimensions, so 1-bit documents store 1024 bits instead of 768. This is 33% more storage. The improvement ratio (0.723 / 0.629 = 1.15) matches $\sqrt{d'/d} = \sqrt{1024/768} = 1.155$ almost exactly, confirming that the entire dot-product advantage is attributable to the extra bits, not the rotation.

On anisotropic data, the block-diagonal rotation does help dot-product accuracy (0.690 → 0.602 for block 64), which is the real value from coordinate equalization. Hadamard goes further (0.566), but the incremental improvement over a full dense rotation at the same dimension (0.595 → 0.566) is again consistent with the padding benefit.

The practical implication: for CPU-based search where storage efficiency matters, the block-diagonal preconditioner delivers the same MSE improvement as Hadamard at the same effective bit rate, works for any dimension without padding, and the dot-product gap we see in our experiments is a padding artifact, not a preconditioning advantage.

Head-to-head: dot-product accuracy

MSE measures reconstruction quality, but search engines rank by dot products. These are different objectives, and the gap between them is where OSQ's design choices pay off.

We measure relative dot-product error: $\sqrt{\sum(q \cdot x - \hat{q} \cdot \hat{x})^2 / \sum(q \cdot x)^2}$, varying the angle between query and document. The small-angle regime (0°–20°) matters most: real transformer embeddings occupy a narrow cone rather than spreading uniformly on the sphere (Ethayarajh 2019). Furthermore, near-parallel vectors, corresponding to the nearest neighbours of a query in the dataset, are where ranking accuracy is critical.

Our production configuration is 1-bit documents, 4-bit queries, centroid centering, with integer scoring.

Raw dot-product error conflates two distinct components: a multiplicative bias (a global scale factor that preserves ranking order) and noise (random per-pair deviations that can swap rankings). For search, only the noise matters: a biased estimator that consistently scales all scores by the same factor produces the same ranking as the exact scores. TurboQuant's MSE quantizer at 1-bit has a well-known multiplicative bias of $2/\pi \approx 0.64$, meaning raw dot-product errors of ~0.36 are almost entirely this ranking-irrelevant scale factor. To give a fair comparison, we report the debiased RMSE after fitting and removing the best multiplicative scale: $\alpha = \sum(\hat{d} \cdot d) / \sum(d^2)$, then measuring $\sqrt{\sum(\hat{d}/\alpha - d)^2 / \sum d^2}$.

Zero-mean corpus ($d = 768$, 500 vectors, 5 queries per vector, lower is better):

Shifted corpus (shift = 2.0, modeling real embedding bias, lower is better):

On zero-mean data, the raw error numbers (omitted for brevity, TQ @1-bit's is ~0.363, almost entirely due to the $2/\pi$ multiplicative bias) are misleading; only the debiased ranking noise matters. The asymmetric column (float query, 1-bit document) is the most directly comparable to TQ since both quantize only the document: at 0° OSQ achieves 2.4× lower noise (0.0035 vs 0.0083). This is the payoff of the anisotropic loss ($\lambda = 0.1$), which concentrates accuracy along the query direction at the expense of off-axis components. The symmetric 4-bit query recovers some of this advantage (0.0035 → 0.0067), showing that query quantization is now the dominant noise source at small angles. Even so, OSQ symmetric still beats TQ @1-bit by 1.2–1.4× through 10°. The tradeoff is visible at wider angles where TQ @1-bit has lower noise than OSQ (0.042 vs 0.049 at 60°): the Hadamard rotation distributes information uniformly across all directions, while OSQ deliberately favors the directions that matter for search.

What about TurboQuant's $Q_\text{prod}$ variant? TurboQuant's inner-product variant ($Q_\text{prod}$) was designed to address exactly this bias, spending $b-1$ bits on the MSE quantizer and 1 bit on a QJL sketch of the residual to produce a provably unbiased estimator. At 1-bit $Q_\text{prod}$ is not viable (0 bits for MSE), so the minimum is 2-bit. But for ranking, the cure is worse than the disease: $Q_\text{prod}$ trades ranking-irrelevant bias for ranking-relevant noise. At 60°, $Q_\text{prod}$'s debiased noise is consistently higher than MSE-only at the same total bit width, 0.031 vs 0.025 at 2-bit and 0.011 vs 0.007 at 4-bit, because each bit spent on QJL correction would have been better spent on quantization. Since search cares only about ranking, MSE-only is the better choice. The bias is harmless and the extra quantization bit reduces the noise that actually matters.

The picture changes on shifted data, where centroid centering gives OSQ a decisive advantage. At 0° the debiased noise drops to 0.0008, which is 9× lower than TQ @1-bit's 0.0073, and 7× lower than TQ @4-bit's 0.0054. Centering removes the dominant shared component before quantization, letting the quantizer focus its bits on the information-bearing residual. TurboQuant's data-oblivious rotation cannot exploit this structure. The advantage persists through 20° (OSQ 0.0041 vs TQ @1-bit 0.012) and only narrows at wide angles (60°: OSQ 0.022 vs TQ @1-bit 0.043), where OSQ remains competitive.

On shifted data, OSQ at 1-bit per document (debiased noise 0.001) beats TurboQuant at 4-bit per document (debiased noise 0.006): better ranking accuracy at over 5× less storage (768 bits vs 4,096 bits, since TQ pads 768→1024 for the Hadamard transform). This is the payoff of the data-dependent design: centering and anisotropic interval refinement extract structure that a data-oblivious rotation cannot.

TQ @4-bit MSE is consistently the lowest-noise option on zero-mean data (debiased 0.005-0.008 across all angles), but at 5× the storage cost per document. On shifted data it is actually substantially worse than OSQ symmetric for angles less than 20°.

Head-to-head: throughput

Throughput is where the uniform grid constraint really shines. Here are the throughput figures on $d = 768$, 10k documents, Apple M2 Max, 100 repetitions:

OSQ's symmetric kernels are 10–40× faster than TurboQuant!

We made a fair effort to optimize both implementations to use ARM NEON instructions effectively, but do not claim these are optimal. The key techniques:

The 1-bit kernel reduces to popcount(a AND b) via NEON's vcntq_u8, processing 32 bytes per iteration with dual accumulators for pipeline parallelism. For $d = 768$ the entire packed vector is 96 bytes, a single pass yields 7 ns/doc.

The 2-bit kernel decomposes each 2-bit index into two bit-planes (precomputed at quantize time), reducing the dot product to 4 AND+popcount passes over the same 96-byte planes: $\sum(2x_1+x_0)(2y_1+y_0) = 4 \cdot \text{pc}(x_1 \wedge y_1) + 2 \cdot \text{pc}(x_1 \wedge y_0) + 2 \cdot \text{pc}(x_0 \wedge y_1) + \text{pc}(x_0 \wedge y_0)$. At 14 ns/doc this is 2× the 1-bit time rather than the naive 4× because all four plane pairs share the same data loads, each 96-byte plane is read once and reused across passes.

The 4-bit kernel uses direct NEON nibble multiply with vandq/vshrq to split packed bytes into lo/hi nibbles, multiply, and accumulate via vpaddlq_u8 widening adds. At 22 ns/doc, this is faster than the 16-popcount bit-plane alternative ($4^2 = 16$ plane combinations).

A mixed 4×1 kernel is the production workhorse. It precomputes the 4-bit query's 4 bit-planes at quantize time (each 96 bytes in the same 1-bit packed layout as the document). Per-document scoring is then 4 AND+popcount passes, i.e. the RaBitQ decomposition: $\sum \text{idx4}_i \times \text{idx1}_i = 8 \cdot \text{pc}(\text{plane3} \wedge \text{bits1}) + 4 \cdot \text{pc}(\text{plane2} \wedge \text{bits1}) + \ldots$ At 14 ns/doc this is 21.3× faster than TurboQuant's 1-bit path at 3/4 the document storage.

TurboQuant's bottleneck is the data-dependent gather: each coordinate requires a scalar load from the centroid table to build a NEON float vector. The arithmetic (FMA) is essentially free in comparison.

Conclusion

TurboQuant is a theoretically elegant construction that builds directly on the OSQ formulation. The provable MSE bound, the unbiased inner-product estimator, and the clean data-oblivious design are real contributions. For applications requiring calibrated scores (not just rankings), or running on GPU hardware where gather operations are cheap, TurboQuant's architecture is well-motivated. The calibration-free design is also a natural fit for settings where quantization must happen on the fly with zero training overhead, KV cache compression during LLM inference is a prime example. There, every vector is quantized once as it enters the cache and discarded after the forward pass, so there is no opportunity to amortize a per-vector coordinate descent. A fixed codebook derived from the known post-rotation distribution is exactly the right tool: rotate, snap, store.

But for CPU-based vector search, the setting where Elasticsearch and most operational systems execute queries, the empirical picture is clear across all three axes:

MSE: TurboQuant's advantage comes from the Hadamard rotation, not the Lloyd-Max centroids. OSQ with the same rotation matches TurboQuant at 1–2 bits and comes within 1.1× at 3–4 bits. OSQ's sparse preconditioner already provides this benefit without padding overhead.

Dot-product accuracy: After removing ranking-irrelevant multiplicative bias (including TQ's $2/\pi$ scale factor at 1-bit), OSQ has 1.2–1.4× lower ranking noise than TQ @1-bit at small angles on zero-mean data even with a quantized query and without the 25% pad, thanks to the anisotropic loss concentrating accuracy along the query direction. On shifted data, the regime that matters in practice because embeddings typically have a non-zero mean, centering amplifies the advantage further: debiased noise of 0.0008 at 0° vs TQ @1-bit's 0.0073 and even TQ @4-bit's 0.0054. OSQ at 1-bit beats TurboQuant at 4-bit on ranking accuracy at less than 1/5 the storage. TurboQuant's $Q_\text{prod}$ variant addresses bias explicitly but trades it for higher noise, making MSE-only the better choice for search.

Throughput: 10–40× faster symmetric scoring, with the mixed 4-1 kernel at 14 ns/doc versus TurboQuant's 293 ns/doc using NEON intrinsics. This reflects a fundamental architectural divide between integer arithmetic and lookup-table gather, not a constant factor that disappears with batching.

The uniform grid, far from being a compromise, turns out to be the right trade: it sacrifices a theoretical MSE margin that almost vanishes under equivalent rotation, and in return unlocks the integer-arithmetic pipeline that makes sub-millisecond search at scale practical.

Views don't store data; they re-execute on every read, so results always reflect the current data and the current definition. If you've used views in SQL databases, this will feel familiar. The difference: ES|QL views are engine-level virtual indices stored at the Elasticsearch cluster level, not saved query text that gets expanded client-side. They appear in Kibana autocomplete, support cross-cluster search (CCS), and are governed by dedicated role-based access control (RBAC) privileges.

A simple view

A view can wrap any ES|QL query. Start with a straightforward filter — HTTP 500 errors from the API gateway:

PUT _query/view/error_triage
{
  "query": """
    FROM svc-gateway-*
    | WHERE http.response.status_code >= 500
    | KEEP @timestamp, http.response.status_code, url.path, source.ip
  """
}

Now anyone can write FROM error_triage without knowing the index pattern or filter condition:

FROM error_triage
| STATS error_count = COUNT(*) BY url.path
| SORT error_count DESC

The query is defined once. Consumers reference a name.

Views support full create, read, list, update, and delete (CRUD) via the _query/view REST API.

Update propagation

Say the team decides error_triage should also capture client errors, not just 500s. Update the definition in place:

PUT _query/view/error_triage
{
  "query": """
    FROM svc-gateway-*
    | WHERE http.response.status_code >= 400
    | KEEP @timestamp, http.response.status_code, url.path, source.ip
  """
}

Every dashboard panel, alert rule, and ad-hoc query using FROM error_triage immediately reflects the broader filter. No saved objects to hunt down. No stale copies. Change once, update everywhere.

Nested views

Views can reference other views, enabling layered abstractions. Create views for suspicious IPs and threat intelligence, and then compose them:

PUT _query/view/suspicious_ips
{
  "query": """
    FROM svc-auth-*
    | WHERE event.action == "login" AND event.outcome == "failure"
    | STATS attempts    = COUNT(*),
            first_seen  = FIRST(@timestamp, @timestamp),
            latest_user = LAST(user.name, @timestamp)
        BY source.ip
    | WHERE attempts > 3
  """
}

PUT _query/view/known_threats
{
  "query": """
    FROM threat-intel
  """
}

PUT _query/view/security_overview
{
  "query": """
    FROM suspicious_ips, known_threats
  """
}

FROM security_overview
| WHERE source.ip IS NOT NULL
| EVAL is_known_threat = threat.category IS NOT NULL
| KEEP source.ip, attempts, threat.category, threat.severity, is_known_threat
| SORT is_known_threat DESC, attempts DESC

Security teams query FROM security_overview without knowing the underlying data model. They're also shielded from any changes made to suspicious_ips by its owner; the abstraction boundary is real, not syntactic.

Multisource views with subqueries

A view can wrap any ES|QL query, including multisource compositions, using subqueries in FROM. Each subquery branch queries one service independently (its own filters, its own field normalization), and the results combine automatically:

PUT _query/view/all_errors
{
  "query": """
    FROM
      (FROM svc-gateway-*
       | WHERE http.response.status_code >= 500
       | EVAL service = "gateway",
              error_detail = CONCAT("HTTP ", http.response.status_code::string)
       | KEEP @timestamp, service, error_detail, source.ip),
      (FROM svc-payments-*
       | WHERE transaction.status IN ("failed", "timeout")
       | EVAL service = "payments", error_detail = transaction.status
       | KEEP @timestamp, service, error_detail, source.ip)
  """
}

Consumers just write:

FROM all_errors
| STATS error_count = COUNT(*) BY service
| SORT error_count DESC

Two indices, two independent pipelines, one name. To add a third service later, add a third branch; existing branches don't change, and every downstream dashboard and alert reflects the update automatically. For a deep dive on subquery syntax and what you can do inside each branch, see Three Indices Walk Into a FROM Clause.

How views work under the hood

When you write FROM view_name, ES|QL resolves the view's stored query and executes it inline. Views are re-executed on every read, so results always reflect the current data and the current definition.

Views share a namespace with indices, aliases, and data streams. A view cannot have the same name as any of these (enforced at creation time). This keeps FROM my_name unambiguous regardless of whether the name resolves to a view, an index, or an alias.

Security model

Views are governed by four dedicated RBAC privileges: create_view, read_view_metadata, delete_view, and manage_view. Elasticsearch checks the privileges of the user running the query (invoker security), not the user who defined the view. The user querying a view needs permissions on both the view and its underlying indices.

Kibana integration

Views appear in Discover's ES|QL editor autocomplete alongside indices. ES|QL-based dashboard panels work with views transparently. In the initial Tech Preview release, view management is API-only. A Kibana UI for creating and managing views is planned.

Cross-cluster search

A view's definition can reference remote indices using CCS syntax:

PUT _query/view/cross_cluster_errors
{
  "query": """
    FROM cluster-west:logs-*, cluster-east:logs-*
    | WHERE log.level IN ("error", "crit")
  """
}

Consumers query FROM cross_cluster_errors without knowing which clusters are involved.

Current constraints

In the Tech Preview release, view management is API-only and SET directives can't appear inside view definitions; the caller applies them when querying. Subquery-based views can't be nested inside other multisource FROM expressions. See the views documentation for the full list.

What's next for views

Views today are always fresh; they re-execute on read. Materialized views flip that tradeoff: Pre-compute once, read instantly. Think pre-aggregated rollup views for Service Level Agreement (SLA) dashboards that load in milliseconds instead of scanning raw data on every refresh. A Kibana CRUD UI for views, including a "Save as View" workflow in Discover, is also planned.

Try it

Logical views are available as a Tech Preview. Try them in Kibana Dev Tools or Discover. We'd love your feedback; file a GitHub issue with the ES|QL label.

ES|QL logical views are a Tech Preview feature. Tech Preview features are subject to change and are not covered by the support SLA of GA features. The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Agent Builder in 9.4 makes context the agent's problem, not yours. Skills provide reusable instructions that load on demand, so only what the current task needs is in context. Large result sets go into a conversation context store rather than sitting in the prompt. For long-running tasks, context gets compacted selectively so the agent doesn't drift. Token and turn counts are monitored as the conversation runs. And connectors handle reaching enterprise data where it lives.

The goal of all of it is the same. Load the context you need, when you need it. Internally we've seen this cut token costs by up to 40%, and the agent's context stays reliable across datasets when it would have degraded before.

Getting agents to know what you know

Three context problems show up again and again: managing bloated prompts, executing complex actions, and controlling enterprise data sources.

Firstly, prompts get bloated because every instruction has to live in them. Skills fix that by loading on demand and with fewer input tokens. Skills provide structured descriptions of how an agent should operate and act in a specific task. Agent Builder ships with built-in skills for common data analysis patterns, but the real value is that users can also build custom skills. A security team can encode its triage playbook as a skill. An SRE team can describe how they want root cause analysis to proceed. A developer can encode their API design conventions and error handling patterns. Skills are reusable and shareable across agents, which means a pattern that works for one team's deployment doesn't have to be reinvented by the next.

In practice, this looks like: a team lead defines a "Summarize this incident" skill with the process they care about, the severity classification their org uses, and the output format their runbook expects. Anyone on the team invokes it by typing in the chat input and selecting it from autocomplete. Skills follow the Agent Skills open format, so you can pull them from a shared library, write your own, or craft skills using an agent of your choice.

In internal testing, we found that removing instructions from the agent prompt and placing them in dynamically loaded skills showed a 21 to 39% reduction in input token usage across test datasets. The key architectural improvement is that skills and their associated tools are loaded only when the agent needs them. All other skills stay as lightweight stubs with just a name and description, consuming almost no context.

Chat with your data and act on it too (dashboards, workflows, queries and more): Agentic tasks don't stay simple for long. Agent Builder now has contextual awareness of objects in Kibana. With agentic dashboard creation, a user can describe what they want to see in plain language, and the agent generates a dashboard with panels, visualizations, queries, and everything that's needed. Users can refine it conversationally: "break that out by region," "add a filter for the last 7 days," "swap the bar chart for a line chart."

Dashboards, alerts, and rules also work as inputs. Once a dashboard exists, it can be retrieved from the Agent’s context. This unlocks the "act" side of agents. Once a dashboard or alert is in context, the agent can modify it, extend it, or create new ones. The agent can reason about what the data shows, suggest follow-up analyses, or modify the dashboard based on what it sees. It's a feedback loop: the user describes intent, the agent produces a visual artifact, and both the user and the agent can reason over that artifact together.

For business analysts and operations teams, this collapses the gap between "I have a question about the data" and "I have a dashboard I can share with my team," reducing hours of manual work to a few minutes of conversation.

Lastly, using enterprise data for context creates governance you didn't ask for. Connectors close the loop for data that lives outside Elastic. We added prebuilt OAuth-based connectors for sources like Google Drive, Salesforce, and Slack. The design principle here is worth calling out: data stays at the source. The agent searches data via the connector with the user's own permissions enforced. Agents do not accumulate copies of enterprise data in new locations just to be able to answer questions about it.

This matters more than it might seem. Enterprise data governance isn't just a compliance checkbox; it's a load-bearing infrastructure that most teams don't notice until it fails. When agents start routing around it, accumulating copies in vector stores and context windows, you've quietly created a new class of data sprawl that your security team didn't sign off on, and your audit logs don't capture. The connector approach eliminates this risk by constraint: if the data never moves, it can't end up somewhere it shouldn't. The user's permissions travel with every query because the query goes to the source, not to a cached copy. You get agents that are genuinely useful on enterprise data.

Ensure agents don't outgrow the context window

Giving agents too much context creates a new problem. A security analyst investigating a complex threat might pull in dozens of alerts, correlate across multiple indices, and go back and forth with the agent for twenty or thirty turns. At some point, you're pushing past what the context window can hold and degrading the quality of the model's responses. The problem is that each retrieval call adds latency to the user's request and pushes infrastructure costs higher, and a single user interaction can trigger dozens of these calls.

We built a context store for retrieval results. As the agent retrieves data from indexes, the results can grow large and crowd the context window. We introduced a temporary store that holds the results of a query in an in-memory “file store” and only pulls the results into the active context when needed. This allows for conversations to extend and deal with multiple related data sets without blowing out the context. We are also optimizing the retrieval results themselves, applying top snippets retrieval, which demonstrated a 27 to 34% reduction in token usage.

We also added intelligent context compaction for longer interactions: As a conversation progresses, the agent manages what stays in the active context and what gets compressed into a summary that can be retrieved if needed. This isn't a simple truncation; it's selective compaction that preserves the information most likely to matter for the next turn.

This enables agents to handle larger result sets, more complex queries, and longer conversations without the token cost scaling linearly with every turn. With context compaction agents, the context window remains within a limit even for chats with 30 or more turns, rather than quickly ballooning to max size.

For teams running multi-step investigations or summarizations, this is the difference between an agent that stays coherent through turn thirty and one that starts contradicting itself at turn twelve.

Monitoring: In 9.4, we also shipped monitoring for agents to track token usage. With an API available to monitor conversation turns and tool calIs. This matters because agents aren't static. Their behavior shifts based on the context they receive and the tools they call, and without visibility into those patterns, optimizing cost and performance is guesswork.

Agentic consumption model

To support these new capabilities, we're introducing an agent pricing model that directly aligns the value users gain from their agents and how they scale. Agent Builder usage will be measured by Executions. Executions are free for the first 1,000 each month in Elasticsearch and 10,000 in Elastic Security and Observability projects.

An Agent Builder execution represents a completed round of interaction with the agen. In most cases, sending a chat message and receiving a successful response from the agent counts as one execution. For messages that demand significant processing, it will be calculated as multiple executions based on the total number of input tokens required, grouped into 50,000 input token units. For example, a deep investigative task that requires 130,000 input tokens will be billed as 3 executions. This model ensures your consumption aligns with the value your agents deliver and becomes more cost-effective as your agents achieve greater context efficiency.

Where are we going with agents

Agents that can optimize context over operational data need the same kind of careful context engineering that we've spent years applying to search relevance. Getting the right information in front of the model at the right time and at the right level of detail is the new retrieval problem. These capabilities are foundational towards enabling agents that are more reliable, scalable, and cost-efficient as they scale.

Get started with an Elastic Cloud Trial, and check out the documentation here. For existing customers, Agent Builder is available in Cloud Serverless and on the Enterprise Tier in Elastic Cloud Hosted and self-managed.

From architecture to implementation

Part 3 described the control plane architecture: reverse matching as a lookup primitive, policy documents that separate match from action, and cascading transformations that compose multiple policies into a single execution plan. This post goes hands-on with the Elasticsearch feature that powers the policy lookup: the percolator query.

The percolator is a natural fit for governance because it inverts the direction of search in exactly the way a control plane needs. This post walks through the implementation step by step, starting with a clear explanation of what the percolator does and why it matters, and then moving through index design, policy storage, query-time evaluation, and multi-policy composition.

How normal search works

In an ecommerce system, you may have hundreds of thousands or millions of product documents containing fields such as title, category, and price. When a user searches for matching documents, you're asking Elasticsearch to compare the user’s search string against one or more stored fields in these product documents. Elasticsearch's default analyzer, the standard analyzer, lowercases text and splits it into tokens. A search for “oranges” matches “Oranges” because of lowercasing. With a language-aware analyzer that includes stemming, it also matches “orange” because both forms reduce to the same stem. For example, the following match query returns documents that have “orange” or “oranges” in their “title” field.

POST products/_search
{
  "query": {
    "match": {
      "title": "oranges"
    }
  }
}

So for the above query, Elasticsearch returns the product documents whose title field matches “oranges”, which could include results such as “Orange Fruit Spread”, “Orange Juice”, “Juicy oranges”, “Orange Marmalade”, and so on. The key point to remember is that Elasticsearch is commonly used to compare a search string against documents and to return the documents that match the search string.

The governance problem: Finding relevant policies before searching for products

As established in Parts 1 through 3, a governed search system does not send the user's search string directly to the product catalog. First, it checks whether any policies apply to that search string.

A merchandiser has decided that when someone searches for exactly "oranges", results should be restricted to the Oranges category, eliminating orange juice, orange marmalade, and orange soda. That business decision is stored as a policy. When a user types "oranges", the control plane needs to find that policy, read its instructions, and modify the search against the product catalog accordingly. In order to do this, the control plane needs to figure out which stored policies are relevant for this search string.

An enterprise deployment might have hundreds or thousands such policies. Checking them one by one with if/else logic is the application-layer anti-pattern described in Part 2. What we need is a way to store all of those policies in an index and instantly find the ones that match a given search string. This is where the percolator comes in.

Flipping the direction: The percolator

We previously mentioned that in a normal search, Elasticsearch is commonly used to compare a search string against documents and to return the documents that contain that search string.

The percolator inverts this. With a percolator, you have an index where each document stores a query pattern, and then an incoming search string is checked against these stored queries to determine which of these stored query patterns has triggered.

For governance, the "stored query patterns" are policies. Each policy contains a pattern that describes the kind of search string it should match. For example, does the search string exactly match “oranges”, or does the search string contain “olive oil”? The incoming string is the user's search text, which arrives at query time and needs to be checked against all stored policy patterns. This is covered in a related PRISM video at 4:09.

Step by step: How a search for "oranges" finds its policy

The policy

A merchandiser has authored a policy that matches if a user searches for exactly "oranges" without any other words. Once the percolator matches, the remainder of the document includes the rules that the control plane will use to build the Product query; in this example, one of the rules is to restrict (filter) results to the Fruits category.

{
  "percolator": {
    "match_phrase": { "query": "START oranges END" }
  },
  "rule_type": "filter",
  "rule_args": {
    "filters": [
      {
        "field": "categories",
        "values": ["Fruits"],
        "mode": "hard_filter",
        "on_conflict": "soft_boost",
        "on_conflict_boost_weight": 1.0
      }
    ]
  },
  "priority": 0,
  "enabled": true
}

The percolator field contains the pattern that defines when this policy should fire. In this case, it matches the phrase "START oranges END". The rule_type and rule_args fields define what the policy should do when it fires. The START and END tokens are boundary markers, which we will explain shortly.

You can see how a policy is authored in the PRISM Studio UI at 2:52 of the related PRISM video.

The user searches

A shopper types "oranges" into the search bar.

The control plane checks for matching policies

Before searching the product catalog, the control plane intercepts the user search string, wraps it in boundary markers, and sends it to the percolator:

POST policies/_search
{
  "query": {
    "percolate": {
      "field": "percolator",
      "document": {
        "query": "START oranges END"
      }
    }
  }
}

The string "START oranges END" is checked against all stored policy patterns. Internally, Elasticsearch runs the stored policy patterns against this string and returns the ones that match. That's the percolator. The user's search string was checked against all stored policy patterns, and the ones that matched were returned. No if/else chains. No sequential evaluation. The index handles the matching.

The control plane applies the policy

The control plane reads the matched policies’ actions. The above policy instructs the control plane to restrict results to the Fruits category. The control plane builds the final Elasticsearch query against the product catalog as follows:

POST products/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "title": "oranges" } }
      ],
      "filter": [
        { "terms": { "categories": ["Fruits"] } }
      ]
    }
  }
}

The user searched for "oranges”. The product catalog receives a query for "oranges" constrained to the Fruits category. Because of this constraint, orange juice, orange marmalade, and orange soda are excluded.

Why "orange marmalade" does NOT trigger the oranges policy

Suppose a different user searches for "orange marmalade”. The control plane wraps the string and percolates: "START orange marmalade END". The oranges policy's pattern is match_phrase: "START oranges END". The oranges policy does not match and therefore the policy isn’t applied, and the results aren’t constrained to the Fruits category.

This is the purpose of the START and END boundary markers. Without them, a policy that matches on the word "oranges" could accidentally fire on a query like "orange marmalade". By wrapping the user's search string with START and END and including those markers in the policy's pattern, we ensure that the policy only fires when "oranges" is the complete search string, without any other words. This matches both the shoppers and the merchandiser's intent.

A second policy: "olive oil" on the stemmed field

Not every policy needs an exact string match. The “olive oil” policy matches on a stemmed field, so it fires regardless of minor word-form variations:

{
  "percolator": {
    "bool": {
      "should": [
        { "match_phrase": { "query.stemmed": "START olive oil END" } }
      ]
    }
  },
  "rule_type": "filter",
  "rule_args": {
    "filters": [
      {
        "field": "categories",
        "values": ["Olive oils"],
        "mode": "hard_filter",
        "on_conflict": "soft_boost",
        "on_conflict_boost_weight": 1.0
      }
    ]
  },
  "priority": 300,
  "enabled": true
}

This policy's pattern matches against query.stemmed instead of query. When the user's search string arrives, it’s stored in both a query field (the exact text) and a query.stemmed field (analyzed with a stemming analyzer that reduces words to their stems, so "olives" and "olive" both reduce to the same stem, as do "oils" and "oil"). The policy's pattern is checked against the stemmed version of the string, so it fires regardless of minor word-form variations.

The START and END boundary markers work on the stemmed field, as well, ensuring this policy only fires when "olive oil" is the entire search string, not when it appears as part of something longer.

The rest of this post covers the implementation details that make this production-ready: the index mapping that supports both matching modes, how highlights drive phrase removal and consumed phrase tracking, and how multiple conflicting policies compose into a single execution plan.

The policy index mapping

The policy index needs a percolator field to hold stored query patterns and a text field that mirrors the structure of the incoming search string the percolator will match against. The mapping below is simplified for clarity. A production deployment is more complex, using custom analyzers to handle boundary markers, variable pattern matching (for example, recognizing that "under $4" contains a currency value), and other kinds of analysis.

PUT policies
{
  "mappings": {
    "properties": {
      "percolator": {
        "type": "percolator"
      },
      "query": {
        "type": "text",
        "fields": {
          "stemmed": {
            "type": "text",
            "analyzer": "stemming"
          }
        }
      },
      "rule_type": { "type": "keyword" },
      "rule_args": { "type": "object", "enabled": false },
      "priority": { "type": "integer" },
      "enabled": { "type": "boolean" }
    }
  }
}

The index is named policies because each document represents a complete governed policy as defined in Part 2. This includes match criteria, action, priority, and metadata. The rule_type and rule_args fields contain the action component of the policy, which contain the instructions that the control plane will use to compose the query for execution against the product catalog.

The query field is the string that the percolator matches against. It has two variants: an exact version and a stemmed version. When the user's search string arrives, it’s placed into this field in the temporary in-memory index. Policies that match on query see the exact string; policies that match on query.stemmed see the stemmed version.

Percolating with highlights, filtering, and sorting

The simple examples above showed minimal percolation requests. In practice, the control plane adds highlighting, filters disabled policies, and sorts by priority:

POST policies/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "percolate": {
            "field": "percolator",
            "document": {
              "query": "START olive oil END"
            }
          }
        },
        {
          "term": { "enabled": true }
        }
      ]
    }
  },
  "highlight": {
    "fields": {
      "query": {
        "matched_fields": ["query.stemmed"]
      }
    }
  },
  "sort": [
    { "priority": { "order": "desc" } }
  ]
}

The highlight configuration uses "query" as the field key with "query.stemmed" in matched_fields. This tells Elasticsearch's unified highlighter to return highlights on the parent query field but to also consider matches from the query.stemmed subfield when determining which tokens to highlight. This is what allows a policy that matches on the stemmed field to still produce accurate highlight spans on the original text, which the control plane needs for phrase removal and consumed phrase tracking.

The enabled: true filter ensures that disabled policies are skipped. The sort on priority ensures that higher-priority policies are returned first, so the control plane can process them in the correct order for cascading transformations. The highlight field is the most important addition; it tells us exactly which words in the user's search string triggered each match.

The response for an "olive oil" search may look as follows:

{
  "hits": {
    "hits": [
      {
        "_id": "en_2c3021c8",
        "_source": {
          "rule_type": "filter",
          "rule_args": {
            "filters": [
              {
                "field": "categories",
                "values": ["Olive oils"],
                "mode": "hard_filter",
                "on_conflict": "soft_boost",
                "on_conflict_boost_weight": 1.0
              }
            ]
          },
          "priority": 300
        },
        "highlight": {
          "query": ["START olive oil END"]
        }
      }
    ]
  }
}

Why highlights matter

Notice the highlight in the response: "<em>START olive oil END</em>". Elasticsearch is telling us exactly which words in the user's search string caused the policy to match. This isn’t cosmetic. The highlight metadata drives two critical downstream behaviors:

Phrase removal. Some policies need to remove the matched text from the search string before constructing the product catalog query. For example, a policy that matches on "cheap" removes that word and converts it into a price filter instead. The highlight identifies exactly which span of the search string the policy matched, so the system knows what to remove.

Consumed phrase tracking. As described in Part 3, when multiple policies match the same search string, a higher-priority policy might remove words that a lower-priority policy also matched on. By comparing each policy's highlight against the current (evolving) search string, the system can detect that a phrase has been consumed and skip the lower-priority policy. This prevents double-processing and ensures deterministic behavior.

You can learn more about how highlighting works in this article.

From percolation to execution plan

The percolator returns a set of matching policies. But as Part 3 described, the lookup is only half the story. The other half is composing those matches into a coherent execution plan. Here’s what that looks like for a concrete query.

Worked example: "Cheap chocolate" during a Christmas campaign

Suppose the system has two active policies: the "Cheap chocolate" policy (priority 210) and the "Christmas chocolates" policy (priority 300), both described in detail in Part 3.

Step 1: Percolate. The user searches for "cheap chocolate." The control plane wraps the search string as "START cheap chocolate END" and sends it to the percolator. Two policies match: The "Cheap chocolate" policy's pattern matches on the phrase "cheap chocolate"; and the "Christmas chocolates" policy's pattern matches on "chocolate" via the stemmed field.

Step 2: Sort by priority. The percolator returns both policies, sorted by priority in descending order. The “Christmas chocolates” policy (300) is processed first, followed by the “Cheap chocolate” policy (210).

Step 3: Apply the cascading transformation. This is the initial state → [Policy A] → state' → [Policy B] → state'' → execution plan model from Part 3.

The “Christmas chocolates” policy (priority 300) applies first:

• Adds a category hard filter: "Christmas foods and drinks," "Christmas sweets".
• Adds a price filter: less than $7.
• Adds a category soft boost: "Advent calendars" (3x).

The “Cheap chocolate” policy (priority 210) applies next against the modified state:

• Attempts to add a category hard filter: "Chocolates," "Milk chocolates"; but the Christmas policy already set this field with on_conflict: override, so the Cheap chocolate categories are dropped.
• Attempts to add a price filter: $2, the Christmas policy set on_conflict: restrict for price, and $2 is more restrictive than $7, so $2 wins.
• Removes "cheap" from the search string.

Step 4: Build the Elasticsearch query. The control plane assembles the execution plan into a single Elasticsearch query against the product catalog:

POST products/_search
{
  "query": {
    "function_score": {
      "query": {
        "bool": {
          "must": [
            { "match": { "title": "chocolate" } }
          ],
          "filter": [
            { "terms": { "categories": ["Christmas foods and drinks", "Christmas sweets"] } },
            { "range": { "price": { "lt": 2 } } }
          ]
        }
      },
      "functions": [
        {
          "weight": 1
        },
        {
          "filter": { "terms": { "categories": ["Advent calendars"] } },
          "weight": 3
        }
      ],
      "score_mode": "sum",
      "boost_mode": "multiply"
    }
  }
}

The original search string was "cheap chocolate”. The query that reaches the product catalog is a governed, intent-aware retrieval plan: The word "cheap" has been consumed and converted into a price constraint, results are restricted to Christmas seasonal categories, Advent calendar products receive a ranking boost, and the price ceiling reflects the more restrictive value from the lower-priority policy. Every transformation is deterministic, traceable, and explainable.

For a quick overview about how these multipliers interact with the base BM25 score, see 8:45 in the related PRISM video, where we briefly discuss multiplicative boosts.

Why this scales

The percolator is efficient for this use case because of the asymmetry: An enterprise ecommerce system might have millions of products but only hundreds or thousands of governance policies. The percolator is checking one incoming search string against that set of stored policy patterns, not scanning the full product catalog. The cost is proportional to the number of policies, and Elasticsearch applies internal optimizations (indexing terms from stored query patterns, short-circuiting Boolean logic) to keep matching fast.

Adding a new policy is just indexing a new document. Disabling one is a field update. No code changes, no deploys, no restarts.

From lookup to governed retrieval

The percolator provides the fast reverse-matching primitive that makes the control plane architecture from Part 3 practical at scale. Policies are data which are stored and indexed, and efficiently matched against incoming search strings. The control plane composes matching policies into a governed execution plan through the cascading transformation and per-field conflict resolution described in Part 3. And the retrieval engine executes the governed execution plan against the product catalog.

The result is a system where a merchandiser can author a new policy without touching application code, test it against representative queries, promote it to production, and immediately see the effect. The percolator makes the policy lookup fast; the control plane makes the policy composition deterministic; and the governed workflow makes the whole process safe.

What's next in this series

The next post in this series extends the governed control plane into new territory. It introduces a multi-tier search architecture, explaining how to orchestrate strict, relaxed, and semantic retrieval while maintaining stable pagination and facets.

Put governed ecommerce search into practice

The percolator-based control plane described in this post, from index mappings and boundary markers to highlight-driven phrase tracking and cascading policy composition, was built by Elastic Services Engineering as part of our repeatable ecommerce search accelerators. Every query example and policy structure shown here comes from a working system validated against enterprise-scale product catalogs.

If you want to implement a governed, policy-driven control plane on Elasticsearch, Elastic Services can get you there faster. Contact Elastic Professional Services.

Join the discussion

Have questions about search governance, retrieval strategies, or ecommerce search architecture? Join the broader Elastic community conversation.

What is recall?

Recall measures on a scale from 0 to 1 how many of the documents that your users actually want appear somewhere in your search results. If a query should surface three products and your search returns only two of them in the top 10, recall@10 = 0.67 for that query. It’s a set-based metric: It doesn’t care about the position of the relevant documents within those k results. A relevant document in position 10 counts the same as one in position 1. Having a high recall means that you’re not losing relevant results.

The diagram shows two sets: all relevant documents (left) and what BM25 actually retrieved (top 10, right). Only the intersection counts toward recall, prod_1 and prod_2 were found, while prod_3, prod_4, and prod_6 were missed entirely. Result: Recall@10 = 2/5 = 0.40.

Prerequisites

Let's get down to business to better understand how recall works. This demonstration uses Python. You can follow along with it on the companion notebook (notebook.ipynb), where every code block is a cell ready to run.

The code provided uses the following:

• Elasticsearch 9.3+
• Python 3.10+

pip install elasticsearch pandas plotly python-dotenv

• A .env file with your Elasticsearch credentials

ELASTICSEARCH_URL=https://your-cluster-url
ELASTICSEARCH_API_KEY=your-api-key

The dataset

We’ll use a product catalog of 1,000 products, spanning categories such as footwear, electronics, tools, and more.

Each document has four fields:

The dataset is loaded from dataset.csv.

The power and limits of lexical search

BM25 is the default ranking algorithm in Elasticsearch and most search engines. It scores documents by how often your query terms appear in them, adjusted for document length and the frequency of those terms across the entire index. You get analyzers on top: lowercase normalization, stemming, and stopword removal. A query for "running shoes" will match "Running Shoes" and likely "run" as well.

This works well for a large class of queries:

• "running shoes" immediately matches products with those exact tokens in the title.
• "bluetooth speaker" surfaces portable audio products because the tokens appear verbatim.

The results are deterministic and explainable: A document ranks highly because the query terms appear in it. Debugging relevance is straightforward.

Where it breaks

Now let’s try these queries against the same catalog:

• "skincare routine": The word "routine" doesn’t appear in any product title. BM25 can partially match on "skincare," but face serums, body oils, and moisturizers are described using terms like "vitamin C," "retinol," or "brightening," none of which overlap with the query. Products that form a complete skincare routine are scattered across the index with no shared token to anchor them.

ID: B06XX6DS3P, Score: 9.0552, Title: Replenix Retinol Smooth + Tighten Body Lotion - Collagen-Boosting, Regenerating Anti-Aging Body Cream, Reduces Appearance of Stretch Marks, 6.7 oz.

  ID: B08XMPKJ1L, Score: 5.2699, Title: Bio-Oil Skincare Body Oil (Natural) Serum for Scars and Stretchmarks, Face and Body Moisturizer Hydrates Skin, with Organic Jojoba Oil and Vitamin E, For All Skin Types, 6.7 oz

  ID: B01CY764KQ, Score: 5.0057, Title: Nike Up Or Down Men Deodorant - Pack of 2 | Long-Lasting Fragrance, Body Spray Combo for Men | Deodorant for Active Living | Nike Men's Deo Set | Ultimate Odor Protection | Grooming Essentials | Signature Nike Scent | High-Performance Men's Deodorant

• "pet travel accessories": This is a use-case grouping, not a product category. A dog sling carrier, a pet car seat, and a travel crate are all relevant, but their descriptions talk about portability, safety, and comfort rather than "travel accessories." BM25 matches "pet" broadly but has no signal to distinguish travel-specific products from the rest of the pet catalog.

ID: B0BVV7BKTW, Score: 7.4371, Title: Large Foldable Travel Duffel Bag with Shoes Compartment

ID: B07TNPHYNV, Score: 6.6455, Title: 40 Pieces Christmas Bronze Jingle Bells Craft Small Bells

ID: B08R8FRW53, Score: 6.6335, Title: CUBY Dog and Cat Sling Carrier
ID: B08QMCQYGM, Score: 6.5259, Title: YTFGGY Whiteboard Pinstripe Tape 6 Rolls 1/8"
ID: B0CP3LQSWM, Score: 6.2994, Title: Portable Dog Water Bottle 32 Oz

This is a recall problem. The relevant documents exist in your index. BM25 just cannot find them because the user's words and the document's words do not match closely enough.

Adding synonyms helps for known cases. But you cannot enumerate every way a user might express an intent. That is where vectors come in.

Why you should measure recall

Before fixing a problem, you need to quantify it.

Recall@k measures how many of the documents that your users actually want appear somewhere in your search results. Formally:

Recall@k = (relevant documents found in top k) / (total relevant documents)

Precision@k measures the top k results and how many are actually relevant:

Precision@k = (relevant documents in top k) / k

High precision means that the results you do return are good. In ecommerce, missing a relevant product (low recall) is often worse than showing a slightly imperfect result (lower precision), because a hidden product is a lost sale.

Elasticsearch's rank_eval API lets you measure both systematically. You provide a list of queries, each with a set of rated documents, and Elasticsearch computes the metrics for you across all queries.

Setting up the evaluation

The rank_eval API needs a ratings dataset: a mapping of queries to the documents that are relevant for each one, along with a relevance grade (0 = not relevant, 1 = relevant, 2 = highly relevant).

In the notebook, this is the judgments list:

judgments = [
    # Query 1: "running shoes" BM25 handles well (tokens appear in product titles) 
    {"query_id": "q1", "doc_id": "B09NQJFRW6", "grade": 2, "query": "running shoes"},
    {"query_id": "q1", "doc_id": "B08JMD4LMM", "grade": 2, "query": "running shoes"},
    {"query_id": "q1", "doc_id": "B08VRJ6F2Q", "grade": 2, "query": "running shoes"},
    {"query_id": "q1", "doc_id": "B07S8NRRWR", "grade": 2, "query": "running shoes"},
    {"query_id": "q1", "doc_id": "B01HD620I8", "grade": 2, "query": "running shoes"},
    {"query_id": "q1", "doc_id": "B07DX86321", "grade": 2, "query": "running shoes"},
    {"query_id": "q1", "doc_id": "B0968YVLQ8", "grade": 1, "query": "running shoes"},
    {"query_id": "q1", "doc_id": "B093QJ39ZS", "grade": 1, "query": "running shoes"},
    {"query_id": "q1", "doc_id": "B096FGSC39", "grade": 1, "query": "running shoes"},
    {"query_id": "q1", "doc_id": "B01GVQWVV2", "grade": 1, "query": "running shoes"},

    # Query 2: "skincare routine" intent-based, "routine" never appears in product titles
    {"query_id": "q2", "doc_id": "B08XMPKJ1L", "grade": 2, "query": "skincare routine"},
    {"query_id": "q2", "doc_id": "B0BN3WQB92", "grade": 2, "query": "skincare routine"},
    {"query_id": "q2", "doc_id": "B0BT7B7P5T", "grade": 2, "query": "skincare routine"},
    {"query_id": "q2", "doc_id": "B00NPA2WEY", "grade": 2, "query": "skincare routine"},
    {"query_id": "q2", "doc_id": "B06XX6DS3P", "grade": 1, "query": "skincare routine"},
    {"query_id": "q2", "doc_id": "B07PDRD1KT", "grade": 1, "query": "skincare routine"},
    {"query_id": "q2", "doc_id": "B074J7869B", "grade": 1, "query": "skincare routine"},
    {"query_id": "q2", "doc_id": "B08JV31QW4", "grade": 1, "query": "skincare routine"},
    {"query_id": "q2", "doc_id": "B00K3TVJMQ", "grade": 1, "query": "skincare routine"},

    # Query 3: "study desk setup" intent-based, products are desks/stands/organizers
    {"query_id": "q3", "doc_id": "B08CS35J2T", "grade": 2, "query": "study desk setup"},
    {"query_id": "q3", "doc_id": "B09B3LFDXJ", "grade": 2, "query": "study desk setup"},
    {"query_id": "q3", "doc_id": "B07W58LMND", "grade": 1, "query": "study desk setup"},
    {"query_id": "q3", "doc_id": "B0CHYDX91L", "grade": 1, "query": "study desk setup"},

    # Query 4: "pet travel accessories" use-case grouping, products are carriers/crates/seats
    {"query_id": "q4", "doc_id": "B08R8FRW53", "grade": 2, "query": "pet travel accessories"},
    {"query_id": "q4", "doc_id": "B01MYUYX33", "grade": 2, "query": "pet travel accessories"},
    {"query_id": "q4", "doc_id": "B003C5RKE4", "grade": 2, "query": "pet travel accessories"},
    {"query_id": "q4", "doc_id": "B09GF8GBF6", "grade": 1, "query": "pet travel accessories"},
    {"query_id": "q4", "doc_id": "B0CP3LQSWM", "grade": 1, "query": "pet travel accessories"},
]

The mix is intentional: q1 is a query that BM25 handles well (exact tokens in product titles), while q2, q3, and q4 are intent-based queries where the user's intent is expressed as a concept rather than specific product keywords.

Measuring BM25 baseline recall

First, set up the Elasticsearch client and index the raw text data:

import os
import json
import pandas as pd
import plotly.graph_objects as go
from elasticsearch import Elasticsearch, helpers
from dotenv import load_dotenv

load_dotenv()

es = Elasticsearch(
    os.getenv("ELASTICSEARCH_URL"),
    api_key=os.getenv("ELASTICSEARCH_API_KEY")
)

INDEX_NAME = "ecommerce-products"

Now build the rank_eval request for BM25. Each request in the list combines a query with its ratings:

judgments_df = pd.DataFrame(judgments)

bm25_requests = []
for query_id, query_text in (
    judgments_df[["query_id", "query"]].drop_duplicates().values
):
    relevant_docs = judgments_df[judgments_df["query_id"] == query_id]
    ratings = [
        {"_index": INDEX_NAME, "_id": row["doc_id"], "rating": row["grade"]}
        for _, row in relevant_docs.iterrows()
    ]

    bm25_requests.append({
        "id": query_id,
        "request": {
            "query": {
                "multi_match": {
                    "query": query_text,
                    "fields": ["title", "description"]
                }
            }
        },
        "ratings": ratings,
    })

bm25_eval = {
    "requests": bm25_requests,
    "metric": {"recall": {"k": 10, "relevant_rating_threshold": 1}},
}

bm25_result = es.rank_eval(index=INDEX_NAME, body=bm25_eval)
print("BM25 Recall@10:", bm25_result.body["metric_score"])

Result:

BM25 Recall@10: 0.43

0.43 means that across all four queries, BM25 finds only 43% of the documents it should find. The shortfall is concentrated in the intent-based queries: "skincare routine" misses face serums and body oils because "routine" never appears in product titles, and "pet travel accessories" retrieves off-topic pet products while missing carriers and crates described in terms of portability and safety rather than "travel accessories."

This is our baseline. Now we have a number to beat.

Adding vector search with Jina embeddings

Vector search encodes documents and queries as high-dimensional vectors, a type of vector made up of hundreds or thousands of numerical values, each encoding a specific feature of the data it represents. Documents with similar meaning end up close together in vector space, even if they share no words. "Gym equipment" and "dumbbell set" will be nearby because the concepts are related. I chose Elasticsearch as my vector database because it supports hybrid search, giving me both semantic understanding and keyword precision out of the box.

EIS includes out-of-the-box support for embedding models through its inference API.

Step 1: Using Jina embeddings v5 as an inference endpoint

INFERENCE_ENDPOINT_ID = ".jina-embeddings-v5-text-small"

If your cluster has GPU resources (available in Elastic Cloud and Elasticsearch 9.3+), the embeddings are generated on GPU, which is significantly faster than CPU inference and removes the performance trade-off that historically made vectors expensive at scale.

Why Jina embeddings specifically? jina-embeddings-v5-text is a multilingual model (119+ languages) with a 32,000-token context window and support for task-specific Low-Rank Adaptation (LoRA) adapters. It works well for short product descriptions out of the box. Read more about jina-embeddings-v5-text model here.

Step 2: Create the index with a semantic field

index_mappings = {
    "mappings": {
        "properties": {
            "title": {"type": "text", "copy_to": "semantic_field"},
            "description": {"type": "text", "copy_to": "semantic_field"},
            "brand": {"type": "keyword"},
            "category": {"type": "keyword"},
            "semantic_field": {
                "type": "semantic_text",
                "inference_id": INFERENCE_ENDPOINT_ID,
            },
        }
    }
}

if not es.indices.exists(index=INDEX_NAME):
    es.indices.create(index=INDEX_NAME, body=index_mappings)
    print(f"Created index: {INDEX_NAME}")

The semantic_text field type is the key here. It’s a higher-level abstraction over dense_vector: You point it at an inference endpoint, and Elasticsearch takes care of generating embeddings automatically.

The copy_to property on title and description means content from both fields flows into semantic_field for embedding, so a single vector captures the full product representation.

Step 3: Index the products

def bulk_index(products, index_name):
    actions = []
    for product in products:
        doc_id = product.get("_id")
        source = {k: v for k, v in product.items() if k != "_id"}
        action = {"_index": index_name, "_source": source}
        if doc_id:
            action["_id"] = doc_id
        actions.append(action)

    success, failed = helpers.bulk(es, actions, raise_on_error=False)
    if failed:
        for error in failed:
            print(f"Error: {error}")
    else:
        print(f"Successfully indexed {success} documents")

bulk_index(products, INDEX_NAME)

At index time, Elasticsearch calls the inference endpoint for each document and stores the resulting embedding in semantic_field. No extra code on your side.

Hybrid search: Combining BM25 and vectors with RRF

Adding vectors improves recall, but using vectors alone risks losing precision on exact-match queries; "running shoes" should still rank verbatim matches first. Hybrid search retains the lexical component specifically to preserve that precision.

Hybrid search with Reciprocal Rank Fusion (RRF) keeps the best of both:

• BM25 handles exact and near-exact queries with high precision.
• Semantic search handles intent-based and multilingual queries with high recall.
• RRF combines the two ranked lists into a single ranking.

The RRF formula assigns each document a score based on its rank in each result list:

score = sum(1 / (rank_constant + rank))

A document that ranks highly in both lists gets a higher combined score. The rank_constant controls how much weight lower-ranked documents receive.

hybrid_requests = []

for query_id, query_text in (
    judgments_df[["query_id", "query"]].drop_duplicates().values
):
    relevant_docs = judgments_df[judgments_df["query_id"] == query_id]
    ratings = [
        {"_index": INDEX_NAME, "_id": row["doc_id"], "rating": row["grade"]}
        for _, row in relevant_docs.iterrows()
    ]

    hybrid_requests.append({
        "id": query_id,
        "request": {
            "retriever": {
                "rrf": {
                    "retrievers": [
                        {
                            "standard": {
                                "query": {
                                    "multi_match": {
                                        "query": query_text,
                                        "fields": ["title", "description"],
                                    }
                                }
                            }
                        },
                        {
                            "standard": {
                                "query": {
                                    "match": {
                                        "semantic_field": {"query": query_text}
                                    }
                                }
                            }
                        },
                    ],
                    "rank_window_size": 50,
                    "rank_constant": 5,
                }
            }
        },
        "ratings": ratings,
    })

hybrid_eval = {
    "requests": hybrid_requests,
    "metric": {"recall": {"k": 10, "relevant_rating_threshold": 1}},
}

hybrid_result = es.rank_eval(index=INDEX_NAME, body=hybrid_eval)
print("Hybrid Recall@10:", hybrid_result.body["metric_score"])

Result:

Hybrid Recall@10: 0.75

Hybrid improves substantially over BM25 (0.43) and preserves precision for exact-match queries like "running shoes."

Results: Before and after

Here’s the full comparison across all three approaches:

methods = {
    "BM25 (Lexical)": bm25_requests,
    "Hybrid (BM25 + Vectors)": hybrid_requests,
}

recall_metric = {"recall": {"k": 10, "relevant_rating_threshold": 1}}

comparison_data = []
for method_name, requests in methods.items():
    result = es.rank_eval(
        index=INDEX_NAME,
        body={"requests": requests, "metric": recall_metric}
    )
    comparison_data.append({
        "method": method_name,
        "recall@10": result.body["metric_score"]
    })

comparison_df = pd.DataFrame(comparison_data)
print(comparison_df.to_string(index=False))

Result:

Breaking it down by query:

Conclusion

Throughout this post, we saw that BM25 lexical search is reliable when users type exact queries, but it loses recall when they search by intent rather than keywords. Using rank_eval, we established a reproducible baseline to measure that gap with real numbers. From there, we added a semantic_text field powered by Jina embeddings and ran the evaluation again. The result: Hybrid search improved recall from 0.43 to 0.75 while preserving precision on exact-match queries, though the actual margin will depend on your query mix.

The pattern scales beyond this example: Collect judgments from your users' actual queries, run rank_eval as a baseline, add semantic_text, and measure again. You'll know exactly what improved and by how much.

Next steps

• Dive deeper into recall and vector search: Recall and vector search quantization by Jeff Vestal
• Add reranking for even better precision on the top results
• Explore Elasticsearch hybrid search documentation
• Read more about the rank_eval API

Why query interpretation is often a challenge

Storing policies as code (if/else blocks in the application layer) produces tens of thousands of lines of brittle logic that lacks any indexing for efficient policy retrieval at query time. Iteration is slow (a single query behavior change may require a six-week deployment cycle), accountability is unclear (why did results change?), and business users cannot modify search behavior without engineering involvement. This is shown on the left side in the following image:

Storing policies as data in an Elasticsearch index is shown on the right side of the above image. This approach solves all of the issues associated with hard-coded query resolution logic. However, for this to work, you need a way to quickly determine which policies match the user query and how conflicts should be resolved. This is where the governed control plane comes in.

The control plane pattern

A governed control plane sits between the raw user query and an Elasticsearch retrieval. It receives user text as its input, and its output is an execution plan that includes filters, boosts, and retrieval routing decisions.

A control plane pipeline consists of:

1. User query: A user enters a string of what they’re looking for, such as “oranges” or “gift for grandpa”.
2. Policy lookup: Match the user query against the policy index.
3. Return matching policies: Policies that match the user query are returned from the policy index.
4. Policy application: The control plane analyzes these returned policies and composes matched policies into a single coherent execution plan that includes filters, boosts, overrides, and guardrails and that applies the appropriate retrieval method (for example, lexical versus semantic versus hybrid).
5. Execute: The modified intent-aware Elasticsearch query is passed to the application to be executed against a product catalog index.
6. Explain (optional): In addition to creating a query that provides business and intent-aligned results, the control plane provides an optional explainability payload to show which policies were triggered and how they were combined.

Finding which policies should be applied for a user’s search string requires a fast reverse-matching primitive, which we solve with the percolator query. After retrieving relevant policies, combining multiple matched policies into a unified execution plan requires a judgment framework: priorities, conflict strategies, consumed phrase tracking, and cascading transformations that apply policies in sequence rather than independently. Additionally, the most appropriate retrieval technology needs to be selected (for example, BM25 for “oranges” versus semantic search for “gift for grandpa”).

Policy lookup: Checking the query before searching for products

When a shopper types a query, a search system with a governed control plane doesn’t send that query directly to be executed against the product catalog. First, the query is checked against a set of stored policies and modified to reflect the intent of the query and business priorities.

Policy structure

Each policy is a simple document that defines two things:

• Match criteria: What query text should cause this policy to fire. This could be an exact phrase, a single word, a pattern, or a combination.
• Action: What to do when the policy fires. This could be applying a category filter, excluding products, extracting a price constraint, or changing the retrieval strategy.

The system finds all matching policies, composes them into an execution plan, and only then runs the product search. Taken together, policies act like a knowledgeable store associate who understands what you’re looking for and walks you to the right aisle.

The policy pattern

The first articles in this series introduced examples of policies in action: constraining "oranges" to the produce category, treating "without peanuts" as an exclusion, and routing "gift for grandpa" to semantic retrieval. The key architectural point is that in each case, the query is checked against stored policies before the product search begins. The policies determine what constraints to apply, which text to modify, and which retrieval strategy to use. The query against the product catalog comes after the policies have been applied and a new rewritten query has been created.

Why this is fast

An enterprise ecommerce system might have millions of products but only hundreds or thousands of policies. The policy lookup step is searching against a small curated index, not the full product catalog, and is therefore fast. And because policies are stored as data in their own index, a merchandiser adding a new policy doesn't touch the application code, and an engineer optimizing the product search doesn't touch the policy index. The two concerns evolve independently.

The examples above describe what happens conceptually. Under the hood, the policy lookup is implemented using the Elasticsearch percolator query type, which is purpose-built for this kind of pattern: matching incoming text against a set of stored queries. Part 4 in this series provides a hands-on deep dive into the percolator implementation, including index mappings, boundary markers, and highlight-driven phrase tracking. With the lookup mechanism covered in depth in Part 4, let's turn to what a policy document actually contains and how the control plane composes multiple policies into a single execution plan.

Example policies

Now that we've seen what policies do conceptually, let's look at what they actually contain. The two policies below have been designed to intentionally conflict, which will demonstrate the conflict resolution system described in subsequent sections.

Cheap chocolate

The policy shown below detects if a user has submitted a search containing the phrase “cheap chocolate”. If so, results are restricted to the “Chocolates” and “Milk chocolates” categories. This policy also applies a price filter of $2. Also, notice that this policy has a priority of 210; we’ll come back to this when we discuss conflict resolution in more detail.

The filter mode and conflict strategy settings shown here (hard_filter, soft_boost, restrict, override) are explained in detail in the conflict resolution section below.

When the above policy is activated, a search for “cheap chocolate” respects the price filter of $2 and restricts results to the “Chocolates” and “Milk chocolates” categories. Example results are shown below:

Christmas chocolate

The policy shown below is an example of a policy that one could imagine applying at Christmas. This example restricts results to “Christmas foods and drinks” and “Christmas sweets”, boosts any products that are also in the “Advent calendars” category, and applies a price filter of less than $7 to focus on affordable seasonal items. Additionally, notice that this policy has a priority of 300. We’ll come back to this when we discuss conflict resolution in more detail.

When the above policy is activated without any conflicting policies, a search for “chocolate” respects the price filter of $7, and restricts results to the “Christmas food and drinks” and “Christmas sweets” categories, and boosts any products tagged as “Advent calendars”. Example results are shown below:

Combining matched policies

The policy lookup described above is half the story. The other half is what happens when multiple policies match the same query.

In any nontrivial deployment, a single query will routinely trigger several policies at once. "Cheap chocolate" will match both of the policies that we demonstrated above. Each policy is correct in isolation. The challenge is composing them into a single, coherent execution plan without contradictions, without double-counting, and without one policy silently undoing the work of another.

This isn’t a lookup problem; it’s a judgment problem. The system must decide:

• Order of application: If a negation policy removes "without peanuts" from the query, does the price policy still see the original text or the modified text?
• Filter conflicts: If two policies set different price ceilings, which one wins? Is the loser silently dropped, or does it degrade gracefully into a soft boost?
• Phrase ownership: If two policies both matched on the same word and the first one already consumed it, should the second one still fire?

A naive implementation (apply all matched policies independently, merge the results) breaks as soon as policies interact. The architecture needs an explicit model for how policies compose. The next two sections describe that model: a priority and conflict resolution framework; and a cascading transformation model that makes policy interaction deterministic.

The key insight is that policy application isn’t a set of independent operations; it’s a cascading transformation. Each policy receives the rewrite state produced by all higher-priority policies and transforms it further:

initial state → [Policy A] → state' → [Policy B] → state'' → ... → execution plan

The state carries the rewritten query text, accumulated filters, current intent, and any synonym expansions. A high-priority policy can remove text from the query, and every subsequent policy sees the modified query, not the original. Context accumulates. Order matters.

Precedence and conflict resolution: Determinism matters

The specific conflict strategies are a design choice. Different organizations may resolve conflicts differently, depending on their business requirements. The following approach illustrates the kind of judgment framework a control plane needs. The important thing is not these specific strategies but that the system has explicit, deterministic strategies rather than letting conflicts resolve through unpredictable interactions.

Priority ordering

Policies are sorted by priority (highest first). When multiple policies match the same query, they’re applied in priority order. If two policies try to set the same filter field, the higher-priority policy's declared strategy for that field takes precedence. If there are multiple policies triggered that have the same priority, then the policy with the highest ID is given precedence (as if it were assigned a higher priority); this choice ensures deterministic behavior when conflicts arise.

Per-field resolution, not per policy

A critical design principle: Conflict resolution operates per field (for example, brand, category, or description), not per policy. When two policies produce filters that overlap on specific fields, only those specific fields are affected by the conflict resolution strategy, and the resolution strategy is defined by the highest-priority matching policy. Non-conflicting fields from both policies survive intact.

This matters because the alternative of a per-policy approach would force the system to either accept or reject an entire policy when only one of its fields conflicts.

Per-field resolution preserves the maximum amount of useful constraint information.

Three settings per filter field

Each filter field in a policy has three independent settings:

Filter mode: How the filter is applied when there’s no conflict.

• hard_filter (default): Applied as an Elasticsearch bool.filter clause. This is useful for excluding unrelated products entirely. For example, restricting a search for "oranges" to the produce category eliminates hits such as orange juice and orange marmalade. Non-matching documents are completely excluded from results.
• soft_boost: Applied as an Elasticsearch function_score weight with a configurable boost_weight. Documents that match get a ranking boost, but non-matching documents aren’t excluded. This is useful for something like boosting a brand, without excluding other brands.

Conflict strategy

What happens when a lower-priority policy sets the same field:

• override: This high-priority policy's value wins; the lower-priority value is dropped entirely. Valid for all field types.
• restrict: Take the more restrictive numeric value (for example, the lower ceiling for price__max, the higher floor for price__min). Valid for numeric range fields only.
• merge: Combine both values into a union. Valid for non-numeric fields only.
• soft_boost: Convert the conflicting filter to a function_score weight with a configurable boost_weight instead of a hard filter. For more details on function_score boosting, see Influencing BM25 ranking with multiplicative boosting in Elasticsearch. This is only valid for non-negation fields.

Value: The actual filter value (for example, a categories list, a price threshold).

Strategies by field type: Not all strategies make sense for all field types. For instance, an exclusion is inherently binary, so it cannot be soft-boosted. The following table shows which strategies are available for each field type:

Negation fields cannot be soft-boosted because exclusions are binary. Converting "never show canned foods" to "slightly prefer not-canned-foods" fundamentally changes the semantics; a product from "canned foods" would still appear, just ranked slightly lower, which defeats the purpose of the exclusion.

A concrete example: Searching for "cheap chocolate" during a Christmas campaign

Suppose a merchandiser has created the two policies for chocolate that we previously demonstrated, a lower priority one for cheap chocolate and another higher-priority chocolate-related policy that will be enabled during Christmas. If both of these policies are enabled, then how these are combined depends on the filter mode and conflict strategy of the higher-precedence policy. If both of the previously discussed policies are enabled, they’ll be combined as follows:

This shows two conflicts, one on categories and one on price. It’s worth noting that the query that will be executed after this transformation has the following characteristics:

• Only products from the “Christmas foods and drinks” and “Christmas sweets” categories will be shown.
• Within those categories, if the products are also tagged as being in the “Advent calendars” category, they’ll be boosted up by 3x.
• A price filter for $2 is applied, which came from the lower-priority policy (because the higher-priority policy specified to “Restrict” on conflict).
• The word “cheap” is removed, only returning products matching “chocolate”.

With both of these policies enabled, “cheap chocolate” returns results similar to the image shown below:

Relaxing constraints

Perhaps the retailer doesn’t want to exclude products in the categories of “Chocolates” and “Milk chocolates” during Christmas. The settings on the Christmas policy might have overreached and inadvertently removed categories applied by the “cheap chocolate” policy. This is an example that shows why it might be more desirable to combine lower-priority policies with conflicting higher-priority policies. For example, we could modify the Christmas chocolates promotion so that instead of “Override” on conflict, we do a soft boost. The change to that policy would be as follows:

After this modification, the query rewriter transformation pipeline execution for “cheap chocolate” looks as follows:

With the soft boost on conflict, the conflicting filters are converted into soft boosts rather than being dropped. The query that will be executed on the product catalog after this transformation has the following characteristics:

• Because “On conflict” is specified as “Soft boost” on the higher-priority policy, the conflicts will be converted to boosts as follows:
  • Products from the “Christmas foods and drinks” and “Christmas sweets” categories will have a boost of 1x applied to them.
  • Products from the “Chocolates” and “Milk chocolates” categories will have a boost of 3x applied to them.
• As in the previous example, if the products are also tagged as being in the “Advent calendars” category, they’ll be boosted up by 3x.
• As in the previous example, a price filter for $2 is applied.
• The word “cheap” is removed, only returning products matching “chocolate”.

With relaxed filtering, results look as follows:

Overriding price from a high-priority policy

Or perhaps the retailer wants to allow slightly more expensive chocolates to be shown during Christmas by increasing the price max to $7. To ensure that the max price from the Christmas chocolates policy is not overridden if someone searches for “cheap chocolates”, we can set the conflict mode on the price to “override” rather than “restrict”, as follows:

With this override, the query for “cheap chocolate” ignores maximum price that is defined in the “cheap chocolate policy” and only applies the price specified in the “Christmas chocolates” policy, as follows:

This is similar to the previous example, with the difference being that the max price is set to the $7 value from the higher-priority policy because that policy specified “Override” on conflict. With the Christmas price filter taking precedence, the results look as follows:

These three variations (override, soft_boost, and override on price) demonstrate a key property of the system: A merchandiser can change how two policies interact by modifying a setting on a single field within a single policy, without deploying any code. The conflict strategy is the lever that controls business behavior.

Consumed phrase tracking

There’s a subtler form of conflict: two policies that match on the same phrase. If a higher-priority policy removes "without peanuts" from the query, a lower-priority policy that also matched on "without" has nothing left to act on. The system detects if the matched phrase is no longer present in the rewritten query and skips the lower-priority policy.

Intent policies are exempt from consumed phrase tracking: They set the retrieval strategy based on the original query match, regardless of what text has been removed by higher-priority policies.

Priority ordering, per-field conflict resolution, and consumed phrase tracking together give the control plane a deterministic composition model. With that foundation in place, the system can make a routing decision that would be risky without it.

Governance makes retrieval strategy safe

An important insight about routing to the correct retrieval method (text, semantic, or hybrid) is that it executes after governance. If your policies have already enforced "produce category”, then semantic retrieval becomes far less risky because the candidate set is constrained. A semantic search over 500 product items is a very different proposition from a semantic search over 500,000 SKUs. Governance narrows the blast radius before retrieval begins.

For example, without governance, a semantic query for “Fruit high in vitamin C under $4”, in addition to fruits, might return vitamin bottles, carrots, and green pepper. The control plane ensures that these undesired results aren’t even considered as part of the semantic expansion.

With that constraint in place, the control plane applies pragmatic routing logic:

• Lexical for navigational and head queries where deterministic precision matters.
• Semantic for descriptive discovery queries where concept matching helps.
• Hybrid selectively, when constraints have already been enforced and the business accepts broader recall.

From architecture to implementation

The governed control plane translates business intent into deterministic, composable execution plans, without embedding that logic in application code. Policies are data: matched at query time, resolved through explicit per-field conflict strategies, and applied as cascading transformations that produce explainable results. Elastic Services Engineering has built and deployed this architecture for enterprise ecommerce teams, using repeatable patterns and accelerators that compress the path from concept to production. You can see a demo of our implementation of a control plane on YouTube at: Fixing Search Relevance in Seconds: Introducing PRISM.

What's next in this series

The next post goes hands-on with the implementation: how the Elasticsearch percolator powers the policy lookup, including index mappings, boundary markers, highlight-driven phrase tracking, and concrete query examples.

Put governed ecommerce search into practice

The control plane architecture described in this post (per-field conflict resolution, cascading policy transformations, and governance-constrained retrieval routing) was designed and built by Elastic Services Engineering. Every pattern, screenshot, and transformation pipeline shown in this series comes from a working system built by Elastic Services Engineering and validated against enterprise-scale product catalogs.

If you want to implement a governed, policy-driven control plane on Elasticsearch, Elastic Services can get you there faster.

Join the discussion

Have questions about search governance, retrieval strategies, or ecommerce search architecture? Join the broader Elastic community conversation.

Learn how to get started with practical examples you can use right away.

Elasticsearch ES|QL query builder for JavaScript and TypeScript

If you've ever built an ES|QL query in JavaScript, you've probably written something like this:

const query = `FROM logs-*
| WHERE status_code >= ${minStatus}
  AND host.name == ${hostname}
  AND @timestamp >= "${startDate}"
| STATS error_count = COUNT(*) BY status_code
| SORT error_count DESC
| LIMIT 10`

It looks fine until hostname is O'Brien's server and the whole thing blows up with a parse error. Or until a user passes "; DROP INDEX logs into a search field and you realize you've been building queries with raw string concatenation this entire time.

There's a better way. The ES|QL query builder for JavaScript and TypeScript lets you write queries like this instead:

import { ESQL, E, f } from '@elastic/elasticsearch-esql-dsl'

const query = ESQL.from('logs-*')
  .where(E('status_code').gte(minStatus))
  .where(E('host.name').eq(hostname))
  .where(E('@timestamp').gte(startDate))
  .stats({ error_count: f.count() })
  .by('status_code')
  .sort(E('error_count').desc())
  .limit(10)

Values are escaped automatically. You get autocomplete in your editor. And you can see exactly what the query does, without mentally parsing a template literal.

ES|QL query builders are already available across Elastic's language clients, including Python, Ruby, and others. This article focuses on the JavaScript and TypeScript version, walking through practical examples you can start using today.

Getting started

Install the package:

npm install @elastic/elasticsearch-esql-dsl

Here’s a minimal query:

import { ESQL, E } from '@elastic/elasticsearch-esql-dsl'

const query = ESQL.from('employees')
  .where(E('still_hired').eq(true))
  .sort(E('last_name').asc())
  .limit(10)

console.log(query.render())

This renders:

FROM employees
| WHERE still_hired == true
| SORT last_name ASC
| LIMIT 10

To run it against Elasticsearch:

import { Client } from '@elastic/elasticsearch'

const client = new Client({ node: 'http://localhost:9200' })
const response = await client.esql.query({ query: query.render() })

That’s it. No string interpolation, no manual escaping.

Building a real query, step by step

Let's walk through a realistic scenario: You're building a dashboard that analyzes web server error logs. We'll start simple and layer on features.

Step 1: Filter error logs

import { ESQL, E } from '@elastic/elasticsearch-esql-dsl'

const errors = ESQL.from('logs-*')
  .where(E('status_code').gte(400))
  .limit(100)

FROM logs-*
| WHERE status_code >= 400
| LIMIT 100

Step 2: Add a computed column

Your timestamps are in milliseconds, but you want response time in seconds:

const errors = ESQL.from('logs-*')
  .where(E('status_code').gte(400))
  .eval({ response_secs: E('response_time_ms').div(1000) })
  .limit(100)

FROM logs-*
| WHERE status_code >= 400
| EVAL response_secs = response_time_ms / 1000
| LIMIT 100

Step 3: Aggregate errors by status code

import { f } from '@elastic/elasticsearch-esql-dsl'

const errorBreakdown = ESQL.from('logs-*')
  .where(E('status_code').gte(400))
  .stats({
    error_count: f.count(),
    avg_response: f.avg('response_time_ms'),
  })
  .by('status_code')
  .sort(E('error_count').desc())

FROM logs-*
| WHERE status_code >= 400
| STATS error_count = COUNT(*), avg_response = AVG(response_time_ms) BY status_code
| SORT error_count DESC

The f namespace gives you access to 150+ ES|QL function wrappers: aggregations, string functions, date functions, math, geo, and more. They all return chainable expressions, so you can use them anywhere you'd use E().

Step 4: Use date functions for time-based analysis

const hourlyErrors = ESQL.from('logs-*')
  .where(E('status_code').gte(400))
  .eval({ hour: f.dateTrunc('@timestamp', '1 hour') })
  .stats({ error_count: f.count() })
  .by('hour')
  .sort(E('hour'))

FROM logs-*
| WHERE status_code >= 400
| EVAL hour = DATE_TRUNC(@timestamp, "1 hour")
| STATS error_count = COUNT(*) BY hour
| SORT hour

Step 5: Branch queries safely

Every method returns a new query object. The original is never mutated. This means you can build a base query and branch it for different views:

const base = ESQL.from('logs-*')
  .where(E('status_code').gte(400))
  .where(E('@timestamp').gte('2026-01-01T00:00:00Z'))

const byStatus = base
  .stats({ count: f.count() })
  .by('status_code')
  .sort(E('count').desc())

const byHost = base
  .stats({ count: f.count() })
  .by('host.name')
  .sort(E('count').desc())
  .limit(20)

const recent = base
  .sort(E('@timestamp').desc())
  .keep('@timestamp', 'status_code', 'url.path', 'message')
  .limit(50)

Three different queries, one shared base. Change the filter on base, and all three update. This is especially useful for dashboards where multiple panels query the same dataset with different aggregations.

Three ways to write expressions

The domain‑specific language (DSL) gives you flexibility in how you write conditions. Here's the same WHERE clause written three different ways:

Raw strings: When you're writing a quick one-off:

.where('status_code >= 400 AND host.name == "web-01"')

The E() expression builder: When you want type safety and autocomplete:

import { and_ } from '@elastic/elasticsearch-esql-dsl'

.where(and_(
  E('status_code').gte(400),
  E('host.name').eq('web-01')
))

The esql template tag: -When you want safe interpolation of dynamic values:

import { esql } from '@elastic/elasticsearch-esql-dsl'

const minStatus = 400
const host = 'web-01'
.where(esql`status_code >= ${minStatus} AND host.name == ${host}`)

All three produce the same ES|QL. Pick whichever fits your situation: raw strings for simple cases, E() when building expressions programmatically, and the template tag when mixing literal ES|QL with dynamic values.

Keeping queries safe

If any part of your query comes from user input, you need to think about injection. ES|QL supports parameter binding, and the DSL makes it straightforward:

function searchLogs(userQuery: string) {
  const query = ESQL.from('logs-*')
    .where(E('message').eq(E('?')))
    .limit(100)

  return client.esql.query({
    query: query.render(),
    params: [userQuery],
  })
}

The ? placeholder is replaced server-side by Elasticsearch, so the user's input never touches the query string. No escaping, no injection risk.

Beyond the basics

Once you're comfortable with the core commands, the DSL supports every advanced ES|QL feature:

Hybrid search with FORK and FUSE:

const results = ESQL.from('articles')
  .fork(
    ESQL.branch()
      .where(f.match('title', 'elasticsearch'))
      .sort(E('_score').desc())
      .limit(50),
    ESQL.branch()
      .where(f.knn('embedding', 10))
      .sort(E('_score').desc())
      .limit(50),
  )
  .fuse('RRF')
  .limit(10)

Data enrichment:

const enriched = ESQL.from('logs-*')
  .enrich('ip_lookup')
  .on('client.ip')
  .with('geo.city', 'geo.country')

Conditional aggregation:

const stats = ESQL.from('employees')
  .stats({
    eng_avg: f.avg('salary').where(E('dept').eq('Engineering')),
    sales_avg: f.avg('salary').where(E('dept').eq('Sales')),
    total: f.count(),
  })

AI/machine learning (ML) integration:

const summarized = ESQL.from('docs')
  .completion('Summarize this document')
  .with({ inferenceId: 'my-llm' })

For the full list of commands and functions, check out the ES|QL query builder documentation.

What's next

This is the initial release of @elastic/elasticsearch-esql-dsl. You can find the package on npm, explore the source on GitHub, and read the full documentation in the repository. If you run into issues or have feature requests, open an issue; we're actively developing this and want to build what JavaScript and TypeScript developers actually need.

Users familiar with cross-cluster search (CCS) know that to include a remote cluster in a search you must specifically reference it with a cluster alias prefix, such as remote:metrics* (or *:metrics* to target all remotes). With CPS, "bare" index names or patterns such as metrics* in a search index expression implicitly reference all instances of that index that can be found on the origin project and on all of its linked projects.

In this post, we pull back the curtains on how the Elasticsearch TransportSearchAction, the code that underlies the _search API (and _async_search) APIs, determines which indices, aliases, and datastreams to search on which projects when running a cross-project search.

Analyzing index expressions

One important consequence of the new CPS model is that the Elasticsearch query parameters allow_no_indices and ignore_unavailable require different handling than before. In general, these parameters control whether a search should throw an error if a concrete index name (for example, "logs") cannot be found or isn’t accessible (ignore_unavailable=false), or a wildcard pattern (for example, "logs*") doesn’t match anything or if there are no indices at all to search (allow_no_indices=false).

In CCS, those parameters are analyzed on each cluster separately: Each cluster can just consult which indices, aliases, or data streams exist locally. But in CPS, we need to account for whether each resource (index, alias, or data stream) referenced by an index expression matches on any project (origin or linked), rather than every project.

To illustrate, suppose a user has one linked project (linked1) and they issue a cross-project query with index expression logs*,metrics-1, along with allow_no_indices=false,ignore_unavailable=false. As long as we find one resource (index, alias, or data stream) that matches logs* and one that matches metrics-1 on any project, then the search can proceed. If metrics-1, for example, is found on linked1 but not the origin project, that suffices to pass the ignore_unavailable=false constraint. Only if it’s found nowhere would we throw an IndexNotFoundException.

To handle this, the _search API in cross-project search mode needs to gather information from each linked project before kicking off the actual search.

The serverless node that receives the _search REST request is considered to be the origin project, and it acts as the overall search coordinator. To fully analyze and process an index expression before kicking off the actual search, the search coordinator needs to:

1. Determine which projects are in scope for the search.
2. Determine which indices, aliases, or data streams should be searched in each individual project.
3. Do a final reconciliation step of all the information gathered: 1) Have we found all the indices required to proceed (as defined by the allow_no_indices and ignore_unavailable settings)?; and 2) Are there any projects that should be skipped since they have no matching indices?

To illustrate, we’ll follow a CPS request against the _search API from start to finish. Suppose that the incoming user request is to search metrics* and that the origin project has two indices which match the expression, namely metrics-ES and metrics-KB, and that the origin project is linked to projects P1 and P2, where P1 has index metrics-ES and P2 has no index, alias, or data stream that matches metrics*.

Determining projects in scope

The search coordinator starts by grabbing the full list of linked projects from Elasticsearch cluster state. It then determines, based on Universal Identity and Access Management (UIAM) credentials provided with the query, which of those projects the user has access to and keeps the subset of projects that the user is allowed to access.

If a project_routing parameter is present on the request, that can further limit which projects are in scope for the query. For instance, "project_routing":"_alias:_origin" would cull the list to just the origin project, while "project_routing":"_alias:P*" would include the linked projects P1 and P2 and remove the origin project from the list of projects in scope for the query.

The index expression itself can also limit which projects to access. Qualified index expressions, like "P*:metrics*", would indicate that we should search only on projects whose alias starts with "P". For our case, the "bare" or unqualified index expression "metrics*" means "search on all projects that are in scope," where "in scope" is modified by security access and project_routing.

Determining which indices each project has

Once we know which projects to search, we need to determine which matching indices each one has. To support this, the search coordinator creates a ResolvedIndexExpressions data structure that allows tracking which indices on each project should be included in the search.

For each index expression provided by the user, that data structure tracks:

• Original expression: The index expression, as provided by the user.
• Local resolution: The local expressions that will replace the original together with the resolution result.
• Remote expressions: A set of remote expressions one for each project a query can target.

On the search coordinator that received the request, we’ve so far been able to determine which projects are in scope (origin, P1, and P2) and which, if any, matching indices exist on origin. So, for our example search against metrics*, we’ll have the following structure at that point in time:

• Original expression: metrics*.
• Local resolution: SUCCESS, <metrics-ES, metrics-KB>.
• Remote expressions: <P1:metrics*, P2:metrics*>.

The remote expressions are left unresolved until we later contact the P1 and P2 projects to fill in that information.

The index resolution can be any of the following:

• SUCCESS: Local resolution completed successfully.
• NOT_VISIBLE: Indicates that a non-wildcard expression was resolved to nothing, either because the index doesn’t exist or is closed.
• UNAUTHORIZED: Indicates that the expression could be resolved to a concrete index, but the requesting user isn’t authorized to access it.
• NONE: No local resolution was attempted, typically because the expression is remote-only (for example, P1:index).

Check indices on the linked projects

In the search API, how we check for indices on the linked projects depends on whether the cross-project search is being run with minimize round trips or not. Most searches in CPS are set internally to run with minimize_roundtrips=true, so we’ll focus on that pathway.

In CCS, where index expression analysis is done locally on each cluster, we just send the same request to every cluster and, with minimize_roundtrips=true, each remote cluster sends back an entire SearchResponse that the primary search coordinator collects and eventually merges into all the other responses it receives.

By contrast, for CPS, an additional phase was introduced to search where we contact each linked project to assess which indices are present and which ones the user has access to. This is an additional round trip that uses ResolveIndexAction, the class that implements the functionality of the _resolve/index endpoint. Upon receiving the ResolveIndex responses from all the linked projects, we can fill in the ResolvedIndexExpressions data structure on the primary search coordinator. For this case, since P1 has index metrics-ES and P2 has no matching indices, the updated data structure would be:

• Original expression: metrics*.
• Local resolution: SUCCESS, <metrics-ES, metrics-KB>.
• Remote expressions: <P1: SUCCESS <metrics-ES>, P2: NOT_VISIBLE>.

CPS validator

Once we have all the linked project information, we can run the validation to honor the IndicesOptions specified by the caller.

If the user had specified a qualified index expression (for example, original=P1:metrics*), the CPS validator needs to make sure that project P1 has at least one index matching metrics*, otherwise a 404 index not found exception would be returned to the user. In other words, qualified expressions imply that an index, alias, or data stream matching that name must be present on all projects specified by the qualifier.

On the other hand, for “bare” (unqualified) CPS index expressions (for example, original=metrics*), we only need to check whether the original index expression exists anywhere.

For the example we’re following in this post, the ResolveIndexExpressions show that at least one index, alias, or data stream matching metrics* was found, so the search can proceed.

However, we can also see that the P2 cluster has no matching indices. In that case, we can remove it from the rest of the query. So now, for the actual query, we’ll:

• On the origin project, search metrics-ES, metrics-KB.
• On the P1 project, search metrics-ES.
• Skip the P2 project. Note: Since it wasn’t included in the query at all, it won’t show up on the _cluster/details of the SearchResponse.

Conclusions

We’ve described how the TransportSearchAction class in Elasticsearch supports some key new features of cross-project search. That class supports a large number of Elasticsearch REST APIs, such as, _search, _async_search, _msearch, _eql, _sql and _count, all of which use the mechanisms described in this document.

"Of course! I'd be happy to help you..."

"This should give you a good overview..."

"Feel free to let me know if you need anything else!"

This isn't just annoying; it's expensive. Every token costs money and adds latency. For production Elasticsearch queries, that overhead compounds fast. In this post, we introduce elastic-caveman and share the results of a controlled experiment across eight live Model Context Protocol (MCP) scenarios against an Elasticsearch cluster. The findings: 63.6% average token reduction, 817 tokens saved, and zero loss of technical accuracy.

Enter elastic-caveman

elastic-caveman tests a simple hypothesis: Strip AI responses to pure signal, and measure the impact. The approach:

• Normal mode: Full conversational AI with greetings, explanations, and sign-offs.
• Caveman mode: Raw data with minimal structural labels only.

We tested both modes against a live Elasticsearch instance using MCP with real support ticket and Salesforce case data across eight production scenarios.

Results: 64% token reduction, zero accuracy loss

Here's what we found across eight live MCP tool calls: The Elastic-Caveman initiative has successfully optimized AI response size without compromising quality or functionality.

Key preservations (0% loss):

• Technical accuracy
• API paths
• ES|QL syntax
• Field names

The critical finding: Every field name, case ID, ES|QL query, account name, and sentiment score was preserved exactly. Not approximately. Exactly.

Real examples: Before and after

Example 1. List indices: 87% reduction

User: Show me my indices

Normal mode (107 tokens):

Of course! I'd be happy to help you see your indices. Here's a comprehensive list of all the indices in your Elasticsearch cluster. Each entry shows the index name along with relevant metadata. This should give you a good overview of what's stored in your cluster:-- salesforce-cases-- support-ticketsThese are all the indices I found. If you'd like to dive deeper into any specific index, just let me know and I'll be glad to assist!

Caveman mode (14 tokens):

Indices: salesforce-cases, support-tickets

Saved: 93 tokens (86.9%)

Example 2. Generate ES|QL query: 75% reduction

User: Show me open critical tickets grouped by product area

Normal mode (208 tokens):

Sure thing! I'd be happy to help you write that ES|QL query. What you're looking for is a query that will filter for open tickets with critical priority and group them by product area. Here's how you can accomplish this...

[followed by the actual query, plus 150+ tokens of step-by-step explanation]

Caveman mode (52 tokens):

FROM support-tickets
WHERE status = "Open" AND priority = "Critical"
| STATS count = COUNT(*) BY product_area
| SORT count DESC
| LIMIT 100

Filters open+critical, groups by product_area, counts, sorts desc.

Saved: 156 tokens (75.0%). ES|QL syntax is character-for-character identical in both modes.
Example 3. Search recent support tickets: 35% reduction

User: Show me 5 recent support tickets

Caveman mode (143 tokens):

5 recent tickets, newest first:
Case     | Priority | Status    | Account               | Subject                                    | Days | Sentiment
CASE-0012 | Critical | Open      | Pinnacle Financial    | Regulatory reporting: incorrect tax calcs  | 2    | -0.94
CASE-0021 | Critical | Escalated | Riverstone Capital    | Security audit blocking renewal            | 2    | -0.93
CASE-0046 | Medium   | Open      | Blackrock Partners    | Pen test findings need vendor response     | 2    | -0.70
CASE-0006 | High     | Open      | United Oil & Gas      | Unauthorized access -- urgent               | 3    | -0.91
CASE-0017 | High     | Open      | Riverstone Capital    | Renewal paperwork stalled                  | 4    | -0.86

All negative sentiment. Two critical cases from April 7. All case IDs, priorities, account names, and sentiment scores preserved verbatim.

What gets removed vs. what stays

When we clean up the output, we strip out conversational filler, like “Of course! I’d be happy to help you…”, “This should give you a good overview…”, or “Would you like me to help you prioritize these?”, and we keep every piece of factual content, such as ES|QL snippets, like FROM support-tickets WHERE status = "Open"; field names like sentiment_score, product_area, and resolution_hours; and index names, like support-tickets and salesforce-cases. We also preserve concrete identifiers and business entities, such as case IDs CASE-0012 and CASE-0002; account names, like Pinnacle Financial and United Oil Gas Corp; along with all numeric values, for example, a sentiment_score of -0.94, counts like 47 duplicates, durations such as 18 days, or metrics like 27.0 average hours, so the edited text is tightly focused on query syntax, entities, and numbers while discarding only the polite scaffolding.

Results varied by operation type:

Complete evaluation breakdown

Token savings by query type across all eight scenarios against live MCP data:

Technical accuracy verification:

Zero information loss. 64% fewer tokens.

Why this matters for Elastic users

For teams building AI assistants on Elasticsearch, 64% token reduction means 64% savings on output costs at scale, faster streaming responses, and more context window space for actual data rather than fillers. When you're debugging an ES|QL query at 2 a.m., you don't need an AI telling you it's delighted to help; you just need the query response!

The bigger picture: Rethinking AI interfaces

This experiment reveals something fundamental: Conversational AI interfaces optimize for the wrong metric. They optimize for sounding human when users often just want accurate data, fast.

For technical workflows, especially data queries, there's a strong case for mode-switching:

• Conversational mode: When exploring or learning.
• Caveman mode: When you know what you want and need it now.

The Elastic MCP server makes this possible by returning structured, accurate responses that work in both modes without modification.

How elastic-caveman works

elastic-caveman is an Agent Skill, that is, a markdown file with YAML front matter that any compatible AI agent reads and follows. No runtime. No binary. No API calls. Just instructions that reshape how your agent talks when working with Elasticsearch.

Install with:

npx skills add srikolag/elastic-caveman

Supported agents: Claude Code, Cursor, Codex, Windsurf, GitHub Copilot, Gemini CLI, Roo

Trigger with:/elastic-caveman

Disable with:"normal mode" or "verbose"

Live in action

We tested elastic-caveman with the Claude model to measure its impact on token usage and cost:

• With elastic-caveman: Token usage was 368 tokens (in) and 1.6k tokens (out), resulting in a cost of $0.11.
• Without elastic-caveman: Token usage was 367 tokens (in) and 1.8k tokens (out), resulting in a cost of $0.12.

Prompt: Get me the critical support tickets from the support-tickets index in kibana for Pinnacle Financial

This test demonstrates the efficiency of elastic-caveman.

What's next

Caveman mode is just the beginning. Consider dynamic mode switching: Flip between concise and conversational mid-session. Or a hybrid approach: Lean on success, explanatory on errors. Or custom verbosity levels for teams that want something in between. The goal isn't to make AI assistants robotic; it's to give users control over the signal-to-noise ratio.

Try it yourself

Test caveman mode with your Elasticsearch data:

1. Set up the Elastic MCP server.
2. Install elastic-caveman.
3. Run queries in both normal and caveman modes.
4. Compare token counts and accuracy.

Full evaluation methodology and scripts available in the GitHub repo.

The bottom line

Across eight real scenarios with live Elasticsearch data, elastic-caveman delivered 64% average token reduction with zero accuracy loss and 100% preservation of ES|QL syntax, field names, and technical values. Sometimes the best AI response isn't the chattiest one. Sometimes you just need the data; and with elastic-caveman, you can get it 64% faster. Ready to optimize your Elasticsearch AI workflows? Check out Elasticsearch Labs for more tutorials, integrations, and research on building with Elasticsearch and AI, or start building with Elasticsearch today.

Want to optimize your Elasticsearch AI workflows? Check out Elasticsearch Labs for more tutorials, integrations, and research on building with Elasticsearch and AI. Ready to try it yourself? Start building with Elasticsearch today.

Why time series discovery matters

Elasticsearch uses time series data streams (TSDS) to efficiently store metrics. Backed by a fully columnar store, metrics stored in TSDS in Elasticsearch 9.4 require up to 17x less storage compared to using a standard index. Starting with Elasticsearch 9.2, we've also added time-series support in Elasticsearch Query Language (ES|QL) as a fully supported capability when querying data stored in TSDS.

If you operate TSDS in Elasticsearch, you already know the pattern: dimensions identify a series, metrics carry typed values like gauge or counter, and the TS source command in ES|QL enables time series aggregation functions such as RATE and AVG_OVER_TIME.

What that pipeline can't tell you (but you need to know just as often) is which metrics and time series actually exist right now, for the slice of data you care about. Field mappings enumerate every field that was ever declared; they don't show what's actively being ingested in a specific cluster, environment, or time window. That gap shows up across very different workflows:

• Dashboard building. Metric and dimension pickers should reflect what the cluster currently holds for the user's filters, not every field that has ever been mapped. Otherwise, dropdowns stay cluttered with stale options and panels render empty.
• Onboarding to an unfamiliar TSDS. A new cluster, a new integration, a customer's data. A quick list of the metrics being ingested, with their types, units, and applicable dimensions, replaces hours of mapping spelunking and ad hoc probe queries.
• Data quality investigations. Mapping drift (the same metric declared gauge in one backing index and counter in another) and dimension-cardinality explosions both surface immediately in the catalog output.
• Query validation. Before running an expensive TS ... | STATS aggregation, confirm that the metric and dimensions you're about to use really have data in your window.

Kibana already relies on this internally. The dynamic metrics catalog in the observability experience appends METRICS_INFO to the user's active TS query so the UI only offers metrics that truly exist for the current filters, rather than every field in the mapping.

The problem: Mappings are an inventory of fields, not time series

Operations teams routinely need answers to questions that mapping APIs alone cannot answer:

• Which metrics actually have data in this environment, for this cluster, in this time range?
• How are those metrics typed, and which dimensions apply when building or validating a query?
• How many distinct time series exist per metric?

Until now, answering these questions meant piecing together mapping APIs, ad hoc queries, and guesswork. METRICS_INFO and TS_INFO turn those questions into single-line ES|QL queries that fit naturally into the same pipeline you use for STATS:

TS k8s
| WHERE cluster == "prod"
| METRICS_INFO
| SORT metric_name

How these commands integrate with ES|QL pipelined queries

Both commands are processing commands. Once you run one, the table is replaced: Downstream commands, like KEEP, WHERE, or STATS, operate on metadata rows, not the original time series documents.

A few rules to keep in mind:

• They apply only after a TS source. Using them after FROM or without a preceding TS source produces an error.
• They must appear before STATS, SORT, or LIMIT run on the time series rows returned by TS. For example, TS ... | STATS ... | METRICS_INFO is invalid; TS ... | METRICS_INFO | STATS ... is valid because STATS then runs on the metadata table.
• You can filter and aggregate after METRICS_INFO or TS_INFO on the metadata columns with the usual processing commands.
• You can include filters before them, for example, narrowing by @timestamp or dimensions, so that the produced metadata reflects series that match your query context, not the entire index.

Conceptually, the pipeline looks like this:

TS + filters  →  METRICS_INFO or TS_INFO  →  KEEP / WHERE on metadata  →  STATS / SORT / LIMIT

This design means you can scope a catalog to exactly the slice of data you care about and then post-process the result with more ES|QL commands as desired.

How to use METRICS_INFO and TS_INFO in practice

METRICS_INFO retrieves information about the metrics available in your time series data streams, together with applicable dimensions and other metadata, all scoped to the current TS query. TS_INFO does the same for individual time series. Each row is one metric plus the dimension values that identify one series.

Each command offers a different view to time series metadata: METRICS_INFO collapses what you see into one row per distinct metric signature: the metric name plus how it's declared (type, unit, field type, which dimension fields apply) as observed across backing indices. TS_INFO adds one row per metric and time series, with a dimensions column that holds the concrete label set for each series, formatted as a JSON object (for instance, {"job":"elasticsearch","instance":"instance_1"}).

If the same logical metric name shows up with incompatible metadata in different places, you get multiple rows or multi-valued cells. That's a useful signal when you're tracking down mapping drift.

Both commands expose the same core columns; only TS_INFO adds dimensions.

Start with a catalog of names and types. The smallest useful query is a TS source, METRICS_INFO, and a sort so the table is easy to scan:

TS k8s
| METRICS_INFO
| SORT metric_name

You can post-process the result as usual in ES|QL. For instance, you can trim columns or filter on metadata before aggregating:

TS k8s
| WHERE cluster == "prod" AND TRANGE(1d)
| METRICS_INFO
| KEEP metric_name, metric_type
| SORT metric_name

To find how many distinct metric names match a pattern (not which series), combine METRICS_INFO with STATS:

TS k8s
| METRICS_INFO
| WHERE metric_name LIKE "network.total*"
| STATS matching_metrics = COUNT_DISTINCT(metric_name)

Document predicates before the catalog command narrow the processed time series to data samples that actually exist in your window. The metrics listed are those with matching data, not every field that has ever been mapped:

TS k8s
| WHERE cluster == "prod" AND TRANGE(1d)
| METRICS_INFO
| SORT metric_name

Run the same scoped pipeline, but swap the middle command for TS_INFO, and the question shifts from “which metrics match” to “which time series identities match”. Each row is one metric plus one combination of dimension values; sort on metric_name and dimensions so related series group together:

TS k8s
| WHERE cluster == "prod" AND TRANGE(1d)
| TS_INFO
| SORT metric_name, dimensions

That extra column can be used to deduce metric cardinality. Each TS_INFO row is one time series for a given metric, so grouping with STATS counts how many distinct time series exist per metric:

TS k8s
| TS_INFO
| STATS series_count = COUNT(*) BY metric_name
| SORT metric_name

Choosing between them: Use METRICS_INFO when you want a compact inventory of metric names and types in the filtered TS context. Use TS_INFO when you need label combinations, per-metric series counts. In practice, skim with METRICS_INFO and then switch to TS_INFO when the answer depends on which dimensions apply, not just what metrics exist.

Under the hood: How the commands are executed

Both METRICS_INFO and TS_INFO run inside the same distributed ES|QL execution that powers any TS query. In addition to standard features, like shard-level parallelism, Lucene filter pushdown, and coordinator-side merging, special care has been taken during implementation so that the cost scales with the number of matching time series, not the number of documents. Here's how each output row gets produced:

1. The TS command defines the scope. TS resolves your data stream pattern to its TSDS backing indices and turns any filters you place before the catalog command, such as a time range on @timestamp or dimension predicates in WHERE, into a Lucene query that runs on every shard that can match. Shards in backing indices outside the time window are pruned up front and never touched.

2. Each shard iterates over matching documents and tracks one per series. A TSDS index is physically sorted by _tsid first, then by @timestamp (descending). That sort matters here: All documents belonging to the same time series sit next to each other on disk, so as a shard processes documents in order, it only needs to keep the first document it sees for each new _tsid and skip the rest. The result is one representative document per time series that has at least one document matching your filters.

3. The mapping tells us what each field is. The backing index mapping is the source of truth for the metadata that describes each field:

• Fields declared with time_series_metric are metrics, and the mapping carries each metric's metric_type, field_type, and (if declared) meta.unit.

4. Synthetic source fills in the actual dimension and metric presence. For the one representative document per series, the shard reads a subset of _source containing only the dimension (and metric) paths the mapping declares. TSDS uses synthetic _source, so that subset is reconstructed primarily from doc values — no stored _source is needed. From that reconstructed sliver of JSON, the shard learns two things:

• The dimension key/value pairs for this series (the dimensions JSON for TS_INFO, and the set of dimension keys that feed dimension_fields for both commands).
• Which metric fields actually have data for this series in this backing index.

5. Partial aggregation happens inside each shard. Shards don't ship raw per-series rows upstream. They partially aggregate first, which is a big part of why catalog queries stay cheap.

6. The coordinator merges across shards and data streams. Each data node first reduces its own shards' partial results and streams them to the coordinator, which applies the same merge logic one more time.

7. The rest of the pipeline runs as usual. Everything after the catalog command (KEEP, WHERE, STATS, SORT, LIMIT) runs against this consolidated metadata table on the coordinator, exactly like any other ES|QL stage.

The net effect is that catalog queries do just enough work to identify one representative document per series, read a small reconstructed slice of that document, classify its fields against the mapping, and fold the results into a handful of metadata rows. Because the output cardinality is bounded by the number of matching series (for TS_INFO) or by the number of distinct metric signatures (for METRICS_INFO), not by the number of documents in the window, these commands stay responsive even against long retention windows and high-ingest data streams.

Running these commands against the full high cardinality TSDB benchmark corpus without a time range filter (1.84 B documents / 1.4 M time series / 2.77 TB uncompressed) on a single-node Elasticsearch (AWS c8gd.8xlarge, 24 cores, 24 GiB heap, NVMe SSD, 3 primary shards, force-merged), METRICS_INFO returns in ~4 seconds.

Beyond ad hoc queries

These commands also support product workflows inside Kibana. The UI appends METRICS_INFO to a user's TS query (when the query doesn't already include STATS) to build a metric catalog aligned with the user's filters, rather than relying solely on mappings.

These new commands are also the foundation of Prometheus-compatible metadata APIs we're adding to Elasticsearch, which Prometheus-based tools can use. Stay tuned for a dedicated blog post that goes into more detail.

Data quality

Multi-valued unit, metric_type, or field_type in the METRICS_INFO output, is a concise warning that backing indices aren't aligned on a metric's definition. TS_INFO makes it easier to see whether an explosion in cardinality comes from a handful of metrics or from dimension cardinality you should account for in alerts and aggregations. For example, ranking metrics by series count surfaces outliers at a glance:

TS k8s
| TS_INFO
| STATS series_count = COUNT(*) BY metric_name
| SORT series_count DESC

When a single metric dwarfs the rest, as network.eth0.rx does above, the explosion is concentrated in a handful of metrics, and inspecting that metric's dimensions pinpoints which label is growing. Comparable counts across metrics instead point at shared dimension cardinality, such as a newly introduced pod or instance value propagating through every time series.

Availability

METRICS_INFO and TS_INFO are generally available in Elastic Cloud Serverless and in Elasticsearch basic starting with the 9.4.0 release.

For command pages (syntax, restrictions, and examples), see METRICS_INFO and TS_INFO.

For background on TSDS and the TS command itself, start with the official documentation on time series data streams and the TS source command.

When “something is slow” finally has an answer

Query activity is already available in your Elastic Cloud Serverless project today. For Elastic Cloud Hosted and Elastic Self-Managed clusters, it ships with Kibana 9.4 and is available across all deployments and clusters on that version. Query activity is the Kibana view on top of the Tasks Management API in Elasticsearch. It’s purpose-built for search-related tasks in any query language, including Elasticsearch Query Language (ES|QL), DSL, SQL, and Event Query Language (EQL).

It always starts the same way. Someone pings you on a Friday: Discover feels stuck. The exec dashboard won’t load. Did we change something? You open your monitoring tabs, squint at CPU, maybe tail a log, and you’re still guessing. Is it one giant ES|QL pipeline? A dashboard nobody remembers? A background rule doing honest work at the worst possible time? The cluster isn’t mysterious on purpose. In-flight work is simply invisible unless you enjoy living in Dev Tools and reconstructing life stories from task IDs and snippets of JSON.

We built Query activity for everyone who has ever muttered, Just tell me what’s running. It’s a new screen in Kibana that lists active search work in ES|QL, DSL, SQL, or EQL. It shows the queries that are consuming your cluster right now, with enough context to move from panic to diagnosis without a scavenger hunt.

The play you know and the one-minute rewrite

If you’ve operated Elasticsearch for more than a week, you’ve lived the old script. In Act I, someone says the cluster feels slow. In Act II, you scatter across shards, heap, slow logs, and sticky-note task IDs. Hours pass, and you still cannot name the query. In Act III, maybe you find the culprit before dinner, or maybe next month Act I opens again with the same villain in a fake mustache.

Query activity replaces that meandering Act II with one default sequence. It’s the same story, compressed from symptom to evidence to origin to action in about a minute. Paste this into your runbook or send it to your on-call channel. It’s the whole innovation in practice.

1. Open Query activity as soon as Act I hits. On Elastic Cloud Hosted and Elastic Self-Managed clusters, go to Stack Management and then Cluster performance. On Elastic Cloud Serverless, go to Admin and Settings and then Project performance. Do this before you fork into guesswork.

2. Refresh the list of queries once so you’re looking at right now, not five minutes ago.

3. Surface the pressure. Sort by run time, or tighten the “Run time” filter, until the expensive work floats to the top.

4. Open the flyout on the worst offender. You’ll see duration, query type, index breadth, and full query text. That’s your evidence without opening Dev Tools.

5. Name the owner. Use trace.id to jump into Discover and filter on the audit or query logs for that trace, or use X-Opaque-Id to figure out which dashboard, saved search, or rule this query originates from.

6. Resolve Act III. Let the query finish, fix the upstream, or cancel when it’s appropriate and Elasticsearch says the task is cancelable.

That’s one pass through what used to be three acts. You get attribution instead of folklore and decisions instead of theater.

Query activity deep dive

The one-minute sequence above is the habit. What follows is the machinery: the concrete controls and signals in Kibana that make that rewrite possible. You get what’s executing, how long it’s been running, where it came from, and what to do next, without stitching clues across tabs.

Under the hood, this view is powered by Elasticsearch’s Tasks Management API for long-running search tasks. It’s translated into an operational UI that’s built for speed. You can find the outlier quickly, inspect rich details, and act with confidence.

Here’s how the UI backs each beat of the runbook.

The main view is a filterable list of running queries. It includes a search bar so you can match anything in the table, including task ID. You also get filters for run time, query language, and source (for example Discover, Dashboard, and similar surfaces). You stay in control of what “noisy” means.

Refresh is manual on purpose. The table does not auto-refresh. You click Refresh when you’re ready, and the UI shows when the last refresh happened. You shouldn’t have to wonder whether the list is stale.

When you click a task ID, a detail flyout opens. It shows start time, run time, how many indices the query touches, and full query text. When X-Opaque-Id is present, it can help you trace an Elasticsearch query to its origin in Kibana so you can turn “mystery load” into “that dashboard, that version.” Previous and next navigation let you walk the queue without jumping back to the list. When trace.id is available, you can open Discover with that trace prefiltered. That helps when the incident channel is already busy.

Where work is cancelable, you can request cancellation from the list or the flyout. There’s a deliberate confirmation step. After you confirm, the cancel control shows a spinner until Elasticsearch reports that the task actually stopped. The goal is oops-proof, not oops-fast.

Viewing and managing active query work requires appropriate cluster privileges. The UI states clearly when something is missing. For example, users without cluster:manage may not be able to take destructive actions. Users without cluster:monitoring may not see task details. You shouldn’t get a blank screen that feels like the stack is playing hide-and-seek.

If you’ve been following our broader story around query observability, this is the live side of the house. It’s what is happening now, in the product, with controls you can use. Over time, pair it with historical views, such as query logs and AutoOps long-running search tasks insights when you need to ask whether this has happened before. When you need to answer what’s eating your cluster in this minute, start with the new Query activity UI in Kibana.

Who this is for (and who becomes the hero)

Cluster and platform admins get the obvious win: faster incident response and less time translating APIs into narratives for stakeholders.

Centers of excellence and internal search champions get something equally valuable: a teachable moment you can screenshot. This is the query pattern that blew up shared capacity. This is what “interactive” versus “background” pressure looks like when everyone is busy.

Anyone on the hook for Service Level Agreements (SLAs) gets a cleaner bridge between user experience (“the app is slow”) and search reality (“these three requests are still running, and one of them is huge”).

You don’t have to be the person who wrote the query to be the person who explains the cluster calmly. That’s the whole point.

Not every task is cancelable, and deep tuning work still has its place. Query activity doesn’t fix your queries for you. It surfaces in seconds which ones might need attention, and it gives you faster evidence, clearer attribution, and better decisions before you reach for heavier tools.

Where to find it

You’ll find Query activity in the performance area of each deployment model. In Elastic Cloud Hosted and Elastic Self-Managed clusters, open Stack Management and then Cluster performance. In Serverless projects, open Admin and Settings and then Project performance.

Threshold hygiene: Open Stack Management and then Advanced Settings. The running_queries:minRunningTime setting defaults to 100 ms. Only tasks that have been running longer than that appear. That way you can dial through noise without drowning in instantaneous work.

What to do next

Walk through the six-step sequence once when the cluster is calm. When Act I hits, you won’t be learning a new UI under pressure. Then repeat it during the next slow moment. The gap between assuming and seeing is the whole product story.

If you aren’t on Elastic Cloud yet, you can still get hands-on with the stack at elastic.cloud/registration.

This post answers those questions. A governed control plane doesn't just improve search relevance; it changes the operating model. It moves search behavior changes from engineering deployment cycles to business-driven workflows, without sacrificing safety or accountability.

The scenario that exposes the operating model

Imagine that you’re in the weeks leading up to Christmas, and your merchandising team has identified three urgent changes that must immediately be made to search behavior:

• Campaign launch. Due to an ordering error, there’s an oversupply of in-house branded turkeys. Therefore any query for "turkey" must boost the in-house brand.
• Product recall. A supplier has recalled a product line. Queries that would surface those products shouldn’t be shown.
• Seasonal reinterpretation. Queries for "stocking" are returning women's hosiery and tights. During the holiday season, "stocking" should resolve to Christmas stockings and stocking stuffers. Once the season ends, the policy can be reverted in minutes.

Under the traditional operating model, where search logic is embedded in application code, each of these changes requires an engineering ticket, a code change, a review cycle, a staging deployment, and a production release. In organizations with conservative release processes, that's a timeline measured in weeks, not hours or minutes. The Christmas shopping window closes before engineering can ship the necessary modifications.

The bottleneck isn’t the retrieval engine; it’s the operating model. The core challenge is that business intent cannot be translated into search behavior without engineering acting as a constant intermediary, turning every strategic pivot into a technical ticket.

The anti-pattern: Search logic in application code

Part 1 described how search logic embedded in application code can turn into a "spaghetti" implementation, which creates operational friction. Here’s what that friction looks like at scale. What starts as a few targeted overrides, a filter here, a boost there, grows over time into tens of thousands of lines of if/else branching, regex patterns, and conditional query modifications. This creates problems beyond just technical debt:

This model introduces four systemic frictions that hinder both organizational speed and system scalability:

Coupling. Business strategy changes daily. Application infrastructure should remain highly stable. When both live in the same codebase, a merchandiser's request to boost a seasonal product becomes a deployment risk, and a scoring function refactor can silently break a campaign.

Latency (organizational and computational). A single query behavior change can require a six-week deployment cycle: ticket, investigation, code change, review, staging, release. Furthermore, the application layer lacks any indexing mechanism to efficiently determine which policies apply to a given query, so policy evaluation often adds meaningful latency at query time as the system walks through sequential if/else checks.

Accountability gaps. When results change unexpectedly, nobody can quickly answer why. Was it a synonym update? A scoring change? A new filter added three releases ago? When business logic is distributed across thousands of lines of application code, shipped by different teams across different releases, tracing a relevance change back to its root cause becomes an archaeology project.

Misallocated engineering. This model turns skilled software engineers into full-time relevance mechanics. Instead of building platform capabilities, they spend their cycles translating merchandising requests into code changes and debugging interactions and conflicts between hard-coded business policies.

The paradigm shift: Policies as data

The solution is to decouple business policies from application code entirely. Instead of hard-coding query modifications in middleware, store governed policies as structured documents, each one expressing a discrete business intent, and evaluate them at query time in a dedicated governed control plane layer.

A policy is a first-class data object. It has match criteria (when should this policy fire?), an action (what should it do?), a priority (how does it interact with other policies?), and metadata (a title and a description). The control plane evaluates matching policies, resolves conflicts deterministically, and produces an execution plan including constraints, boosts, and routing decisions that Elasticsearch executes against a product catalog.

For each additional search requirement, the application code doesn't change. The retrieval engine doesn't change. What changes is that business decisions are no longer encoded in code. They live in a policy index as data that can be updated without a deployment.

This changes your org chart, not just your query.

Policies vs. triggers vs. rules

A note on terminology used in this series: a policy refers to this complete governed document, including a trigger (match criteria), rule (action), priority, enabled/disabled, and metadata. A trigger refers to the matching criteria that determines when this policy fires, and a rule refers specifically to the action inside the policy, such as applying a filter or changing the retrieval strategy.

The workflow: Author → Test → Promote

Moving policies out of code and into data opens the door for business-driven search management. But enabling non-technical teams to alter search behavior requires strict operational guardrails. The goal is fast and safe iteration with governance.

To empower non-technical teams to modify search behavior with confidence, we suggest a three-stage workflow: Author, Test, and Promote. Let’s examine the components of this workflow in detail.

Author. A merchandiser creates a policy using structured fields: what the policy should match, what action it should take, and at what priority. The interface guides the business user through what’s expressible.

Elastic Services has built and deployed a governed framework for enterprise ecommerce customers, which has an admin UI that looks as follows:

Test. The policy is validated in a non-production environment where the merchandiser can run representative queries and verify that the policy produces the expected behavior, including how it interacts with other active policies. Because the control plane infrastructure is identical across environments, what works in the test environment will work in production.

Review. Before a policy is promoted to production, it passes through review. Depending on the organization's risk tolerance, this might be a peer review from another merchandiser, an approval from a search lead, or an automated validation that checks for conflicts with existing policies.

Promote. Once approved, the policy is promoted to the production policy index. It takes effect on the next query: no code deployment, no engineering release, no staging build. The entire promotion is a data operation: the same JSON document, moved to a different index.

Disable. If a production policy produces unexpected behavior, it can be disabled immediately without engineering involvement. Disabling removes the policy from query evaluation instantly, without affecting any other policy in the system.

This is the "zero-deploy" promise. It doesn't mean "no process." It means the process operates on policy data, not application code. This distinction compresses the change cycle from weeks to hours or minutes.

Why "zero-deploy" matters for revenue-critical queries

The economics of ecommerce search are asymmetric. A small number of high-volume queries ("milk," "bread," "oranges," "diapers") drive a disproportionate share of revenue. When one of these queries returns unexpected results, the cost is immediate and measurable: Conversion drops, customer complaints spike, and the merchandising team opens an urgent ticket.

Under the traditional model, the response cycle is:

1. The merchant notices the problem.
2. The merchandiser files a ticket with engineering.
3. Engineering investigates, identifies the cause, and writes a fix.
4. The fix goes through code review, staging, and release.
5. Production is updated.

Depending on the organization, steps 2 through 5 may take weeks. For a revenue-critical query during a peak sales period, that latency costs money.

Under a governed control plane, the response cycle compresses:

1. The merchant notices the problem.
2. The merchandiser drafts a policy fix (or modifies an existing policy).
3. The policy goes through review and is published.
4. The fix is live.

The difference isn't just speed. It's ownership. The person closest to the business context (the merchandiser who understands why "oranges" should resolve to produce, not beverages) is the person making the change. Engineering is freed from the daily merchandising loop to focus on the platform. This shift also unlocks something that's nearly impossible under the traditional model: attributing search performance changes to specific business decisions.

Measurability: Which policy moved conversion

When policies are discrete, versioned documents that are stored in an Elasticsearch index, each one becomes independently deployable and therefore its impact can be more easily measured. You can answer questions that are nearly impossible to answer when business logic is scattered across application code:

• Did the "cheap laptops" price threshold policy improve conversion for that query class, or did it suppress it?
• What was the click-through rate impact of the holiday campaign boost?
• When we rolled back the "oranges" category constraint last Thursday, what happened to add-to-cart rates?

This turns search governance into a data-driven discipline. Instead of vague "relevance tuning," where a release contains a dozen changes and nobody can attribute the outcome, you get measurable, attributable impact per policy. Merchandisers can iterate with evidence. Engineers can evaluate whether a policy schema change produced the expected downstream effect. Leadership can see which policies are driving revenue and which are inert.

What this means for each role

For merchandisers and business users

Search behavior becomes something you can directly influence through structured policies without understanding Elasticsearch syntax or scoring algorithms. You can see what policies are triggered for a given query to understand why it produces specific results, and make changes within hours instead of weeks. The same policy mechanism also supports sponsored product placement: A merchandiser can create a boost policy that elevates a product or brand and flags it for a 'Sponsored' indicator in the UI, without requiring engineering involvement or additional infrastructure.

For search engineers

The control plane separates two concerns that are currently entangled: retrieval optimization and business logic. Instead of maintaining tens of thousands of lines of application code that encodes business decisions, you maintain the retrieval engine and the control plane infrastructure. When a merchandiser needs a new campaign boost, they don't need engineering to write it.

This doesn't eliminate engineering involvement. Engineers design the policy schema, maintain the control plane, set guardrails on what policies can express, add new capabilities as required, and handle edge cases that fall outside the policy framework. But the day-to-day operational cadence of modifying query behavior shifts to the people who own the business context.

For site reliability engineers and platform teams

Because policies are structured documents rather than application code, they fit naturally into existing operational workflows. Policies can be stored in version control, reviewed through pull requests, and deployed through the same continuous integration and continuous deployment (CI/CD) pipelines the team already uses. Conflicts between policies are detected and resolved deterministically at query time through the control plane's priority system, not through unpredictable interactions between code branches shipped in different releases.

When something does go wrong, diagnosing the cause is straightforward: Policies are discrete, named, and individually toggleable. A problematic policy can be disabled or deleted immediately without affecting anything else in the system. Compare that to debugging a relevance regression caused by an interaction between a synonym update, a scoring function change, and a new analyzer, all shipped in the same release with no clear attribution.

Beyond manual authoring: Large language model–assisted (LLM-assisted) policy suggestions

The policies described so far are authored by humans (a merchandiser identifying a gap and drafting a fix). But the same governed workflow supports a second mode: LLM-assisted policy suggestion.

An LLM can run offline or in the background, analyzing query logs, identifying patterns where search results underperform, such as queries with high exit rates, low click-through, or frequent reformulations. An LLM can then suggest new policies that enter the same Author → Test → Promote pipeline, where a human evaluates each one before it reaches production.

Governance is the enabler, not the constraint

It might seem counterintuitive: Adding a governance layer makes the system faster to change, not slower. This is the same pattern that works in other domains. CI/CD pipelines don't slow down software delivery; they make it safe to ship frequently. Access control doesn't slow down collaboration; it makes it safe to share broadly.

A governed control plane works the same way. The reason a query behavior change takes six weeks isn't that the code change is complex; it's that nobody is confident enough to ship it faster, because the blast radius is unclear and the rollback path is uncertain.

Governance provides that confidence. When every policy is explicit, every conflict is resolved deterministically, and every change can be instantly disabled and then rolled back (because policies are structured JSON documents that can be version controlled using existing workflows), the cost of iteration drops dramatically. Business teams move at the speed of the market. Engineering focuses on the platform.

From operating model to architecture

The shift from business logic in code to business policies as data is more than a technical refactoring; it's an organizational change that puts relevance ownership with the teams closest to the business context. But it raises an architectural question: How do you evaluate policies at query time without adding latency or turning the control plane itself into a new form of spaghetti?

The next post will dig into exactly that: the design pattern that enables fast, deterministic policy evaluation at query time.

Put governed ecommerce search into practice

The workflow described here, merchandisers authoring, testing, and promoting search policies without engineering deployments, is already available. Elastic Services Engineering designed and built it, and Elastic Services has the skills to deploy it for enterprise ecommerce teams.

If your organization is ready to move from deployment-gated relevance tuning to business-editable search with governance and auditability, we can accelerate your implementation. Contact Elastic Professional Services.

Join the discussion

Have questions about search governance, retrieval strategies, or ecommerce search architecture? Join the broader Elastic community conversation.

Here, we’ll provide some intuition about the problem and how preconditioning solves it.

The problem

BBQ quantizes each dimension of a vector independently: Values above the mean become 1, values below it become 0. This works well when every dimension carries roughly the same amount of information. Transformer-based embeddings tend to have this property naturally such that their dimensions are learned representations that distribute variance evenly.

But there are lots of real-world vectors that aren’t like this. Consider a 784-dimension vector representing a grayscale image, like in the Fashion-MNIST dataset. Some pixels near the center of the image, where the clothing actually appears, vary a lot across the dataset. However, other pixels, such as those near the corners, are mostly one color and barely vary at all. When BBQ quantizes these vectors, the high-variance dimensions lose precision because a single bit can't capture their range, while the low-variance dimensions become useless. The resulting quantized vectors are poor approximations of the originals, and recall suffers.

Picture of a representation of Fashion-MNIST images. (credit: geeksforgeeks.org)

Precondition

To fix the problem, we want to spread the information more evenly across dimensions so that each bit captures roughly the same amount of information.

Preconditioning applies a linear transformation to every vector before quantization. The transformation is an orthogonal rotation that reshuffles how information is distributed across dimensions without changing the distances between vectors. If you want to dig into the math, take a look at this in-depth analysis on optimized scalar quantization (OSQ) with preconditioners.

Here’s a graphic to help illustrate how preconditioning can help when applying quantization. This simplified two-dimensional diagram illustrates the idea that the orthogonal rotation helps to increase the spread, or range, of information that was previously quite compressed. While this two-dimensional animation is not an exact representation of preconditioning, it gives a good intuition for what roughly happens in higher dimensions where buckets of dimensions are transformed independently and a random projection can greatly improve the distribution. Imagine that the y-axis represents one pixel of our Fashion-MNIST corners that are primarily one shade with very low variance and the x-axis represents a pixel of clothing at the center of the image with very high variance. Without preconditioning, quantizing vectors to a single representative point is not a particularly good discriminator.

Let’s look at the data

Today, preconditioning is supported in DiskBBQ. Here’s a benchmark showing the impact when visiting different percentages of the total vector dataset.

Fashion-MNIST Recall (784 dimensions, 60K docs, 5x oversample, k: 10)

GIST (960 dimensions, 1M docs, 5x oversample, k: 10)

SIFT (128 dimensions, 1M documents, 5x oversample, k: 10)

That’s a nice boost in recall; however, this boost comes with a cost. Applying it to all embeddings blindly is inefficient, causing ~2–4% overhead in query latencies with no improvement in recall for datasets that don’t need to be preconditioned. And upwards of 20% additional overhead at index time. For production use cases where you see initially low recall, you may want to evaluate the impact of preconditioning with your specific model and dataset.

Here’s the how

Preconditioning is available for the bbq_disk index type. Simply set precondition to true in the index_options, like so:

{
  "mappings": {
    "properties": {
      "my_vector": {
        "type": "dense_vector",
        "dims": 784,
        "index_options": {
          "type": "bbq_disk",
          "precondition": true
        }
      }
    }
  }
}

Take a look at the dense vector mapping docs for more details.

Conclusion

BBQ is highly effective for deep learning embeddings, but it can be less effective with embeddings that have uneven variance across dimensions, as can occur in feature-engineered vectors. Preconditioning redistributes that variance so quantization can be more effective. On some datasets, like Fashion-MNIST, we see as much as a 74% improvement in recall!