A context engine understands your data ahead of the question. It reads your sources and stores what it learns as small structured units of context, so when a task arrives, an agent gets the relevant ones and shortcuts discovery. We built one on Elasticsearch, and every stage of it turned out to be a search problem. Automations search and precompute knowledge, storing it in a specialized AI index with native hybrid search and retrieval enabled over it. With a context engine, agents pull what they need rather than digging through raw data. In an example, with the same model and data, tool calls dropped from 33 to 9, and input tokens fell 71%. Plus, latency went from 213s to 64s, and the agent’s response was more complete, surfacing relationships that would otherwise be missed.

What an agent does without a context engine

Ask a data agent a strategic business question like: Give me a summary of the issues hitting our largest customers. Without help, a capable agent starts by listing indices to see what data exists. It reads mappings to work out which fields mean "customer" and which mean "issue." It scans contracts to figure out account value and find the largest accounts. Along the way, it misses that five accounts roll up under the same parent company, so it undercounts one of your biggest relationships. It also doesn't know that account_id is sometimes null in the tickets index, so part of its query quietly returns the wrong rows.

The model's reasoning was fine, but the agent spent most of its budget working out what the data is before it could reason about what the data says, and it will repeat all of that on the next question.

Provided with the right context however, the agent can be much more efficient. A context engine built on Elasticsearch precomputes knowledge from the raw data sources like tickets and contracts, and surfaces it to the agent upfront so it avoids the costly steps before actually reasoning about an answer. The table below demonstrates this benefit for that business question above, with the same system prompt, model, and data. The only difference is whether the agent had a context engine to read from. We’ll explore what it means to build a context engine and how it provided this benefit through this post. 

Same prompt and model*

Without Context Engine

With Context Engine

Tool calls

33

9

Input tokens

2,279,253

653,511

Latency

213s

64s

Found parent-company relationship

No

Yes

Cost

Baseline

71% lower

Across multiple dimensions, Context Engine improves agent performance. The agent reduces tool calls, tokens, and latency, while finding more complete answers. 

* Runtime model: claude-sonnet-5, using LangChain Deep Agent Harness, precompute ~636K input tokens with claude-haiku-4-6.

Why AI agents need a context layer

Most context engineering work so far has been about what goes into one agent's context window: prompts, tool definitions, conversation history, compaction. That work matters, but when we worked with customers building on Elastic Agent Builder, the context that decided whether an agent succeeded usually wasn't in the conversation at all. It was business knowledge, like what each data source is for, how data sources relate across systems, what trends or anomalies exist in the data, and how to build efficient queries against that data.

When there's nowhere to keep that knowledge, the same problems show up in production, including:

• Per-query discovery cost. Agents pay a discovery cost on every query, spending tokens to work out what data exists, what fields mean, and how sources relate, even when nothing has changed.

• Context drift. Whatever context someone does write down in markdown files or catalog entries drifts out of date as data and usage change.

• Siloed knowledge. What one agent learns never helps another, so quality doesn't compound and costs don't fall. Plus, the gap between demo and production stays open.

What a context engine does

A context layer sits between your raw data and your agents. It does the work of understanding your data ahead of time and stores the results as small structured units of context. Then hands agents the relevant ones when they need them.

We call these units Knowledge Indicators (KIs). A KI might describe:

• An entity and its relationships, such as "LongHaul (ACC-1007) is a subsidiary of Orion Holdings".

• A data source profile, such as "the tickets index holds customer support issues; search it semantically on the issue description".

• A data quirk, such as "account_id is not always present, so filter out nulls".

• A fact, anomaly, or pattern that would otherwise take a full scan to find, such as “what is the peak and median query load over the last 30 days”. 

The layer creates and serves KIs in a continuous loop:

1. Gather. Read each source, and learn its shape. Find the content that matters for what your agents need to do.

2. Build. Turn that content into KIs (metadata, entities, relationships, anomalies, facts, and query guidance), and store them.

3. Retrieve. When an agent gets a task, it pulls a small, ranked set of relevant KIs instead of doing schema discovery and document scans itself.

4. Improve. Every agent interaction produces a trace, and traces feed back in as a new source, so context keeps up with how agents are actually used.

Why every stage of context engineering is a search problem

Search usually comes up in these discussions as the retrieval step at the end. In practice, each stage of the loop depends on finding the right information.

Stage

The search problem

Gather

You're working with gigabytes to petabytes of data and can't reprocess all of it continuously. Narrowing to what matters takes high recall and high precision: agentic search to understand intent, lexical search for precision, semantic search for recall, and workflows to automate it.

Build

KIs precompute what an agent would otherwise work out at runtime. What to keep and what to throw away depends on how the KI will be retrieved, so KIs need embeddings, metadata fields, and combined fields designed for retrieval.

Retrieve

What goes into the context window has to be relevant and complete, as well as concise. If it's noisy, the agent ignores it and goes back to exploring the raw source. Tuning this is a relevance problem.

Improve

Traces and transcripts are unstructured. To surface recurring failures and the patterns they share, you need hybrid search.

We built the context layer on Elasticsearch because its stages rely on hybrid search, Elasticsearch Query Language (ES|QL), and vector search, often within the same request. Earlier blog posts covered each of these individually, and this post shows how Context Engine runs them together.

How this compares to approaches like agentic RAG and longer context windows

Context engineering isn’t a new practice, and there are many existing approaches to help manage the same challenges. Creating context, through the loop above, augments and helps address many of the tradeoffs and challenges with existing approaches today. 

Approach

What happens at query time

Tradeoff

Static context files or catalogs

Agent reads hand-written descriptions

Goes stale and doesn't learn from use

Agentic retrieval augmented generation (RAG)

Agent searches raw chunks and loops until satisfied

Reassembles knowledge on every task and misses relationships that span documents

Coding agent with grep

Agent navigates files directly

Slow and token-hungry; struggles to tell which match is the right one

Longer model context window

Agent loads data directly into the context window to reason over

Longer context windows struggle with problems of context rot and attention, and as context builds, each subsequent turn becomes more expensive. 

A context layer trades the cost of precomputing and processing data for the benefits of improved efficiency at runtime. Because precomputation can be defined ahead of time, it can be more prescribed and use smaller, cheaper models, making this tradeoff much more manageable while addressing many of the efficiency and effort concerns of the approaches above. 

Building a context engine

Context Engine consists of five components, as shown in the following image: sources, automations, AI index, agent memory (agents), and the feedback loop.

Within Elastic, we’re developing a context engine to deliver the boosts in accuracy and efficiency that agents need. Let’s explore each component in detail.

Sources

Context can come from any Elasticsearch index or data stream, so you build it where your data already lives. It can also come from external applications and data sources, through connectors, and from agent traces captured in Agent Builder or sent from other agent frameworks over OpenTelemetry (OTel).

How automations build context

Automations populate the AI index. They're Elastic Workflows with specialized steps that search raw content, process results with a large language model (LLM) or other models, structure the output for retrieval, verify provenance, apply access control, track state, and write KIs to the AI index.

You don't have to write them from scratch. A setup agent looks at your connected sources and suggests automations. They can run on small, inexpensive models or plain ES|QL, which keeps build costs low while the savings show up on every query afterward.

The AI index, built on the Elasticsearch vector database

The AI index is the context store. It runs on Elasticsearch as a vector database and includes ES|QL and hybrid search. It holds the KIs that automations produce. Its schema and mappings are designed for agents to retrieve from.

How agent memory works

As users complete tasks, agents can store and retrieve memories with remember, recall, and forget tools. Memory consolidation and organization-wide sharing are next on the roadmap.

How agent traces feed back into context

Every agent interaction produces a trace that records how the agent worked through the task, including where it succeeded and where it got stuck, in addition to where it spent tokens. The feedback loop looks for inefficiencies in these traces and suggests changes to your automations, so KIs improve from real usage as well as from any test set you wrote up front.

From raw data to a better answer with a context engine

Here’s how the context engine was used to achieve the results at the top of this post. We evaluated a simulated account analysis agent on questions like the one above. It had to pull from several sources:

• Four Elasticsearch indices holding customer tickets, subscribers, an internal knowledge base, and support page search query logs.

• A Google Drive folder with about 50 simulated contracts between the enterprise and its customers or vendors.

Automations created two kinds of KIs from these sources. Index profiles describe what each source is for and how to query it. Account entities pull accounts, projects, and people, along with their relationships to other entities, out of raw contract and ticket text. That produced about 80 KIs, each holding the extracted content plus metadata that tracks provenance, lifecycle, and how to query the context. The automations ran on smaller, cheaper LLMs, so extraction cost stayed low.

We connected the AI index to two agent frameworks, Elastic Agent Builder and LangChain Deep Agents, both over Model Context Protocol (MCP).

How the baseline agent spent 33 tool calls

By using the following request, Give me a summary of the issues hitting our largest customers, the baseline agent:

1. Ran discovery, looking at each data source, its fields, and sample documents to learn the structure.

2. Worked out what "customer" means in the data (there are subscribers and contracted accounts) and how to define "largest" (there are fields for annual contract value [ACV] and monthly recurring revenue [MRR]).

3. Queried the tickets index to find problems.

4. Pulled the full text of every contract that mentioned an affected customer, looking for relationships to other organizations.

The discovery and contract reads are the expensive part, and they account for most of the token cost in the baseline run.

What changed with the context layer

With the AI index, the agent started with knowledge that automations had precomputed. It already knew which fields identify customers and which define the largest ones. The entity KIs also captured relationships from the raw contract text, such as one parent company sitting over several smaller ones. Reading the KIs up front let the agent skip the expensive queries and full document scans, and run lightweight checks against the original data instead. As a result, the run used 71% fewer input tokens, and the context window stayed free of noise.

Why context engineering for agents is a search problem

The first wave of context engineering was about the context window: what to put in the prompt and what to compact, along with what to drop. That's still important, but for agents that work across enterprise data, the harder problem is building the context in the first place. Business knowledge has to be gathered, shaped, kept current before a conversation starts, and shared across agents so each one doesn't rebuild it.

A context layer does that job, and it changes how agents work. Agents get consistent guidance on where to look and how to query, so answers still come from live data without the dead ends. Context refreshes from the source, so it adapts to changes in data, user behavior, and use cases without someone maintaining it by hand. Every agent run also shows what was missing, so the knowledge base gets better with use and every agent that shares it benefits.

All of this depends on search. Finding the signal in large datasets, shaping context so it can be found again, retrieving only what's relevant, and spotting patterns across thousands of traces are all search problems. If you're designing the context layer for your agents, build it on search.

We're actively partnering with developers who want to optimize their agents in production. If you're interested in exploring Context Engine for your agents, you can request access here.

ECK's defaults secure every cluster with Transport Layer Security (TLS) and rotate their own certificates, which works well until your security policy says that every certificate must come from your enterprise PKI. At that point, you need central control over issuance and trust, and you still don't want to renew certificates by hand. Vault and cert-manager give you both.

Self-signed certificates that ECK generates by default

The ECK operator generates self-signed certificates for various components of the Elastic Stack. Below is a high-level overview of the different types of self-signed certificates generated by ECK for Elasticsearch and Kibana:

Where should your CA private key live?

Using custom certificates with ECK introduces a tradeoff between the risk of exposing a CA private key and automating certificate lifecycle management.

Bring your own CA and private key to ECK

The ECK operator automates certificate issuance and renewal when provided with the CA certificate and private key through a Kubernetes Secret. ECK then manages the certificate lifecycle across the Elastic Stack, including both HTTP and transport certificates for Elasticsearch. However, storing the CA private key within the Kubernetes cluster is considered a security risk and may not align with enterprise security and PKI policies.

Use manually issued certificates with ECK

ECK can consume externally issued certificates provided through Kubernetes Secrets, keeping the CA private key outside the Kubernetes cluster. However, certificate issuance and renewal must be managed separately, as should certificate rotation. This creates an operational bottleneck, particularly for transport certificates, which need to be provisioned dynamically as Elasticsearch pods scale up or down.

Use Vault PKI with cert-manager

Cloud-native tooling removes the tradeoff between bringing your own CA and key and using manually issued certificates. HashiCorp Vault and cert-manager move certificate lifecycle management on Kubernetes out of  the ECK operator, while keeping the enterprise CA private key secure.

• HashiCorp Vault is the external PKI system, securely managing the CA private keys and issuing certificates signed by the enterprise CA. Certificate signing operations stay secure without ever exposing the CA private key as a Kubernetes Secret.

• cert-manager can automate the certificate lifecycle within Kubernetes. It can request certificates from Vault and monitor their validity. It can also automatically renew them before expiration. Once issued, cert-manager can create and update the corresponding Kubernetes Secrets which, in turn, are consumed by Elastic components.

Using Vault PKI with cert-manager gives each component a single responsibility: Vault manages the trust and signing authority, and cert-manager manages the certificate lifecycle. ECK components consume the resulting certificates. 

Vault and cert-manager let organizations keep their existing enterprise PKI controls while reducing the operational overhead associated with certificate issuance, renewal, and rotation.

The table below compares the three approaches:

Approach

Where the CA key lives

Certificate lifecycle

Transport certificates as pods scale

Fit with PKI policy

Bring your own CA and key

Kubernetes Secret

Automated by ECK

Automatic

Depends on whether policy allows CA keys in the cluster

Manually issued certificates

Outside the cluster

Manual

Manual

Keeps the CA key outside the cluster

Vault PKI with cert-manager

Vault

Automated by cert-manager

Automatic via the cert-manager CSI driver

Keeps the CA key outside the cluster

ECK certificate management architecture

The following diagram illustrates the proposed architecture for the solution:

Prerequisites and tested versions

This blog assumes that the user has a basic understanding of the Vault and cert-manager setup. In our post, we focus only on the configuration required to implement this setup.

Refer to the official Vault and cert-manager documentation for installation instructions related to Vault, cert-manager, and csi-driver.

Test versions

We used the following software versions for the demonstration in this blog. For compatibility considerations or configuration differences with other versions, refer to the respective official documentation.

• Elastic Stack 9.5.3

• cert-manager 1.21.2

• HashiCorp Vault 2.0.3 (Community Edition)

HashiCorp Vault as your enterprise PKI

On the Vault server, the PKI secrets engine needs to be enabled to securely store CA certificates and its private key that’s used to sign leaf certificates.

As a best practice, the Root CA private key remains outside Vault. The Root CA is used to sign the Intermediate CA certificate signing request (CSR), after which the Intermediate CA certificate and private key are stored securely in Vault. Vault then uses this Intermediate CA and key to issue and sign leaf certificates.

In addition to the PKI secrets engine, the Kubernetes authentication method is also enabled so Vault can authenticate requests originating from Kubernetes. A dedicated Vault role is configured with narrowly scoped permissions, ensuring that it issues certificates only for requests from the specified Kubernetes ServiceAccount and namespace.

At a high level, the following tree structure illustrates the Vault components required for this setup:

Configure Vault PKI and Vault Kubernetes auth

1. On a Linux host that has access to the Vault command line interface (CLI), generate a root CA certificate and key:

NOTE: This is used for demonstration purposes only. In a production implementation, skip the above step and use your organization's existing enterprise CA and key instead.

2. Enable the PKI secrets engine at the eck/esintca path, and configure a maximum certificate lease duration of one year. This PKI engine is used to store the intermediate CA and issue certificates for the Elastic Stack components:

3. Generate an internal Intermediate CA key and CSR using the eck/esintca PKI engine. The generated response includes the CSR and is stored in eck_esint.json for use in the subsequent signing step:

4. Extract the Intermediate CA CSR from the Vault response, and save it as eck_esint.csr. This CSR is submitted to the Root CA for signing.

5. Create an OpenSSL extensions configuration file defining the Intermediate CA attributes. The configuration marks the certificate as a CA and limits the certificate path length to zero. It also enables the required key usage and certificate identification extensions. This configuration is used when signing the Intermediate CA CSR with the Root CA.

6. Sign the Intermediate CA CSR using the Root CA certificate and private key to generate the Intermediate CA certificate. The certificate is valid for 365 days and uses SHA-256, with the CA extensions defined in eck-ext.cnf. The resulting eck_esint.crt is used by Vault to issue and sign leaf certificates.

7. Combine the Intermediate CA certificate with the Root CA certificate to create the complete CA certificate chain. Vault uses this chain when issuing certificates so that clients can trust the chain back to the Root CA.

8. Upload the signed Intermediate CA certificate chain to the Vault PKI engine. This completes the Intermediate CA configuration and enables Vault to use it for issuing certificates signed by the Elasticsearch Intermediate CA.

9. Configure the Vault PKI engine with the URLs used to retrieve the issuing CA certificate and Certificate Revocation List (CRL). Replace <replace-with-vault-server-endpoint> with the Vault server endpoint specific to your environment. Clients use these URLs to retrieve the CA certificate and to check certificate revocation status when required.

10. From the Linux host that has access to Kubernetes cluster, retrieve the Kubernetes API server endpoint from the current kubectl context. This endpoint is used to configure Vault's Kubernetes authentication and to allow Vault to communicate with the Kubernetes API server for ServiceAccount token validation.

11. Extract the Kubernetes API server's CA certificate from the current kubectl context, and decode it into kubeapi-ca.crt. Vault uses this CA certificate to establish a trusted TLS connection with the Kubernetes API server.

NOTE: If the same Linux host has access to both the Kubernetes cluster and the Vault server, run the following commands from that host. Otherwise, copy the Kubernetes API server endpoint, audience url, and CA certificate to the host from which the Vault server is accessible, and run the commands there.

12. Enable the Kubernetes authentication method at the eck path. Vault then authenticates requests from Kubernetes workloads using their ServiceAccount tokens.

13. Configure Vault with the Kubernetes API server endpoint and its CA certificate. Vault uses the CA certificate to establish and verify a trusted TLS connection to the Kubernetes API server.

14. Create a Vault PKI role that defines the domains and certificate parameters allowed for Elasticsearch certificates. The role permits the specified Kubernetes service domains and subdomains, and it accepts the common name (CN) and subject alternative names (SANs) from the CSR. It also limits issued certificates to a maximum time to live (TTL) of 168 hours (seven days).

Note: In this example, the Elasticsearch cluster is named elasticsearch and is deployed in the elastic-cluster namespace. Therefore, the Elasticsearch HTTP service is referenced as elasticsearch-es-http and the domain as elastic-cluster.svc. Replace the cluster name, namespace, and corresponding service name with the values applicable to your environment.

15. Create a Vault policy that grants permission to use the elasticsearch PKI signing role. The policy provides only the required update capability on the certificate signing endpoint, following a least-privilege approach.

16. Create the eck-es-policy policy in Vault using the previously defined policy file. This policy is associated with the Kubernetes authentication role and grants the permissions required to request Elasticsearch certificates.

17. Create the eck-es-role role, and bind it to the cert-manager-vault-access ServiceAccount in the elastic-cluster namespace. The role uses the specified Kubernetes token audience and attaches the eck-es-policy, so the ServiceAccount can authenticate to Vault and request Elasticsearch certificates. The authentication token is configured with a 10-minute TTL.

Note: In the Kubernetes cluster, ensure that Vault authentication requests originate from the elastic-cluster namespace and use the cert-manager-vault-access ServiceAccount, along with its associated token.

Automate certificate renewal with cert-manager

cert-manager is used to manage the lifecycle of certificates required by the Elastic Stack. It monitors Kubernetes Certificate resources and requests certificates from the configured Issuer, which, in this solution, is backed by Vault.

cert-manager authenticates to Vault using a dedicated Kubernetes ServiceAccount. Vault then validates the identity of the requesting workload and authorizes certificate issuance based on the configured Vault role and policy.

Once Vault issues the certificate, cert-manager creates and maintains the corresponding Kubernetes Secret containing the certificate and private key. cert-manager also monitors certificate validity and automatically manages renewal and rotation before certificates expire.

Issue transport certificates with the cert-manager CSI driver

For Elasticsearch transport certificates, the cert-manager CSI driver provides certificates directly to the Elasticsearch pods. The CSI driver requests certificates and mounts them directly into pods through the Kubernetes CSI interface, so the certificate and private key never need to be persisted as a Kubernetes Secret.

With the CSI driver, cert-manager still manages the certificate lifecycle and obtains certificates from the Vault-backed Issuer. The CSI driver handles the delivery of these certificates to the Elasticsearch pods, where they can be used for secure transport communication between Elasticsearch nodes.

Create the Vault Issuer and HTTP certificate

1. On the Kubernetes cluster, create the elastic-cluster namespace and a dedicated cert-manager-vault-access ServiceAccount within it. This ServiceAccount is used to obtain the Kubernetes token that cert-manager presents to Vault for authentication.

2. Create the required role-based access control (RBAC) permissions in the elastic-cluster namespace to support Vault authentication. The Role and RoleBinding allow the cert-manager ServiceAccount to request a token for the dedicated cert-manager-vault-access ServiceAccount. The ClusterRoleBinding lets Vault use this ServiceAccount to validate Kubernetes ServiceAccount tokens through the Kubernetes API, so Vault can verify the identity of the requesting workload.

3. Generate a token for the default ServiceAccount, and decode its JSON Web Tokens (JWT) payload to inspect the token claims. This is used to identify the token audience (aud) when configuring Vault's Kubernetes authentication.

4. Create an Issuer object that connects cert-manager to the Vault PKI signing endpoint. Configure the Vault server endpoint and PKI signing path, along with the Vault server CA certificate, so cert-manager can establish a trusted connection to Vault. The Kubernetes authentication configuration specifies the eck-es-role Vault role and the cert-manager-vault-access ServiceAccount. cert-manager uses this identity and audience to authenticate to Vault and to request Elasticsearch certificates. In the YAML below, replace the Vault server endpoint and CABundle, as well as the audience generated in the previous step, and save it in the issuer.yaml file:

Apply the manifest file to create the object:

5. Create a cert-manager Certificate resource in the elastic-cluster namespace to request the Elasticsearch HTTP certificate from the eck-es-issuer Issuer. The certificate is configured with the required domain name system (DNS) names for Elasticsearch service access and a seven-day validity period, in addition to renewal 60 minutes before expiry. cert-manager stores the generated certificate and private key in the eck-es-http Kubernetes Secret, using an RSA 2048-bit private key encoded in Public-Key Cryptography Standards #1 (PKCS#1) format.

6. Verify that cert-manager has successfully processed the Certificate resource and created the corresponding Kubernetes Secret. The Secret should contain the certificate and private key generated for the Elasticsearch HTTP endpoint. The Certificate should show a READY status of True, indicating that the certificate was successfully issued and the Secret was created.

Make your custom CA available to ECK

Vault and cert-manager already provide the ca.crt file needed for transport trust between Elasticsearch nodes, mounted through the CSI driver. However, in scenarios where the CA changes frequently or remote cluster setup requires the CA to be trusted across clusters, the custom CA also needs to be made available as a ConfigMap in the elastic-cluster namespace so that the ECK operator itself is aware of it.

There are two options for providing this ConfigMap.

Sync the CA automatically with trust-manager

trust-manager is a Kubernetes operator that keeps trust bundles up to date automatically. It adds a Bundle custom resource that reads the CA from a source, such as a Kubernetes Secret, and publishes it into a ConfigMap, keeping it in sync whenever the source changes.

Install trust-manager with its source secret namespace set to elastic-cluster so that it can read the eck-es-http Secret used in the Bundle below. Refer to its official documentation for more information.

Create the CA ConfigMap manually

The eck-es-http secret created earlier by cert-manager already contains the CA certificate under its ca.crt key. You can extract it directly and use it to create the ConfigMap.

Deploy Elasticsearch with enterprise PKI certificates

Deploy the Elasticsearch cluster using the following ECK configuration. The cluster is configured to use the certificates provisioned through the previously configured cert-manager and Vault integration rather than relying on ECK's default self-signed certificate management. The HTTP certificate is provided through the Kubernetes Secret created by cert-manager, while the Elasticsearch transport certificates are delivered through the cert-manager CSI driver.

Because the transport certificates arrive through the CSI driver as mounted files rather than as a Kubernetes Secret, the configuration below points xpack.security.transport.ssl.key and xpack.security.transport.ssl.certificate directly at those file paths:

Verify that Elasticsearch is using your enterprise PKI certificates

Verify that Elasticsearch is using the custom certificate issued through the Vault-backed cert-manager configuration. First, retrieve the certificate from the Kubernetes Secret created by cert-manager, and then inspect its subject and issuer, along with its validity period.

The output should show the expected Elasticsearch service name in the certificate's CN/SANs, the Vault-managed Intermediate CA as the issuer, and the certificate's notBefore and notAfter values.

Verify that Elasticsearch is presenting the certificate issued through the Vault-backed cert-manager configuration. The output should show the expected Elasticsearch service name in the certificate details and the Vault-managed Intermediate CA as the issuer. It should also show the certificate's validity period.

Use the following command to verify the health status of the Elasticsearch cluster:

Extend ECK certificate management across the Elastic Stack

The approach we described in this blog post can be summarized in the following diagram:

We focused here on configuring custom certificates for Elasticsearch. The same approach can be extended to other Elastic Stack components, such as Kibana, by using HashiCorp Vault and cert-manager to manage their certificate lifecycle.

Replacing ECK's default self-signed certificates with certificates backed by an enterprise PKI can be implemented using cloud-native tooling, such as Vault and cert-manager. The configuration we’ve demonstrated here provides a practical starting point that can be adapted to different enterprise PKI environments and certificate requirements across the entire ECK stack.

Try the approach in your own environment, starting with a non-production ECK cluster, and explore how it can fit into your organization's existing certificate management and security practices.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Thin shards write Lucene (the search library that Elasticsearch uses for indexing and search) segments locally and upload them to the object store as batched compound commits. Then they delete them from disk.

A thin shard that isn't ingesting and isn't expected to do so in the near future still holds an IndexWriter, segment readers, mappings, and the rest of its engine. At data stream scale, those idle engines make up most of the indexing heap and add time to the relocations that we need for balancing and for shutting nodes down during autoscaling.

A hollow indexing shard is a thin allocated primary that keeps only the last commit's metadata and doesn't create its IndexWriter or segment readers until the next write. Search never used them on the indexing node anyway.

Why do idle indexing shards use memory in Elasticsearch Serverless?

Elasticsearch Serverless stores Lucene commits and translogs in the object store, along with cluster state. Indexing nodes and search nodes are separate tiers. Durability no longer depends on replica shards, and relocating an indexing shard doesn’t copy files from node to node. The Symposium on Cloud Computing (SoCC) paper covers that architecture.

After the files left the node, Lucene's in-memory view of them stayed:

• The standard IndexEngine holds an IndexWriter (even with an empty RAM buffer) and a DirectoryReader with its SegmentReaders opened so the writer can see existing documents for updates and deletes, in addition to version checks.

• It kept per-index mappings and field infos. It also kept shard commit state.

Heap dumps of idle primary shards showed most of the retained memory in the engine's segment readers, on the order of a few megabytes each. The writer and mappings were a smaller slice. A hollow engine cut that footprint, and the node's retained heap fell with it.

On projects with heavy data stream usage and many backing indices, that’s the difference between the indexing node you need while idle and the one a live engine formula predicts.

Ingest autoscaling still sizes the indexing tier from memory formulas that assume a live engine. Hollow shards exist so we can stop assuming that, once those formulas catch up.

How hollow shards differ from fully materialized indexing shards

A fully materialized indexing shard runs an IndexEngine. A hollow shard runs a HollowIndexEngine over the same Lucene files, with no search or write path. Table 1 compares the two:

IndexEngine (fully materialized)

HollowIndexEngine (hollow)

Lucene IndexWriter

Open

None

Segment readers

Open

None

Accepts writes

Yes

No; ingestion is blocked until the shard is unhollowed

Local search

Theoretically possible

Not possible

Merges

Run on the indexing node

Require unhollowing first

Engine stats

Computed live

Served from stats stored in the hollow commit

Recovery

Translog replay and cache prewarm

No replay, no prewarm, no blob store LIST

Translog node ID in commit

Set

Empty (this is the hollow flag)

Table 1. IndexEngine versus HollowIndexEngine on an indexing node.

When we hollow, we flush everything to Lucene, commit, and upload to object storage with a hollow flag. We reuse an existing metadata field, the "translog node ID," and leave it empty. Empty means that this is a hollow commit and that recovery has no translog to replay, so it opens a HollowIndexEngine instead of the standard IndexEngine.

Having no readers makes the HollowIndexEngine a useful trip wire if some call site tries to search on the indexing tier. It also throws if indexing is attempted against it, but ingestion is gated before it reaches the engine, so we can unhollow the shard and reload an IndexEngine.

Search routing is unchanged; and queries continue to run on the search tier. Indexing routing still points at this primary. Since ingestion is blocked, a write queues until we unhollow.

When Elasticsearch Serverless hollows a shard

A shard can be hollowed only when all of these hold:

• The index isn’t a system index.

• It has been idle long enough for its type. (A regular index or the current write index of a data stream needs to be idle for one day by default, while older data stream backing indices need to be idle for 15 minutes.)

• There’s no in-flight ingest or translog upload.

• No merge is running or scheduled.

Why the first write to a hollow shard is slower

The first write is slower than a write against a warm engine because we unhollow first. We get less idle heap and faster shutdowns, and we accept a hitch on the first bulk to a cold backing index. On data streams that hitch almost never happens, because rolled-over backing indices aren’t the write index. They sit hollow until someone runs an update-by-query or a similar correction.

Hollowing a shard during  relocation

We only hollow a shard when it relocates. Relocations already block ingestion and already flush, so we reuse that window.

On the source node:

• Ingestion is blocked (for the relocation).

• If the shard can be hollowed, we force-flush a hollow commit and upload it. If it was already hollow, we skip the flush.

• The target recovers that commit and sees the empty translog node ID (the hollow flag). It then loads a HollowIndexEngine.

• We keep ingestion blocked on the target so the first write cannot sneak into the hollow engine.

Hollow recovery skips translog replay and cache prewarm because an idle shard doesn’t need them. If an indexing node restarts, a hollow commit on the object store comes back as a hollow engine.

How the first write unhollows a shard

The hollow engine cannot index. The first ingest, or a force-merge, has to put an IndexEngine back.

Incoming writes wait behind the ingestion block and trigger one asynchronous unhollowing that:

• Runs the recovery work we skipped while hollow, including blob cache prewarming.

• Closes the HollowIndexEngine and opens an IndexEngine, which is where Lucene's IndexWriter and DirectoryReader get created.

• Skips translog replay. A hollow commit is clean; there’s nothing to replay.

• Force-flushes a non-hollow commit, stamped with this node's translog ID, so a crash before the next regular commit upload still knows where the translog lives.

• Releases the ingestion block. The waiting ingest continues against a real writer.

Reads don’t unhollow, and in general, they don’t come to the indexing tier. Real-time gets can be answered from the search tier. The data is already in the last hollow commit.

Force-merge is the other explicit materialization point. Serverless does not expose force-merge as a user-facing API, but we still use it internally; for example, on data stream backing indices. A merge needs an IndexWriter, so we unhollow and do the work. We let the shard go hollow again on a later relocation.

Swapping the engine on a live shard

The hardest part was an invariant that Elasticsearch has had for a long time; that is, once an IndexShard is started and live, the engine it opens is the engine it keeps.

There was no concept of changing the engine type under a started shard. Hollowing and unhollowing have to swap the engine on a live shard. Call sites that do getEngine().foo() assume the reference that they just got will still be open a few lines later. Two failure modes showed up immediately:

1. A single engine call in the middle of a reset (for example, AlreadyClosedException on a real-time get).

2. A call site that captures the engine and invokes it more than once (flush, block ingestion, and then flush again) across a swap.

We shipped a new resetEngine() on IndexShard that takes the shard and engine mutexes, plus a withEngine read/write lock so reset is the only writer. Getting there meant chasing deadlocks through refresh listeners, which now receive the reader they need without re-entering the engine, and putting withEngine guards on the call sites that assumed the engine couldn’t vanish. The result is fenced and a bit ugly.

Keeping stats and recovery off the Lucene readers

Telemetry would have reopened the readers that we had dropped. Doc stats wanted to open a searcher. After we hollow a shard, we compute the basic stats and store them in the hollow commit. We serve them from there, with no reading on the indexing node. 

Recovering a hollow commit used to LIST the blob store and prewarm cache chunks for files that the hollow engine will never read. We skip LIST now and keep the minimum file chunks inside the hollow commit. We skip prewarming unless we’re unhollowing.

Current limitations of hollow shards

There are two operational risks that we designed for and have not fully taken on yet: 

• Autoscaling still counts hollow shards as full when it sizes the indexing tier, which is conservative. Later, we’ll charge them for the small footprint they actually have and keep a reservation for shards that might unhollow.

• The related problem is a burst. After the autoscaling formulas tell the truth, a large update that wakes many backing indices at once could try to materialize more heap than the node has. We plan to throttle concurrent unhollowings against reserved heap.

Shutdown times and memory usage in production

In the first month after the rollout, fleet-wide indexing-node shutdown times improved at high percentiles by up to 30%, as Figure 3 shows. Unhollowings stay rare, while hundreds of thousands of shards have aged into a hollow state.

Indexing tier heap usage also dropped across the fleet, though we’re still working out how to publish this footprint to the autoscaler. The stability effect that we actually felt was shutdown time. Nodes finish leaving the fleet faster, which is what autoscaling needed.

On one project, shutdown times and indexing tier heap improved by up to 3x, as Figure 4 shows:

We also took heap dumps from that project. Before rollout, segment readers dominated the retained heap. After rollout, those types dropped off the top of the dump and the indexing tier scaled down to smaller pods.

What hollow shards mean for Elasticsearch Serverless projects

Thin indexing shards moved Lucene files off the local disk and onto the object store. Hollow shards move Lucene objects off the heap until a write needs them. A hollow commit is a batched compound commit (BCC) whose empty translog node ID means recover metadata, not a writer. It also uses a live engine swap, which IndexShard didn’t previously allow on a started shard, so the first bulk waits instead of racing the swap.

In production, high-percentile fleet shutdown times dropped by up to 30%, and we got back some of the indexing tier heap. The cost shows up on the rare first write to an idle shard or when we run a merge.

If you’re running Elasticsearch Serverless, some of your shards may already be hollow. Idle backing indices should relocate faster during upgrades and scale events, in line with the shutdown time drop in Figure 3, and they should sit cheaper on the indexing tier while they wait for the next write (or never get one).

Acknowledgments

Hollow shards were built by the Elasticsearch Distributed team. In addition to the authors, we would like to thank Francisco Fernández Castaño, Tanguy Leroux, Benjamin Lerer, Albert Zaharovits, Artem Prigoda, and Henning Andersen, who designed, implemented, and productionized the work. We would also like to thank the wider Elasticsearch engineering team.

If tracing is on, the agent-builder-traces skill is already loaded on every agent in your space and there’s nothing to install. The first post in our series covered enabling OTel tracing and the out-of-the-box (OOTB) dashboards, along with threshold alerts. This post is about asking.

What is the agent-builder-traces skill?

agent-builder-traces is a built-in Agent Builder skill. It takes a natural-language question, generates an ES|QL query from it, executes that query against your trace index, and returns a plain-language summary. The index it targets is traces-agent_builder.otel-<space-id>, where <space-id> is the Kibana space that you’re working in. Using the exact space-scoped index pattern, rather than a wildcard, keeps data from other spaces out of your results.

Under the hood, the skill uses a single inline tool: agent-builder-traces.generate_esql. Instead of calling this tool directly, you ask a question and the agent calls the tool on your behalf, passing your question as the prompt for ES|QL generation. The tool resolves the current space's trace index, builds a query using the default model, executes it against Elasticsearch, and returns the result.

Which privacy settings control what your traces capture?

Several fields containing sensitive information are off by default. You can enable them in Gen AI Settings (Stack Management > AI Assistants), under the "Agent Builder Traces" section. Expand the Advanced privacy settings to find them.

Setting key

What it captures

agentBuilder:tracing:includeUserPrompts

User messages

agentBuilder:tracing:includeLlmResponses

Large language model (LLM) response text

agentBuilder:tracing:includeToolDetails

Tool call arguments and results

agentBuilder:tracing:includeSystemPrompt

The agent's system prompt

agentBuilder:tracing:includeRealNames

Real tool, agent, and conversation names (hashed when off)

agentBuilder:tracing:includeRealIds

Real conversation and workflow IDs (hashed when off)

agentBuilder:tracing:includeUserData

Real user IDs and usernames (hashed when off)

Why is my trace analysis query returning empty rows?

If the skill returns empty rows or tells you that a field is unavailable, check two things: whether the relevant setting is enabled in your configuration, and whether your query time window overlaps with any recorded spans. The skill reports exactly what the query returned; it doesn’t fabricate content when fields are empty..

What agent performance questions can you ask?

Examples of questions that you can ask include:

How many tokens have my agents used, by model?

Ask: How many input and output tokens have my agents used in the last 24 hours, broken down by model?

ES|QL: 

The TO_LONG() cast around each token field is required. These fields can surface as mixed integer and long types across index generations, and ES|QL’s SUM() needs an explicit numeric conversion before it can aggregate them. If you write your own queries against this index and hit unexpected type errors, this is usually why.

Which tools are failing most often?

Ask: What is the error rate for each tool over the last 7 days?

ES|QL: 

The tool error rate query aggregates across spans named ‘execute_tool’ and compares status.code == "Error" counts to total counts per tool name. The result shows which tools are failing most often. 

Which conversation turns are slowest?

Ask: Show me the 10 slowest conversation turns in the last hour.

ES|QL: 

The skill queries spans where span.name LIKE "invoke_agent *" and attributes.elastic.inference.span.kind == "CHAIN". These correspond to individual conversation turns, from the moment a user sends a message to when the agent returns a response. Durations are stored in nanoseconds, so the skill converts to seconds before sorting.

What have users been asking my agents?

Ask: What questions have users been asking in the last 30 minutes?

ES|QL: 

The recent-prompts query returns data only when agentBuilder:tracing:includeUserPrompts is set to true. When the setting is off, the attributes.gen_ai.input.messages field is empty and the skill will suggest that you check your Include User Prompts privacy setting.

When should you use Discover or Kibana Lens instead?

The skill targets one index pattern: traces-agent_builder.otel-<space-id>. Two scenarios fall outside that boundary.

Can the skill query my own application indices?

If you want to run ad hoc questions against data that you've indexed yourself, use a general data exploration skill or write ES|QL directly in Discover. The agent-builder-traces skill won’t query outside the Agent Builder traces index.

Can the skill build or edit dashboards?

The skill focuses on ad hoc queries and generating summary text. If your goal is to build a permanent visualization from your trace data, use Lens or the dashboard-management skill instead. We covered that specific process in depth in the first post in our series.

The skill sits alongside two other ways of watching your agents.

Tool

Best for

You use it when

agent-builder-traces skill

Ad hoc agent performance questions

You notice something and want an answer without leaving the chat

OOTB dashboards

Ongoing visibility

You want trends over time without asking anything

Threshold alerts

Automated monitoring

You define a condition once and get notified when it's breached

How to build evaluation pipelines from agent trace data

Spans in traces-agent_builder.otel-<space-id> capture real execution data from every conversation turn and LLM call running in your space. That record is the raw material for evaluation pipelines: automated checks on response quality and latency budgets per agent type, along with regression detection when you update a system prompt. The next post in our series covers how to build those eval loops from trace data that Agent Builder is already collecting.

Elasticsearch Workflows solves this with a persistent execution state. A workflow can pause at a human approval gate for days and resume exactly where it left off. Every decision is persisted in Elasticsearch as searchable data.

In this article, we’ll build this in practice with a real scenario: Documents that failed during ingestion get stuck in a data stream’s failure store. 

In this scenario, every step is reproducible. By the end, you’ll have an end-to-end workflow triggered by alerts, with pre-execution approval, automatic remediation, post-execution verification, and a rejection-and-revision path.

What you’ll learn

We’ll cover how to:

• Use the failure store to capture and remediate documents that failed ingestion.

• Build a long-running workflow with structured and binary approval gates using waitForInput and waitForApproval.

• Invoke Elastic Agent Builder AI agents automatically after human approval.

• Trigger workflows automatically from alerting rules.

• Handle approval and rejection paths (real human-in-the-loop, not just a checkbox).

What’s a long-running AI agent?

The difference between a session-bound agent and a long-running agent is the nature of the work rather than the speed of the large language model (LLM).

A session-bound agent lives within a session: It receives a question and processes it. Then it responds to the question. If the process dies, the context dies with it. 

Operational processes, like data remediation or infrastructure provisioning, or like change review, are different. The processing itself is fast. What takes time are human decisions at unpredictable moments. The person who needs to approve a reindex might be handling an incident or in a different time zone. Or they may simply be at lunch. Long-running agents handle these breaks in workflow by assembling separate infrastructure: a database to store execution state, a server for the orchestrator, a UI for approvals, and integrations for notifications. 

Elasticsearch Workflows runs where the data already lives. The workflow's long-term execution state is persisted in Elasticsearch, surviving restarts and waiting days between steps. AI agents come from Agent Builder and maintain a session only for the duration of each step, such as when the step ends or when the session ends. The workflow holds the context that spans across steps and days, and alerts come from the same rules already monitoring your data. The approval UI is in Kibana, and there’s no additional infrastructure to provision.

The workflow's internal execution state is managed by Kibana. The remediation-runs index is the audit trail that the workflow itself writes and that you can query like any other Elasticsearch index.

What’s the Elasticsearch failure store?

When Elasticsearch receives a document it cannot index, it has two options: Reject the document with an error, or store it somewhere safe for later analysis. The failure store is that second option.

Imagine a logs-demo-app data stream with the price field mapped as float. If the source application sends "price": "N/A" instead of a number, Elasticsearch cannot index the document. With the failure store enabled, it’s redirected to a dedicated index within the data stream itself rather than losing that document.

Documents in the failure store preserve the original content and include information about the error, including the exception type, the message, the pipeline and processor that failed, and even the stack trace. You can query the failure store using the data_stream::failures syntax. 

A typical document looks like this:

The document.source field contains the original document that failed, and the error field describes what went wrong. This is exactly the structure that the failure-analyst agent will read to diagnose the problem.

But the documents remain stuck there. Someone needs to diagnose the problem and fix the pipeline or mapping. They also need to reindex the documents back into the data stream. This is exactly the kind of work that combines AI automation with human approval, and it’s what we’re going to build.

How the AI agent orchestration works, from alert to resolution

The workflow connects four building blocks: an alerting rule, two AI agents, human approval gates, and an audit index. The diagram below shows how these components interact.

The alerting rule monitors logs-demo-app::failures every minute. When it finds documents written within the previous five minutes, it starts a workflow execution.

The workflow first runs read_failures to retrieve the failed documents. It then passes them to the failure-analyst agent through the diagnose step. The agent inspects the destination mapping and identifies the root cause. It then produces a structured remediation plan without making any changes.

The workflow records this plan in the remediation-runs index with the status awaiting_fix_approval and pauses at Gate 1.

Gate 1 uses waitForInput, a structured form that collects the reviewer's decision and optional notes. While the workflow waits, its state remains persisted in Elasticsearch. No agent session or polling loop needs to remain active, which means that the gate can wait for hours or days without continuously consuming compute resources.

If the reviewer approves the plan, the workflow invokes the remediation-executor agent through the execute_fix step. The agent calls the execute-failure-store-fix skill and creates the ingest pipeline. It also performs the bounded reindex and returns a detailed execution report.

Gate 2 appears only after the execution finishes. It uses waitForApproval to present the agent's report and asks the reviewer to choose between Yes, mark as resolved and No, escalate. The workflow then records the final outcome in remediation-runs and includes the complete report in agent_report.

If the reviewer rejects the initial plan at Gate 1, the workflow sends the diagnosis and the reviewer's feedback back to the failure-analyst agent, which then produces a revised plan that’s presented at Gate 1b. If approved, the revised plan follows the same automatic execution and verification path. If the plan is rejected again, the workflow records the case as fix_rejected and ends without applying any changes.

The remediation-runs index records the diagnosis, revision requests, revised plans, execution reports, and terminal outcomes as searchable audit documents. The shared workflow execution_id correlates records belonging to the same remediation run.

Production consideration: Exception-based verification

The diagram above shows the deliberately conservative implementation used in this tutorial: Every completed remediation pauses at Gate 2, including executions that appear fully successful. This makes the human-verification mechanism explicit and may be appropriate for low-volume or high-risk environments, but it can create review fatigue at scale. A production extension would add a deterministic verification step after execution. When the expected documents are present in the destination and the reindex reports no failures or policy-defined errors, the workflow can record resolved automatically. Only partial, failed, or ambiguous outcomes should pause at Gate 2 for site reliability engineering (SRE) review and possible escalation.

The following sections walk through each part: the data stream setup, the agents and skills, the alerting rule, and the workflow.

Prerequisites

To follow this tutorial, you’ll need:

• An Elastic Cloud Serverless project or an Elastic Stack 9.4+ deployment.

• Agent Builder (generally available [GA] on Elasticsearch projects, enabled by default).

• The following environment variables configured:

Setting up a data stream with the failure store enabled

All code in this tutorial, including the workflow YAML, setup scripts, and agent instructions, is available in this repository. 

Start by getting the code from the repository:

Then run the setup script:

The script prepares the test environment: It removes previous test resources, creates a failure-store-enabled index template, ingests three valid baseline documents, initializes the failure store, and creates the remediation-runs audit index. The price field is mapped as float with ignore_malformed: false ; nonnumeric values generate a parsing error and get redirected to the failure store instead of being silently dropped. The data stream itself is created automatically when the first document is indexed.

The failure store is enabled at the index template level, not on the data stream directly. The template includes a data_stream_options block that tells Elasticsearch to activate the failure store for any data stream created from that template:

The data stream is created automatically upon the first ingestion request, inheriting the failure store configuration from the index template. From that point on, any document that cannot be indexed is redirected to the failure store instead of being rejected with an indexing error.

The script also ingests three valid documents with numeric price values. These documents establish the healthy baseline: The data stream exists and contains valid data before any ingestion failures occur.

After the 01-setup-failure-store.sh script completes, the environment is ready:

The invalid documents that trigger the alerting rule and start the workflow are ingested later, when you run 02-trigger-test.sh in the testing section. This sequence shows a realistic failure scenario: The data stream is operating normally with valid data, and then an upstream change causes malformed documents to arrive.

Importing the agents and skills into Agent Builder

Run the restore script to import the two agents and two skills, along with the workflow into your Kibana instance:

Expected output:

Running the script again updates existing components without creating duplicates.

The AI agents that diagnose and execute the fix

After running the restore script, two agents will be available in Agent Builder: one to diagnose failures and another to apply approved fixes. In Kibana, go to Agent Builder and verify that both agents and their associated skills were created correctly.

failure-analyst, the read-only diagnosis agent

This agent reads failed documents from the failure store and inspects the destination mapping. It produces a structured remediation plan that includes an ingest pipeline definition and bounded replay instructions. The plan also includes a risk assessment. The agent doesn’t execute the proposed remediation.

The agent instructions establish its objective and read-only boundary. Key instructions include:

Its associated failure-store-remediation-planner skill provides the specialized rules used to construct a safe replay. For example:

The response contract is also constrained so that the workflow can consume the plan reliably:

Together, the agent instructions and skill allow failure-analyst to produce an actionable but bounded remediation plan while leaving execution under the control of the workflow and its human approval gate.

remediation-executor, the agent that applies the approved fix

The workflow invokes this agent after the remediation plan has been approved at Gate 1. Unlike the first agent, remediation-executor is allowed to perform the write operations required to apply the approved remediation.

Its agent instructions intentionally keep this role focused on immediate execution:

The associated execute-failure-store-fix skill defines how the approved plan is retrieved and executed. Key instructions include:

The skill searches by both execution_id and approval status. This ensures that the executor retrieves the plan approved for the current workflow execution rather than an unrelated remediation run. The status determines whether it uses the original diagnosis or the revised_diagnosis produced during the revision path.

This separation gives the two agents distinct responsibilities: failure-analyst investigates the failure and proposes a bounded plan without making changes, while remediation-executor applies the selected plan only after the workflow has completed the required human approval step.

The setup scripts import the complete agent instructions and skill definitions. The excerpts above highlight the instructions most relevant to understanding the responsibilities and behavior of each agent. The complete definitions are available in backup/save_agents.json and backup/save_skills.json in the repository.

After verifying that the agents and their skills have been imported correctly, the next step is to configure the alerting rule that detects new documents in the failure store and starts the remediation workflow.

How do I trigger a workflow from an Elasticsearch alert

The remediation process starts with an Elasticsearch alerting rule that monitors the failure store and starts the workflow when new failures are detected. Because the restore.py script has already imported and enabled failure_store_remediation, you can connect the workflow while creating the rule.

Creating the alerting rule that watches the failure store

The rule checks the failure store every minute and creates an alert when it finds documents written within the configured five-minute time window.

1. Go to Management → Rules → Create rule.

2. Select Elasticsearch query.

3. Configure:

• Query type: ES|QL.

• Query:

• Select time field: @timestamp.

• Select alert group: Create an alert if matches are found.

• Time window: 5 minutes.

• Check every: 1 minute.

4. In the Actions section, click Add action → Workflows.

5. Select the failure_store_remediation workflow.

6. Under Run workflow for, select New alerts.

7. Set Action frequency to Run per alert.

8. Save and enable the rule.

Whenever the rule creates a new alert, it starts one failure_store_remediation workflow execution for that alert. The failure store is still empty at this stage; the invalid documents are ingested later by 02-trigger-test.sh.

With the alert trigger configured, let’s examine the workflow that performs the diagnosis, approval, remediation, and verification steps.

Building the agentic workflow, step by step

At this point, the workflow has already been created in Kibana by the restore.py script executed earlier. To understand how the workflow operates, we’ll examine it incrementally, using relevant YAML excerpts to explain the role of each step in the remediation process. The complete workflow definition is available in the tutorial repository.

Declaring the alert trigger and the 7-day timeout

The workflow declares an alert trigger that allows it to be initiated by the alerting rule created earlier.

Two details are important here. First, the timeout: 7d defines that the workflow can remain active for up to seven days. We set this explicitly because the workflow will pause at human approval gates that may take hours or days to respond.

Second, declaring type: alert makes the workflow available to alerting. The workflow action configured in the rule establishes the connection between the alert and failure_store_remediation.

Reading failed documents from the failure store

The workflow begins by retrieving failed documents using the ::failures syntax:

The range query limits the search to failure-store documents created during the same five-minute window monitored by the alerting rule. Within that window, size: 50 bounds the batch to the 50 most recent matching documents. For larger volumes, use search_after or pagination.

How the AI agent diagnoses the root cause

The next step sends the failed documents to the failure-analyst agent:

The {{ steps.read_failures.output.hits.hits | json }} template injects the documents retrieved in the previous step into the agent's prompt. The agent analyzes the error patterns and identifies that the price field received string values where floating-point values were expected. It then proposes a remediation pipeline.

The agent's output arrives in steps.diagnose.output.message as a single JSON object. The exact diagnosis, proposed remediation, pipeline definition, and reindex request may vary between executions because they’re generated by the AI agent from the failures it receives. In our tests, the agent consistently identified the root cause and generated working remediation pipelines.

Despite this natural variability, the response contract remains fixed so that the workflow can persist the plan and the executor can reliably extract its executable components. The response always contains the six required top-level keys: root_cause, failure_type, affected_docs, proposed_fix, remediation_pipeline, and risk_assessment.

Recording the diagnosis for audit

Before requesting human approval, the workflow writes the diagnosis and proposed remediation plan to the remediation-runs index. This creates a durable audit record that can later be retrieved by the remediation-executor if the plan is approved:

The execution_id correlates the diagnosis with the current workflow execution. The record stores both the number of failure documents loaded into the bounded batch and the total number of failures matching the query. After human approval, the executor uses the execution_id together with the approval status to retrieve the correct remediation plan.

How do human approval gates work in Elasticsearch Workflows?

The workflow uses two kinds of pause steps for different kinds of human decisions. Gate 1 uses waitForInput to collect a structured approval decision and reviewer feedback. Gate 2 uses waitForApproval because verification requires a binary choice: Mark the case as resolved, or escalate it.

Gate 1

Gate 1b

Gate 2

Step name

gate_fix

gate_fix_revised

gate_verify (approved path) / gate_verify_v2 (revised path)

Step type

waitForInput

waitForInput

waitForApproval

Timeout

72h

72h

72h

Decision shape

Approve or reject, plus notes

Approve or reject, plus notes

Yes, mark as resolved / No, escalate

Presents

AI diagnosis and proposed plan

Revised plan

remediation-executor report

Appears

Always

Only after Gate 1 rejection

Only after execution completes

Gate 1 is defined like this:

When the workflow reaches this step, execution pauses, and its state is persisted in Elasticsearch. No agent session or polling loop needs to remain active while the gate waits. Every gate accepts a response for up to 72 hours, subject to the workflow's overall seven-day timeout.

The schema collects an approval decision and an optional notes field. When the proposal is rejected, the workflow passes the notes to the failure-analyst agent as context for revising the remediation plan. The testing section shows the approval and rejection payloads at the point where each is submitted.

What happens when the reviewer approves or rejects the plan

When Gate 1 is submitted, route_fix resumes the workflow without rerunning the completed steps. The simplified excerpt below shows the first transition in each branch. (The complete executable YAML is available in the tutorial repository.)

Approval sends the current workflow execution_id to the remediation-executor. The executor uses this identifier together with the approval status to retrieve the remediation plan associated with this specific execution before applying it and continuing to Gate 2.

If Gate 1 is rejected, the workflow stores the reviewer’s feedback and asks the failure-analyst agent to produce a revised proposal. Gate 1b (gate_fix_revised) then presents that proposal for another human decision. When the revised plan is approved, execute_fix_revised sends the same workflow execution ID to the executor:

The executor uses the approval status to determine which plan belongs to the current path: the original diagnosis associated with awaiting_fix_approval or the revised_diagnosis associated with awaiting_fix_approval_v2. Both paths then continue to Gate 2. If the revised proposal is rejected at Gate 1b, the workflow records fix_rejected and ends without applying any changes.

Verifying the result at Gate 2 before closing the case

Gate 2 is a verification checkpoint. By the time it appears, the remediation-executor agent has already created the ingest pipeline and run the reindex. It has also returned a detailed report. The gate embeds the executor's report, steps.execute_fix.output.message on the approved path, or steps.execute_fix_revised.output.message when the revised plan was executed, and asks the reviewer to select either Yes (mark as resolved) or No (escalate).

Gate 2 uses waitForApproval because the workflow only needs a binary verification decision. The operation details, including the pipeline ID, documents matched and reindexed, version conflicts, failures, and execution errors, come from the remediation-executor agent’s report rather than from fields entered manually by the reviewer.

In our validated execution, the agent created the logs-demo-app-price-remediation pipeline and matched and reindexed all five approved documents. It reported zero version conflicts and zero failures. The logs-demo-app data stream increased from three to eight documents, while the five original records remained in logs-demo-app::failures. This is expected because reindex copies documents; it doesn’t remove them from the failure store.

If the reviewer selects Yes, mark as resolved, route_verify writes a new document to remediation-runs with status resolved and stores the complete agent report in agent_report. If the reviewer selects No, escalate; the workflow writes an escalated record with the same report. The revised path mirrors this through route_verify_v2, producing the same statuses and the same agent_report field.

Gate 2 exists because a technically completed reindex can still produce partial or unexpected results. Escalating instead of automatically marking the remediation as resolved keeps the audit trail honest. Whether to keep this second gate depends on your environment and risk tolerance.

The complete workflow

After reviewing the workflow, remember that the complete code is available in the repository in the failure-store-remediation.yaml file.

The workflow can finish in five ways, depending on the decisions made at Gate 1, Gate 1b, and Gate 2. The diagram below shows how the original and revised remediation paths lead to resolved, escalated, or fix_rejected.

Every terminal outcome is appended to the remediation-runs audit history. Across the workflow paths, the audit records preserve the diagnosis, reviewer feedback when a revision is requested, the executor report, and the terminal outcome.

With the rule and its workflow action already configured, the end-to-end flow is ready to test.

Testing the workflow end to end

With the alerting rule enabled, run the trigger script to ingest five documents with invalid price values and current timestamps:

The rule searches the previous five minutes, so it detects the new failure-store documents during its next evaluation. Go to Workflows → Executions, and open the new execution. Confirm that read_failures, diagnose, and record_diagnosis have completed and that gate_fix is waiting for input.

Approving the fix at Gate 1

At Gate 1, approve the proposed remediation by submitting:

The workflow resumes at route_fix and invokes the remediation-executor agent through execute_fix. It pauses at Gate 2 with the execution report.

The following clip shows Gate 1 being approved and the workflow resuming automatically at route_fix and execute_fix.

Before completing Gate 2, verify that the five remediated documents were added to the data stream:

Expected result:

The data stream started with three valid documents, so a successful replay increases the total to eight. The original records remain in the failure store because reindex copies documents to the destination rather than removing the source records.

Complete Gate 2 by selecting Yes, mark as resolved or No, escalate. The workflow stores the selected outcome and the execution report in remediation-runs.

Rejecting the plan and reviewing a revised one

To test the rejection path, reset the environment and start another execution:

At Gate 1, reject the proposal with specific feedback:

Although notes is optional in the current schema, provide specific feedback whenever you reject a proposal. The workflow passes this value to the failure-analyst agent when requesting a revised plan.

The workflow records the feedback and asks the agent to revise the proposal. It pauses at Gate 1b. To approve the revised plan, submit:

To reject it definitively, submit:

Approving the revision sends it through the same execution and Gate 2 verification path described above. Rejecting it records fix_rejected and ends the workflow without applying the remediation.

Conclusion

The pattern built in this tutorial combines three elements: persistent workflow state, specialized AI agents, and human control at the points where judgment matters. The failure-analyst agent diagnoses the problem and proposes a bounded remediation. Gate 1 pauses before any change is made, giving the reviewer control over execution. After approval, the remediation-executor agent applies the fix automatically, and Gate 2 pauses again so the result can be verified before the case is marked as resolved or escalated.

Persistent execution state solves more than the problem of expiring agent sessions. The diagnosis, reviewer feedback, execution report, and final decision are stored as searchable data in Elasticsearch and correlated by the workflow execution_id. This correlation allows the executor to retrieve the plan approved for the current workflow run, even when multiple remediation cases exist in remediation-runs. It also makes it possible to analyze how many remediations were approved on the first attempt and which failure types are most frequently rejected, along with how long approval gates remain open and which cases require escalation.

This workflow pattern also applies to index promotions, mapping changes, enrichment pipeline validation, infrastructure operations, and other processes where automation and human judgment must coexist: Diagnose the problem, pause for approval, execute the approved action automatically, pause for verification, and record the final outcome.

Resources

• Elasticsearch Workflows documentation

• Agent Builder documentation

• Failure store documentation

• Tutorial repository

How historical metrics were loaded before backfill

Even if loading historical metrics isn’t a very common use case, it’s an important step when teams are adopting TSDB. Two scenarios have been the most prominent: bootstrapping a new time series data stream and migrating data from a different system or data stream to a time series one.

Bootstrapping a new time series data stream

You want to start a new time series data stream with a week of historical data so you have something meaningful to query from the start. With existing tooling, you had to set index.look_back_time to the seven-day maximum in the index template, and all historical data would land in a single backing index. For anything beyond seven days, you needed to create past backing indices manually.

Migrating metrics from another system

You have months worth of metrics stored on a different system and want to move your full dataset to TSDB. You need to load months of metrics history alongside live ingestion. The workaround was to manually create all the necessary past backing indices with the right time_series.start_time and time_series.end_time and to index into it directly using the index name. You then attached it to the data stream via the modify data stream API. It worked, but it required understanding the index time semantics and repeating the steps for each time window, along with coordinating that process around ongoing writes.

We wanted both scenarios to feel as close to normal bulk indexing as possible.

What time series data backfill changes

In 9.5, Elasticsearch can create backing indices covering past time ranges, which extends the eligible write window backward.

The eligible write windowis the range of @timestamp values that a time series data stream accepts for new documents. 

In the past, the eligible write window was determined only by the existing writable backing indices at the moment the request was received by Elasticsearch.

In 9.5, Elasticsearch can expand the eligible write window in the past by creating backing indices. This converts the eligible write window to a sliding window extending from the present back to the first read-only or destructive lifecycle action. Common examples of these actions, which are typically defined within your lifecycle configuration, are downsampling or searchable snapshots. Examples also include retention configurations. 

So, given that loading historical data is enabled in a cluster, the eligible write window of the data stream with the following lifecycle configuration is determined by the downsampling action, because it’s the first action that makes backing indices read-only. So, for this data stream Elasticsearch accepts documents whose @timestamp is no older than three months.

Why loading historical data into TSDB is hard

TSDB consists of data streams optimized for timestamped measurements. It uses a columnar storage layout and enforces immutable dimensions. It also organizes data into time-bound backing indices; each index covers a specific time range and accepts only documents whose @timestamp falls within it.

As time passes, rollover creates new backing indices to cover upcoming ranges. Until this release, there was no corresponding mechanism for the past. Creating indices in the past is tricky because historical data might span over a long period of time and can arrive at Elasticsearch out of order. Consequently, Elasticsearch cannot determine the write timeframe that its backing index should cover. Our solution to this is to use a preconfigured interval and lazily create past backing indices.

How Elasticsearch creates past backing indices

When a document is detected whose timestamp isn't covered by any existing backing index, Elasticsearch determines the time boundaries for the missing indices and creates them. It then adds them to the data stream in a single atomic operation. 

Lazily creating the indices ensures that a single request in the past won’t overwhelm the cluster by requiring the creation of 300 indices all at once. It also doesn’t create indices before there are docs to write into them.

Proactive vs. reactive: How we chose the index creation approach

We explored two ways to detect when a past backing index needs to be created.

The first is proactive. Inspect each incoming document's timestamp before routing, and create any missing past backing indices up front. This keeps the write path clean. By the time a document is routed, the index it needs already exists. It does require the data stream to already exist with at least one time series backing index, since that's what we inspect to determine the eligible write window and the time boundaries of the new index. The downside is that it adds work to every bulk request targeting a time series data stream, even requests that contain no past timestamps and need no backfill at all.

The second is reactive. Let the document fail the normal indexing, intercept that failure, create the missing index, and retry. This avoids any overhead on the common case, since the extra work only happens when a mismatch actually occurs. The tradeoff is more complexity in the failure handling path and a retry on every backfill document.

We ran performance tests on the proactive approach against bulk requests with no past timestamps and found no measurable regression. The overhead of inspecting timestamps turned out to be negligible. That settled it. Proactive creation is simpler and consistent with how index auto-creation already works in Elasticsearch. Plus, it adds no measurable cost to the workloads that don't use backfill.

How Elasticsearch determines past index boundaries

Each new past backing index has three properties to compute: its duration, its start time, and its end time.

Property

How it's set

Constraint

Duration

Defaults to one day, configurable via the cluster setting data_streams.past_tsdb_index_interval

Minimum one hour. If the triggering timestamp falls in a gap up to 1.3 times the configured duration, Elasticsearch collapses it into a single bridging index rather than creating many tiny ones.

Start time

Anchored to the start of the next existing backing index, working backward in multiples of the configured duration

Increased to match the end time of the previous neighboring index, where they would otherwise overlap.

End time

Start time plus the configured duration

Reduced to match the start time of the next index, where they would otherwise overlap.

Handling concurrent writes

In a distributed setup, multiple nodes can receive bulk requests with overlapping past timestamps at the same time. Each node collects the timestamps that aren’t matching any of the existing indices and sends a request to the master node. 

The master node executes a cluster update that sorts them and then, one by one, checks whether the timestamp is covered by an existing or newly created index. Otherwise, it issues a new create index request with the time boundaries calculated as described above. The cluster updates are always sequential and guaranteed to produce valid cluster states, so new indices are guaranteed to not overlap with existing indices.

How lifecycle age works for backfilled indices

Past backing indices hold old data but are new indices. Without an adjustment, lifecycle features would apply downsampling and retention based on when the index was created rather than when the data is from. We account for this by using the index.time_series.end_time as the index.lifecycle.origination_date. As a result, the age of the index as perceived by both data stream lifecycle and index lifecycle management (ILM) is based on the age of its data and not its creation time.

How to use time series data backfill

How to enable time series data backfill

Backfill support ships disabled by default. Enable it at the cluster level:

Bootstrapping with historical metrics

To load historical data into a new time series data stream:

1. Create your index template. 

2. Initialize your data stream. (This is an important step because an existing data stream is a requirement for creating past backing indices.)

3. Start indexing. 

Past backing indices are created automatically as documents with historical timestamps arrive, each covering one day of data by default. No additional configuration is needed.

Data migration into an existing data stream

Migrating data within the eligible write window

For data that falls within the eligible write window of your data stream, point your migration pipeline at the data stream and let Elasticsearch manage the rest.

Migrating data beyond a read-only action

For data older than the write window (for example, you're migrating 18 months of metrics but downsampling kicks in after seven days), you need a separate data stream without read-only lifecycle actions. Retention isn’t an issue since the data would be deleted anyway. The pattern is:

1. Create an index template for the historical data stream, using the same mappings as the original but without a lifecycle:

2. Create the historical data stream. If this step isn’t executed, the first indexing request might fail. During the first indexing request, Elasticsearch can create the data stream but it cannot yet create any past backing indices, so indexing a historical document might fail. Creating the data stream explicitly ensures that all indexing requests will be accepted:

3. Index historical data into the historical data stream while current data continues flowing into the original.

4. When the load is complete, add lifecycle. This is only supported by data stream lifecycle since this feature functions on a data stream level:

5. Query across both data streams with a wildcard pattern (my-metrics*) or a data stream alias.

6. If retention is configured, delete the historical data streams when their data expires. Data stream lifecycle will delete the data but it won't clean up the data stream itself.

As you see, the historical data needs to fit on the target tier as a whole because lifecycle will be enabled after the data is loaded. If you have a large historical import, you might choose to split it into batches. Make sure each batch can fit on the target tier as a whole at the time of indexing, to avoid running your cluster out of disk space. Lifecycle will start processing the batch's indices as soon as it's enabled, but it will need time to process the whole backlog.

Protecting the cluster during large migrations: Downsampling floodgate

When data stream lifecycle runs against a data stream with many indices that all qualify for downsampling, it queues them simultaneously. Downsampling is CPU and I/O intensive; it reads and rewrites all data in an index. Queuing dozens of operations at once can overwhelm the master node with persistent task updates while it coordinates them.

The downsampling floodgate scenario could occur before backfill support (for example, when adding a lifecycle policy to an existing data stream with months of accumulated data). Backfill makes it more likely by design.

In 9.5 and serverless, we added flood protection to data stream lifecycle. It now tracks how many indices per data stream are actively being downsampled. If that count reaches a threshold, data stream lifecycle pauses queuing further operations for that data stream until the count drops. The threshold is configurable via the cluster setting data_streams.lifecycle.downsampling.max_indices_in_progress. Other data streams aren't affected.

Limitations and prerequisites of time series data backfill

• Backfill doesn’t apply to read-only indices. If downsampling or a searchable snapshot transition has already run on a time period, documents for that period are still rejected.

• The feature requires a preexisting time series data stream with at least one time series backing index.

• System data streams are excluded.

• Replicated data streams rely on the leader data stream, so no direct backfilling is possible.

• Scaling remains your responsibility. Loading months of data can trigger significant storage usage, force merge operations, and lifecycle activity in parallel. Check that your cluster has the headroom to manage it before starting.

Conclusion

Prior to the Elasticsearch 9.5 release, loading historical data into TSDB was a manual process. By automating the generation and management of past backing indices, we aim to transform historical data migration to a native capability of your standard ingest pipelines. The inherent complexity of managing time-bound indices remains, but it has transitioned from a user responsibility into an internal Elasticsearch function. Whether you’re bootstrapping a fresh data stream or migrating extensive historical datasets, the platform now handles the heavy lifting, allowing you to focus on analyzing your metrics. We look forward to seeing how these improvements streamline your adoption of TSDB.

Why snapshots pin primary shards in stateful Elasticsearch

In traditional stateful Elasticsearch, snapshots lock primary shards to their active nodes, completely preventing relocation. That works fine when cluster topology is stable and nodes stay online between maintenance windows, but stateless Elasticsearch works differently. Index data lives in an external object store, with local disk as a cache, and the cluster scales automatically based on CPU, memory, and data size, both vertically (upsizing nodes) and horizontally (adding nodes).

During vertical scale-up, existing nodes must vacate all shards and shut down before new hardware takes over. Since Elasticsearch version 8.13, shard snapshots can pause during node shutdowns and resume after relocation, so a long-running snapshot doesn't block an infrastructure update.

Horizontal scale-out is a different story: No nodes shut down, so pause logic never triggers. New nodes sit idle while existing nodes finish their snapshots, and as soon as a snapshot is queued, the primary shard is pinned to its node, significantly delaying relocation.

Clusters typically scale out because they're already under heavy load. Blocking shard relocations at that moment limits the cluster's ability to reduce pressure, which shows up as degraded indexing throughput and higher latency. The resource imbalance can also trigger unexpected autoscaling behavior. And even when overall topology stays the same, shard locking disrupts hotspot mitigation and workload distribution. These failures used to surface as server-side warnings: "undesired allocation due to snapshot."

Stateful Elasticsearch

Stateless Elasticsearch

Snapshot reads from

Local shard data on the node holding the primary

The object store, using file locations recorded in the commit

Primary shard during snapshot

Pinned to its node until the snapshot completes

Free to relocate at any time

Effect on horizontal scale-out

New nodes wait for in-flight snapshots before taking shards

New nodes take shards immediately, regardless of snapshot state

How stateless snapshots read directly from the object store

A shard snapshot pins primary shards because it needs to read local shard data. In stateless Elasticsearch, that data already lives in the object store, so reading from local disk is unnecessary. Letting snapshots read directly from the object store removes the requirement to lock primary shards. They can relocate freely, and backup is decoupled from cluster balancing.

Stateless commits include location information for each data file in the object store, so snapshots can read and stream directly to the snapshot repository (a separate object store bucket). In the future, we plan to look at server-side ranged copies, which object stores support natively, to skip the local copy step entirely.

Tracking commits when shards relocate mid-snapshot

A snapshot is bound to a specific commit point that determines which files to back up, and those files must remain accessible for the full duration of the operation. In stateful clusters, this is simple: The snapshotting node and the data node are the same, so the commit is managed locally and held until completion.

In a stateless model, the snapshotting node and the data node can be entirely separate, or they can diverge if a shard relocates mid-snapshot. To handle this, we added a transport action that acquires commits on remote data nodes over the network. The data node tracks which commit belongs to which snapshot and releases it once cluster state signals completion.

There's a wrinkle during relocation. A stationary shard relies on its commit point to preserve files. A relocating shard must release its commit so its local store can close cleanly. To keep files accessible through that transition, a newly recovered primary temporarily preserves all existing data files in the object store until notified of snapshot completion via cluster state. This handles both graceful relocations and ungraceful recovery from node or engine failures.

No more allocation delays and improved cache stats

After stateless snapshots shipped, the server-side "undesired allocation due to snapshot" warnings stopped. The chart below shows the before and after, with the release marked by the red arrow. Hotspot mitigation became more responsive because shard relocations no longer had to wait for backup operations.

Cache use is also improved. Snapshots that bypass local shard data stop competing with indexing for cache space. After the release (also marked in the chart), we observed the following two positive changes in cache metrics:

1. The median cache population throughput, defined as bytes per second for filling the local disk cache from the object store, increased about 50%.

2. Cache misses, where data must be retrieved from the object store to fill local disk cache, have dropped more than 60%. 

What comes after stateless snapshots

Object-store-native architectures are increasingly the standard for cloud-native data systems, and stateless snapshots are a step toward fully exploiting that model across Elasticsearch operations. Backups read from the object store, and shards move freely. Neither process waits on the other. Removing the local shard dependency is a step toward further modularizing the stateless architecture.

How dashboard activity logs differ from the Kibana audit log

Dashboard activity logs and the Kibana audit log both write structured logs about user actions, but they answer different questions.

Kibana audit log

Dashboard activity logs

Answers

Who accessed what, and when

Which dashboards are used, and how well they perform

Built for

Security and compliance teams

Kibana admins and dashboard owners

Tracks

Security-relevant events across Kibana

Five dashboard actions: create, update, delete, view, refresh

Enabled by

Its own setting in kibana.yml

user_activity.enabled: true in kibana.yml

What dashboard activity logs capture

The user activity service records structured events every time a user interacts with a dashboard. Each event captures what happened and to which dashboard, in addition to who did it. Five actions are tracked:

Action

Fires when

Includes duration

dashboard_create

A dashboard is created

No

dashboard_update

An edit is saved

No

dashboard_delete

A dashboard is removed

No

dashboard_view

A user opens a dashboard

Yes, time on the dashboard until they navigate away

dashboard_refresh

A user changes filters or time range, or auto-refresh runs

Yes, refresh duration

Very often, a dashboard_view event also triggers a refresh.

Every event carries the same core fields, with two that appear conditionally:

Field

What it holds

Present on

user.name

Name of the user who performed the action

Every event

user.email

Email address of the user

Every event

user.roles

Roles assigned to the user

Every event

object.name

Dashboard name

Every event

object.id

Dashboard ID

Every event

kibana.space

Kibana space the dashboard belongs to

Every event

client.ip

IP address the request came from

Every event

event.action

Which of the five actions occurred

Every event

event.outcome

Whether the action succeeded or failed

Every event

event.duration

Time taken, in nanoseconds

dashboard_view and dashboard_refresh

error.type / error.message

Error class and message when something fails

Events where event.outcome is failure

How Kibana records dashboard activity

Under the hood, Kibana plugins report events from the browser or server through a core client, and valid events are written to a dedicated logger. No data is sent to a third party; because events are standard JSON logs, you control where they go and how they’re ingested.

What you can do with dashboard usage data

Dashboard activity data answers five operational questions that would otherwise require further investigation:

• Clean up unused dashboards. Filter for dashboards with zero dashboard_view events. If nobody's looking at it, archive it. This is critical for customers who are managing thousands of dashboards. 

• Troubleshoot performance. The event.duration field tells you exactly how long each dashboard load or refresh takes. Sort by duration to find your slowest dashboards.

• Edit history. Every create, update, and delete is logged with the user who made the change. You no longer have to wonder who modified a critical dashboard or when it happened.

• Plan capacity. Identify users running heavyweight queries during peak hours. If one user's auto-refresh is hammering the cluster every 10 seconds, you'll see it.

• Monitor errors proactively. Dashboards throwing errors surface immediately through error.type and error.message fields, so you don’t need to wait for users to report them.

How to enable dashboard activity logs in Kibana

Add two lines to your kibana.yml ( the service is disabled by default):

Events will start flowing immediately using a default JSON console appender. You can customize the output appender and filter specific actions using the same logging configuration schema that Kibana already uses:

Ship these logs into an Elasticsearch index (for example, via Filebeat), and you have a fully queryable dataset of dashboard usage.

How to query dashboard activity in Discover

Once your activity logs are indexed, open Discover and point it at your user activity index pattern. You'll immediately see every dashboard interaction as a structured event, and they’re filterable by action type, user, dashboard name, and time range.

From here, you can quickly answer specific questions like the examples below. 

How many times was a dashboard viewed? 

Type your question in natural language in the Discover query editor, and press Cmd+J to automatically generate the Elasticsearch Query Language (ES|QL) query, as shown below:

Which dashboards had zero views in the last 30 days?

Dashboards with no activity simply don't appear in the logs, so you can't filter directly for zero views. Instead, this query works backward, pulling every dashboard created (and not deleted) in the past year and then checking which of those had zero views in the last 30 days.

Which dashboards took longer than 10 seconds to load?

Note that event.duration is recorded in nanoseconds, so the query converts to seconds before filtering:

Which dashboards are throwing errors, and what's failing?

This query shows dashboards with one or more panels throwing errors during dashboard_refresh events, so you can quickly spot recurring issues and prioritize fixes:

Build a dashboard usage overview with AI chat

We’re planning to add out-of-the-box dashboards along with the activity logs, but in the meantime, instead of manually building visualizations, open the AI chat in Kibana and ask it to create a dashboard from your user activity data. 

The generated dashboard gives you at-a-glance visibility into your most-viewed dashboards, heaviest users, slowest-performing panels, and recent errors; that is, exactly the operational view that large deployments need.

Get started with dashboard activity logs

User activity logs are available in Kibana 9.5. Begin by enabling the service. Then ship the logs into an index, and start building the operational visibility that your team has been asking for. For full configuration details and the complete event schema, see the user activity documentation.

What you'll discover

In this post, you'll learn how to:

• Use Elastic APM's built-in views to explore search latency and throughput and to explore errors.

• Define Service Level Objective (SLOs) for search, including latency targets and availability, along with search quality.

• Create SLOs in Kibana that track your search health over time, with burn rate alerting.

• Build operational dashboards with Elasticsearch Query Language (ES|QL) that show latency percentiles and time breakdowns and that include trends.

• Set up alerts for latency regressions and error spikes, along with zero-results rate increases.

• Establish an incident response pattern for search degradation.

What you'll need

• Search instrumentation from Blogs 2–4 (search spans with search.* attributes in Elastic).

• Kibana access with permissions to create SLOs and alert rules.

• Basic understanding of SLOs. (We'll explain the search-specific parts.)

• An Elastic cluster with an Enterprise subscription, an Elastic Cloud trial, or a local deployment with the trial activated.

Why search monitoring matters beyond cluster health

Search is the primary navigation path for a significant share of visitors, and search-initiated sessions tend to show stronger purchase intent than browse sessions. A search outage is a revenue event, rather than a minor feature degradation. A latency regression from 100ms to 500ms changes user behavior before anyone files a ticket.

Most platform teams monitor search at the infrastructure level, checking whether the Elasticsearch cluster is healthy and whether nodes are responding. They also determine whether the disk is full. These are all necessary but not sufficient. A cluster can be green while search quality silently degrades; for example, queries returning stale data after a bad index deployment or latency creeping up as the index grows. This could also include zero-results rates climbing because a synonym list wasn't updated.

The gap is between "search is up" and "search is working well."

A note on examples: As we have throughout this series, we use ecommerce search for concrete examples, but these reliability patterns apply equally to any search application, including content platforms, internal knowledge bases, job boards, and support portals.

OpenTelemetry search spans as monitoring signals

If your team followed Blogs 2–4 in this series, every search request already emits an OTel span with search.* attributes. These include the query text, result count, Elasticsearch execution time, and error status. Those spans land in traces-generic.otel-default in Elastic.

Following along with code? The reference project has all the instrumentation from Blogs 2–4. Generate traffic, and then follow along with the SLO and dashboard setup below. See queries/blog6_reliability.esql for ready-to-run queries.

The search team built that instrumentation for product analytics; that is, understanding what users search for and measuring click-through rates (CTRs) and conversion rates. They’ve prioritized relevance work, but the same spans contain everything you need for operational monitoring. Span duration gives you a latency signal, and search.result_count == 0 value reflects quality. Span errors point to availability signals.

This post shows how to put this operational value to work, beginning with what you can see right now in Kibana and then building SLOs and alerting on top of it, along with incident response.

Search monitoring out of the box with Elastic APM

Before building anything new, let's look at what Elastic APM already gives you out of the box.

If your search API is instrumented with Elastic Distribution of OpenTelemetry (EDOT) (as in Blog 2), it appears automatically as a service in the Elastic APM UI. Open Observability > APM > Services in Kibana, and select your search service (named search-analytics-demo if you're using the reference project). You'll immediately see the following:

Elastic APM service overview for search

The service overview page shows latency distribution and throughput over time, along with error rate, and doesn’t require configuration. You can see at a glance whether search is healthy, and the time-series charts make regressions obvious. If latency crept up after yesterday's deployment, you'll see it here.

Trace waterfall: breaking down search request latency

Click into any transaction, and you'll see the trace waterfall, which is a visual breakdown of every span in the request. For a search API call, this typically shows:

The waterfall makes the invisible visible. You can see that the 503ms API response breaks down into HTTP handling, a 241ms query rules lookup, and a 260ms Elasticsearch query,  plus the custom search span (36ms) carrying all of our search.* attributes. Click any span, and the metadata flyout shows exactly what was captured: search.query: "usb hub", search.result_count: 33, search.took_ms: 43, the index name, hit IDs, and more.

Blog 2 discussed the gap between search.took_ms and span duration. The waterfall shows you exactly where that gap lives, without writing any queries.

Automatic search error capture with OpenTelemetry

One of the most valuable things OTel auto-instrumentation gives you is automatic error capture. When an Elasticsearch query fails because of issues like a tripped circuit breaker or a timeout, or if an index isn’t found,  the span records the exception type and message, along with the stack trace. Blog 4 mentioned this as a side benefit of span-based conversion tracking; here it becomes an operational lifeline.

The errors tab on your service page automatically aggregates these, grouped by error type and frequency. The instrumentation captures the details for you, so you don't need custom error handling or logging. During an incident, this is often the fastest way to understand what's actually failing.

Service map: search API and Elasticsearch dependencies

The service map shows dependencies between your search API and Elasticsearch, making it easy to see whether a latency problem is in your service or in the cluster it depends on.

All of this is available the moment you deploy the instrumentation from Blog 2, without building any dashboards or writing any queries. This is the foundation everything else in this post builds on.

Defining service level objectives for search

An SLO defines good enough in measurable terms. You define what working means, measure it continuously, and alert when you're burning through your error budget too fast, instead of reacting when something breaks.

Elastic Observability has a built-in SLO framework that handles Service Level Indicator (SLI) calculation and budget tracking. It also takes care of burn rate alerting. You create SLOs directly in Kibana. No ES|QL or custom pipelines are required for the core indicators.

Three SLOs every search service needs

Navigate to Observability > SLOs in Kibana, and click Create SLO. The SLO creation workflow walks you through three steps: Define the SLI (what to measure), set the objective (the target), and describe the SLO.

1. Search latency SLO: 99% of queries under 250ms

Indicator type: Elastic APM latency target: 99% of searches complete in under 250ms.

The Elastic APM latency indicator is purpose-built for this. Select your search service (search-analytics-demo), and set the threshold to 250ms.  Elastic handles the rest, including calculating the percentage of transactions below the threshold and tracking your error budget over time.

Note: The Elastic APM latency indicator measures all HTTP transactions for the search-analytics-demo service, including health checks and click/cart/checkout endpoints, along with static asset requests, not just the POST /api/search endpoint. For a search-only latency SLO, use a custom Kibana Query Language (KQL) indicator with name: "search" AND attributes.search.query: * on the traces-generic.otel-default index. The Elastic APM indicator is still valuable for whole-service health. For full coverage, combine both.

This measures end-to-end span duration; that is, what the user actually experiences. If your Elasticsearch query takes 50ms but the user waits 300ms because of network overhead or slow application logic, this SLO catches it. Use search.took_ms in the Elastic APM waterfall to diagnose where the latency lives when the SLO starts burning.

2. Search availability SLO: 99.9% success rate

Indicator type: Elastic APM availability target: 99.9% of searches succeed.

The Elastic APM availability indicator calculates the percentage of successful transactions for your service. When the Elasticsearch client throws an exception or the search endpoint returns a 5xx, the span's status records an error and this SLO counts it.

Note: Like the latency SLO, the Elastic APM availability indicator covers all HTTP transactions on search-analytics-demo, not just POST /api/search. Click/cart/checkout errors will consume this budget. For a search-only availability SLO, use a custom KQL indicator with name: "search" AND attributes.search.query: * for good events and name: "search" as the total query.

An 0.1% error budget on 100,000 daily searches means that you can tolerate 100 errors per day. That's tight, but search errors are hard failures and the user gets nothing. Availability SLOs should be stricter than latency SLOs.

3. Search quality SLO: tracking zero-results rate

Indicator type: Custom KQL target: 85% of searches return at least one result (zero-results rate < 15%); index: traces-generic.otel-default; good query: name: "search" AND attributes.search.result_count > 0; total query:name: "search" AND attributes.search.query: *.

Note on KQL versus ES|QL: The SLO framework uses KQL for its indicator filters rather than ES|QL. KQL uses field: valuesyntax and is the same language you see in the Kibana search bar. The ES|QL queries throughout this series are for ad hoc analysis and dashboards; KQL here is the SLO indicator's document filter. Both query the same traces-generic.otel-default index.

This is the SLO that surprises most teams. A search that returns an empty result set isn't an error; HTTP status is 200 and the span status is OK. Plus, no exception was thrown. But from the user's perspective, it failed. They asked for something and got nothing.

The quality SLO uses the custom KQL indicator type because it relies on our custom search.result_count attribute, which the built-in Elastic APM indicators don't know about. But the SLO framework handles everything else, including budget tracking and burn rate calculation, along with alerting.

A sudden spike in zero-results rate, such as from 12% to 40% over an hour, is almost always an infrastructure event, like a failed index deployment or a mapping change that broke queries. It could also be a synonym list misconfiguration. That's an operational problem, not a relevance problem.

Reading your search health in the SLO overview

Once the latency, availability and quality SLOs are created, the SLO overview page shows your search health at a glance:

Each SLO shows the current value, the target, the remaining error budget, and the burn rate. Green means healthy:  Search availability is at 100%, and Search quality is just above its 85% target. Red means violated: Search latency is at 50% against a 99% objective, with the burn rate breached at 200x the sustainable rate. When a budget bar starts shrinking faster than expected, you know something changed, even before users complain.

Clicking into the SLO detail shows burn rate across multiple time windows (1h, 6h, 24h, 72h) and the historical SLI trend. It also shows remaining error budget. For the latency SLO, the Elastic APM latency indicator tracks the percentage of transactions below your 250ms threshold. For the quality SLO, the custom KQL indicator uses traces-generic.otel-default with the good query filtering for attributes.search.result_count > 0. This is where the custom search.* attributes from Blog 2 pay off, since they're the foundation of meaningful SLOs.

Burn rate alerting for search SLOs

When you create an SLO through the Kibana UI, a default burn rate alert rule is automatically created. This is where the real operational value lives.

Burn rate alerts improve on threshold alerts ("error rate > 1%"), which are noisy and miss slow degradation. : Burn rate alerts measure how fast you're consuming your error budget relative to the SLO window.

A burn rate of 1.0 means that you're spending budget at exactly the sustainable rate, but a burn rate of 10.0 means that you're burning 10x too fast and you'll exhaust the budget in 1/10th of the window.

The default burn rate rule uses a multi-window approach, with four severity levels:

Severity

Burn rate

Long window

Short window

What it means

Critical (page)

> 14.4x

1 hour

5 minutes

Exhausts budget in ~50 hours

High (ticket)

> 6.0x

6 hours

30 minutes

Exhausts budget in ~5 days

Medium (review)

> 3.0x

24 hours

120 minutes

Exhausts budget in ~10 days

Low (awareness)

> 1.0x

72 hours

360 minutes

Trending toward exhaustion

The short window prevents alerting on brief spikes that self-resolve, and the long window catches sustained degradation. Together, they balance responsiveness with alert fatigue.

Routing search alerts to PagerDuty, Slack and Jira

Alerts are only useful if they reach the right people in the right tools. Elastic's alerting framework supports a wide range of connectors out of the box, including:

• Incident management: PagerDuty, Opsgenie, xMatters for on-call routing.

• Chat: Slack, Microsoft Teams for team notifications.

• Case management: Jira, ServiceNow for automatic ticket creation when SLOs breach.

• Custom: Webhooks for integrating with any system via HTTP.

You can also use Elastic's built-in cases to track incidents directly within Kibana, linking alerts and traces in one place, along with investigation notes, with push to Jira or ServiceNow when escalation is needed.

A typical routing setup:

Alert

Severity

Channel

Latency SLO burn rate > 14.4

Page

PagerDuty

Availability SLO burn rate > 14.4

Page

PagerDuty + Slack

Quality SLO burn rate > 6

Ticket

Jira (auto-create) + Slack

CTR anomaly (machine learning [ML] job)

Notification

Slack (search team)

Anomaly detection for search quality

Some search degradations are gradual shifts that slip past threshold-based alerts, rather than sudden spikes. A relevance regression after a model update might reduce CTR by 15% over a week, and latency might creep up by 5ms per day as the index grows. These are real problems, but they don't trigger burn rate alerts until it's too late.

Elastic's anomaly detection is built for exactly this. It learns normal patterns in your search metrics and flags deviations automatically, and you don’t have to configure any thresholds. 

Detecting search quality degradation with ML anomaly detection

• Latency anomalies:

 Elastic APM anomaly detection can be enabled directly from the Elastic APM UI for your search service. It learns the typical latency distribution, including daily and weekly patterns, and alerts when behavior deviates. A gradual 5ms/day creep will eventually register as anomalous before it hits your SLO threshold.

• CTR drops:

A relevance regression is invisible to traditional monitoring; latency is fine and errors are zero, plus the result counts are normal, but the ranking changed and users aren't clicking. Anomaly detection on click volume per query is a practical proxy: When a query that normally receives 20 first-click events per hour drops to 5, something likely changed.

To set this up: In Kibana → Machine Learning → Anomaly Detection, create a new job. Use the Multi-metric wizard, and select traces-generic.otel-default as the index. Configure a count detector on attributes.search.first_click split by attributes.search.query. This creates a per-query click-volume baseline and alerts when individual query engagement drops outside the expected range.

Note: This job detects click-volume anomalies per query, not CTR (which requires dividing clicks by searches). Click volume is a useful proxy (a CTR regression usually manifests as a drop in absolute click count), but be aware that a traffic surge with flat click volume would show as a CTR drop without triggering this alert. For true CTR anomaly detection, use a scheduled ES|QL transform to materialize hourly CTR values and run anomaly detection on the computed ratio.

Route the resulting ML alert rule to your Slack search channel.

• Throughput shifts:

A sudden drop or unexpected surge in search volume can indicate upstream problems (like load balancer changes or traffic shifts) or downstream issues (such as search becoming unresponsive or users retrying).

Configure ML anomaly alert rules to route these to your notification channels. These complement your SLO burn rate alerts; burn rates catch budget consumption, and anomaly detection catches pattern changes.

Building a search monitoring dashboard with ES|QL

SLOs tell you whether search is healthy. When they indicate a problem, you need a dashboard that tells you why.

The search team and the on-call team need different views of the same data. A search engineer wants query-level detail, such as which queries have low CTR and which ones return nothing. They’re also interested in where to invest in relevance. But an on-call SRE wants the operational picture, including whether search is fast and whether it’s up. It also wants to know whether search is degrading, and if so, since when.

Search monitoring panels for the on-call dashboard

Build it in Kibana dashboards using Kibana Lens panels. Lens supports ES|QL as a data source, so the queries from Blogs 2–4 can power dashboard panels directly. The key panels include:

• Search throughput over time:  A sudden drop is often the first sign of a problem.

• Latency percentiles (p50, p95, p99) over time: When they diverge (p50 flat, p99 spikes), you have a subset of slow queries.

• Error rate over time: Spikes here mean hard failures.

• Zero-results rate over time: A step change upward, especially correlated with a deployment, means something changed in the index or query pipeline.

The ES|QL for each panel follows the patterns from earlier blogs. For example, a latency percentile panel:

This includes three lines on one chart. When they diverge, such as when p50 stays flat but p99 spikes, you likely have a subset of queries that are slow while the majority are fine. That's a different diagnosis than all queries slowing down (cluster-level pressure).

Drill-down panels: slowest queries and top zero-result queries

For investigation, add a few detail panels, such as:

Slowest queries: A table showing the queries with the highest p95 latency and their search volume. During an incident, this narrows the problem from "search is slow" to "these specific queries are slow."

• Top zero-result queries: A table showing which queries most frequently return nothing. When zero-results rate spikes, this panel immediately shows which queries are responsible.

These drill-down panels use the same ES|QL patterns as Blogs 2 and 3, just surfaced on a persistent dashboard instead of run ad hoc.

Search incident response using OpenTelemetry traces

As an example, an alert fires, noting that search latency has spiked. What happens now?

The trace data from Blog 2's instrumentation gives you a structured path from symptom to root cause.

Step 1: Assess scope

Start at the on-call dashboard, and get answers to the basics:

• When did it start? Narrow the time range to the degradation window.

• How bad is it? Is p50 affected (all queries slow) or just p99 (a subset)?

• Is it just search? Check the Elastic APM service map to determine whether the Elasticsearch dependency is also degraded.

Step 2: Find the problem queries

If the problem is a subset of queries (p99 spike but p50 is fine), use the slowest queries panel or run:

Adjust the 100 ms threshold to match your environment's normal range. It can be lower for a fast cluster or higher if your data volume makes 100ms typical. This narrows the problem from "search is slow" to "these specific queries are slow." That's the difference between restarting the cluster and investigating a specific query pattern.

Step 3: Drill into the trace waterfall

Pick a slow query, and open it in the Elastic APM trace view. The waterfall shows exactly where time was spent. (Refer back to the trace waterfall GIF above to see a real example of a POST /api/search trace broken down into its component spans.)

The overhead gap between search.took_ms (Elasticsearch time) and span duration (end-to-end time) is your diagnostic tool:

Scenario

search.took_ms

Span duration

Diagnosis

Elasticsearch slow

400ms

430ms

Elasticsearch problem: Check slow log, cluster metrics.

App slow

50ms

350ms

Application / network overhead: Check serialization, network.

Both slow

400ms

700ms

Multiple issues: Investigate both.

If the problem is in Elasticsearch, drill into the Search Profile API or cluster monitoring. If it's application overhead, look at the spans around the search span in the waterfall.

Step 4: Correlate with events

Check whether the degradation correlates with:

• Deployments: Did someone deploy a new version of the search service or push a new index?

• Cluster events: Is Elasticsearch under memory pressure, or are there long garbage collection pauses? Or maybe the disk is I/O saturated?

• Network: Is latency between the search service and Elasticsearch elevated?

Elastic Observability's unified platform makes this correlation straightforward because traces, logs, metrics, and infrastructure data all live in the same Kibana instance. You're adding filters in the same interface, rather than switching between tools.

Going further: Infrastructure metrics and cost attribution

This post focuses on what you get from trace data; that is, the spans your search API already emits. But OTel and Elastic Observability support a wider instrumentation picture that becomes valuable as your search infrastructure matures.

• Infrastructure metrics. Adding host and container metrics (like CPU, memory, disk I/O, and network) alongside your traces lets you correlate search performance with infrastructure use. When p99 latency spikes, you can immediately see whether the Elasticsearch nodes are under memory pressure and whether garbage collection  pauses are increasing. You can also check whether disk I/O is saturated, and you can do all this in the same Kibana interface. The Elastic Agent collects these automatically for your infrastructure, and the infrastructure monitoring UI surfaces them alongside your Elastic APM data.

• Total cost attribution (TCA). With infrastructure metrics flowing alongside traces, you can start attributing infrastructure costs to specific services and operations. How much compute does your search service consume? How does that correlate with query volume? If a new ranking model doubles CPU usage per query, you can see the cost impact directly. This is particularly valuable for teams running search on cloud infrastructure where costs scale with resource consumption; understanding the cost per search helps justify infrastructure investment and identify optimization opportunities.

• Logs correlation. OTel auto-instrumentation injects trace context (such as trace ID and span ID) into your application logs. This means that when you're investigating a slow search in the trace waterfall, you can click through to the exact log lines from that request, including Elasticsearch slow log entries and application debug output. It also includes error details that don't fit in span attributes. The logs correlation feature automatically ties them together.

These are natural next steps once you have traces working. Each one extends the same unified platform, without new tools or separate pipelines.

How search analytics and search monitoring share one data pipeline

Here's how it all fits together:

The data flows from the instrumentation you built in Blog 2. This one investment supports two audiences: The search team gets product analytics (Blogs 2–5), and the SRE team gets operational monitoring (this post). Neither team needs separate data pipelines.

Getting started with search monitoring in Elastic

This is the last post in the series, and it brings us full circle. Blog 1 describes the vision: Instrument search once with OTel, and send spans to Elastic. Then use ES|QL to answer any question about search behavior. Blogs 2–4 build the instrumentation and analytics, and Blog 5 shows how to feed that data back into relevance improvements. This post shows how the same data powers operational monitoring, including SLOs, alerting, anomaly detection, and incident response.

The key takeaway for search engineers is that the instrumentation you built for analytics already generates the signals. The SLOs and alerts are built-in capabilities of Elastic Observability, as are the dashboards. You're closer to production-grade search monitoring than you might think. Observability isn't a separate discipline you need to learn from scratch. 

If you've been following along and built the instrumentation from Blogs 2–4, start here:

1. Open Elastic APM: Look at your search service, and explore a trace waterfall. You can also check the errors tab.

2. Create three SLOs: Latency (Elastic APM latency), availability (Elastic APM availability), and quality (custom KQL for zero-results).

3. Enable anomaly detection: One click in the Elastic APM UI for latency anomalies.

4. Build the on-call dashboard: Four Lens panels with the queries from this post.

By the end of an afternoon of work, your search service can have the same observability coverage as any other critical production system.

Get started

Working code

• Reference project: Working code for the entire blog series; clone, configure, and run.

Elastic APM and traces

• Elastic APM Overview: Elastic APM concepts and trace analysis.

• Elastic APM UI: Service overview, transactions, dependencies, errors.

• Service Maps: Dependency visualization and health.

• OpenTelemetry Integration: OTel ingestion in Elastic.

SLOs and alerting

• SLOs in Elastic Observability: Creating and managing SLOs.

• Create an SLO: SLI types (like Elastic APM latency, Elastic APM availability, or custom KQL), time windows, budgeting.

• SLO Burn Rate Alerts: Multi-window burn rate alerting.

• Alert Connectors: Slack, PagerDuty, webhook, email integrations.

• Alerting Framework: Rule types and configuration.

• Jira Connector: Automatic ticket creation from alerts.

• ServiceNow Connector: ITSM integration.

• Elastic Cases: Built-in incident tracking with external push.

Anomaly detection

• Anomaly Detection Overview: Unsupervised time series anomaly detection.

• Elastic APM Anomaly Detection: Enable ML for latency, throughput, error rate.

• ML Anomaly Alert Rules: Alerting on detected anomalies.

Dashboards and ES|QL

• Kibana dashboards: Building operational dashboards.

• Lens: Visualization editor.

• ES|QL in Lens: ES|QL-powered dashboard panels.

• ES|QL Overview: Language reference.

Elasticsearch operations

• Slow Log Configuration: Threshold-based query slow logging.

• Search Profiling: Profile API for query execution analysis.

• Monitoring Elasticsearch: Cluster stats, search rate, latency.

From this series

• Modern search analytics with OpenTelemetry: The vision.

• Instrument your search API : Search spans and search.* attributes.

• Measuring search quality : CTR, MRR, click distribution.

• From clicks to conversions: Conversion tracking and revenue attribution.

• Personalizing search from behavior: Judgment lists, rank features, Learning To Rank (LTR).

This is the final post in a six-part series on search analytics with OpenTelemetry and Elastic. Start from the beginning: Modern search analytics with OpenTelemetry, or to start building, jump to Instrument your search API.

Observability workloads are storage-intensive by nature, and the composition of that storage determines both cost and query performance. ES95 introduces adaptive compression: rather than applying the same encoding to every numeric field, it automatically selects the encoding that best matches each field's structure. The result is a 33.6% reduction in total doc-values storage, 19% to 74% reduction on floating-point gauge metrics and a 92% reduction in @timestamp. No configuration or migration required.

Observability data is storage-intensive

Data processing systems are rarely limited by how fast they can compute. They’re limited by how fast they can move bytes: off disk, across the network, and through the memory hierarchy. Compression is how a storage engine trades CPU time for memory bandwidth, spending comparatively cheap CPU cycles so fewer bytes have to travel through the parts of the system that are usually constrained. In a read-heavy system like Elasticsearch, that trade-off pays back every time data is queried, often long after it was written.

Storage size and query performance move together; fewer bytes on disk means fewer bytes to read on every range query, every aggregation and every dashboard load. Compression is not just about saving storage. Every byte that is never written is also a byte that never has to be read.

The right encoding depends on the structure of the values themselves, and the largest wins come from exploiting the structure already present in the data rather than squeezing an opaque stream of bytes. Few workloads expose that structure more clearly than observability metrics.

A single host reports hundreds of metrics every few seconds, including CPU utilization, memory ratios, request latencies, and network throughput. Multiply that by thousands of hosts across weeks of retention, and the bytes accumulate fast. Most of that volume is structured but not uniform: timestamps arrive at near-constant intervals from thousands of concurrent series, counters increase monotonically, while gauges like 23.47 or1.15 are short decimal measurements.

A fixed compression approach cannot adapt to that variety. A timestamp column and a floating-point gauge column compress through fundamentally different techniques, but a codec that applies the same approach to both will necessarily handle one of them poorly. For most of Elasticsearch's time-series codec history, gauges were on the losing end of that trade-off.

The structure the old codec was not built to exploit

Elasticsearch stores time-series numeric values in doc values: a column-oriented structure where all values for the same field sit adjacent on disk. That adjacency makes compression possible: the codec compares consecutive values of the same field, finds patterns, and exploits them.

The time-series codec before ES95 applied the same fixed encoding to every numeric field: delta encoding followed by normalization, GCD (greatest common divisor) reduction, and bit-packing. Each encoding technique activated where it helped and skipped where it would not. For timestamps and integer counters, this approach was remarkably effective. For floating-point gauges, it could find almost nothing to work with.

The reason is how they’re stored. To support range queries, Elasticsearch stores floating-point values as integers that preserve numeric ordering. A change of 0.01 in a CPU percentage reading translates to a jump of trillions in that integer space. The codec sees those large jumps and has no strategy to further reduce their footprint. Storage stays near the original eight bytes per value.

The codec was doing the right thing with the representation it had, but the latter was chosen for querying, not compression, and the goals conflict at the bit level.

The cost of a fixed format

A compression stage for floating-point values was already on the roadmap, so the interesting part wasn’t the algorithm. The obstacle was architectural.

The previous codec baked its compression approach into the storage format. Adding a new encoding meant changing the meaning of existing bytes on disk, which forced a format migration, a rollout that can last weeks or months in large production clusters. Over time, that migration burden constrains codec development itself. The question stops being Is this a good compression idea? and becomes Is it worth another format migration? That rigidity limits the cadence of codec evolution and leads to missed compression improvements.

The right encoding without configuration

ES95 solves this at the architecture level for time-series indices. Each field's encoding is no longer baked into the format. It is selected automatically at write time, based on what the field mapping already declares: the field's name, its data type, and its metric role. Timestamps are encoded differently than counters. Counters are encoded differently than gauges. ES95 encodes all of them, and it chooses the right strategy for each.

Users already tell Elasticsearch everything the codec needs to know. The mapping describes the data; the codec chooses the compression strategy.

Compression strategy is a codec concern, not a user concern.

The alternative would have been to expose per-field encoding selection as a configuration parameter, letting users opt in to better compression for specific fields. That would shift the burden of knowing which encoding fits which data type onto those least equipped to make that call and would guarantee that most deployments never see the benefit. ES95 keeps that decision inside the codec, where it belongs. This matters most in managed and serverless deployments, where users expect the system to automatically make optimal storage decisions.

The timestamp result nobody planned for

With the adaptive architecture in place, the team set out to ship the planned float-compression algorithm. Before it arrived, the architecture proved itself by substantially improving compression for timestamps.

A time-series index is sorted first by its time-series identifier (_tsid constructed by the metric’s dimensions) and then by timestamp within each series. Timestamps on disk aren’t one smooth sequence; there are many smooth sequences laid end to end, one per series, with a large jump at every boundary where one series ends and the next begins.

The codec compresses data in fixed-size blocks without regard to those series boundaries. A block straddling a series boundary holds timestamps from two different series. The jump between them breaks monotonicity, reducing delta encoding effectiveness on blocks spanning different time series. A block that would otherwise compress to near-zero bits per value ended up needing nine or more, because bit-packing encodes every value in a block using the same fixed number of bits, so one large jump sets the cost for all of them.

In the ideal case, every block belongs to a single series: timestamps increase at near-constant intervals, delta encoding captures the regularity, and bit-packing compresses the result to near-zero bits per value. With few series, boundary blocks are rare and the overhead barely registers. On an observability cluster ingesting millions of documents across thousands of series, that changes. The cost scales along two dimensions: series count and data density. More series means more boundary events; sparser series means multiple jumps packed into single blocks. In high-churn environments, both compound, and boundary blocks accumulate into a standing tax on the most-read field in any time-series workload. It’s why @timestamp storage grew faster than the data that produced it.

The fix was to detect series boundaries and treat each run as its own independent sequence. Instead of trying to encode across the jump between two series, which forces every value in the block to pay the storage cost of that one large jump, each run is compressed on its own terms. The boundary simply becomes a seam: the jump is never seen by the encoder on either side.

That encoding is called SplitDelta. @timestamp and monotonic long counters now use it by default. No format change. No migration. Existing segments retain legacy encoding.

On the high-cardinality Elasticsearch Rally benchmark, that single unplanned encoding cut counters storage by 20%–30% and @timestamp storage by 92.4%, from 1.03 GB to 79 MB. Gigabytes to megabytes, and no, that isn’t a typo.

The pluggable pipeline had already paid for itself. SplitDelta, which wasn’t part of the original plan, slotted in without a format change or migration before ALP even shipped.

ALP: recovering the decimal that was always there

Most floating-point metrics can be expressed as short decimals with no loss of accuracy: CPU utilization at 23.47, load average at 1.15. ALP (Adaptive Lossless floating-Point compression) recovers that decimal structure from the floating-point representation, converting values into integers that the existing pipeline already handles well. ES95 feeds ALP's output into the same mature integer compression pipeline used for timestamps and counters, extracting additional savings rather than treating ALP as a standalone encoding. Values that don’t fit ALP's model (such as irregular high-precision floats or special values) fall back to direct bit-packing or the original representation without degrading the rest of the block.

ALP lets Elasticsearch treat floating-point metrics according to the structure they actually contain rather than the binary representation they happen to use. It’s applied automatically to double-valued gauge fields, selected by field type and metric role, through exactly the door the architecture had built for it.

What the numbers say

Here’s what the Elasticsearch Rally benchmark looks like on a high-cardinality workload containing 2.26 billion data points. Results are from an internal tsdb-metricsgen benchmark.

Field or metric

Storage reduction (%)

@timestamp

−92.4%

cpu.load_average.5m

−74.3%

system.cpu.utilization

−63%

memory.utilization

-19%

Total doc values

−33.6%

That overall 33.6% reduction deserves context.

A time-series index contains more than metrics. Every data point also carries the labels that identify the series: host names, IP addresses, regions, container IDs. Those dimension fields are stored as keywords. ES95 doesn’t target dimensions.

On this benchmark, two dimension fields, host.ip and host.mac, accounted for 44% of doc-values storage after ES95 ran. The 33.6% total reflects that mix. The per-field breakdown is the honest picture. Compression for dimension fields is an active area of work.

The per-field variation is the most convincing result. Some gauges shrank by nearly three quarters, while others moved by less than a fifth. That spread is direct evidence that ES95 matches compression to the structure actually present in each field. A fixed encoding treats every field identically and misses most of those wins.

Better compression without extra configuration

The storage reductions from SplitDelta and ALP are the most visible results of ES95. The more consequential result is the architecture that produced them.

Before ES95, every new compression technique required a format evolution. That reality shaped which ideas were practical to pursue. Today, new encodings become implementation decisions inside the codec rather than migration projects. Existing data never needs to move, and users gain better compression on newly written data simply by upgrading Elasticsearch. SplitDelta and ALP are the first encodings to benefit from this architecture. They will not be the last.

Asking users to choose compression algorithms would only duplicate information Elasticsearch already has. There are no per-field compression parameters to tune, and no expert knowledge is required to get good storage efficiency. Different fields get different strategies because ES95 understands what kind of data each field contains, not because a user configured it. As the codec evolves, those decisions evolve with it. The API does not.

In Elasticsearch Serverless, good defaults are part of the product. Users expect the system, not configuration, to make storage decisions. ES95 is designed to honor that expectation: encoding that starts right and gets better over time.

The compression was always there

ES95 establishes a new standard for how time-series codec evolution works. New encodings become implementation decisions, not migration projects. Users get better compression on newly written data with every Elasticsearch upgrade.

The compression was already in the data. ES95 just removed what was hiding it.

How to configure frozen_after in data stream lifecycle

frozen_after sits at the top level of the lifecycle, next to data_retention and downsampling.

That's the whole feature, at the API level. Indices in my-data-stream stay on hot for 30 days, then move to frozen for the remaining 60. After 90 days they're deleted, and the backing snapshot goes with them.

It composes with the rest of the lifecycle, including downsampling:

Same options in an index template:

The order of values is enforced: frozen_after has to be less than data_retention and greater than any downsampling.after. The API rejects configurations that don't make physical sense.

Where frozen tier data is stored: the default snapshot repository

Frozen tier data is held as partially-mounted searchable snapshots, which means DLM needs a snapshot repository to write into. Rather than make you choose a repository per lifecycle, 9.5 introduces a cluster-level default snapshot repository.

DLM uses this repository for every frozen tier index in the cluster. On Elastic Cloud Hosted (ECH), the default is pre-populated with found-snapshots so existing clusters work out of the box. You can change it to a repository you control if you'd rather keep your frozen data in a bucket you own (useful if you want object versioning, lifecycle backups to Glacier, or anything else that needs bucket-level access). Wherever you can set frozen_after in Kibana, the UI shows the current default repository inline and links to the place to change it, so you can see where frozen data will be written.

If the cluster doesn't have a default repository configured, you can still write a lifecycle with frozen_after. The API accepts it but returns a warning:

The data stays on hot until a default repository is configured and exists. The same logic applies if the cluster lacks a valid Enterprise license. Errors are visible in the data stream lifecycle status API for that stream.

Configuring frozen_after in Kibana

Kibana in 9.5 lets you set frozen_after and the default snapshot repository from the UI. In Streams, the Retention tab shows the frozen phase on the lifecycle timeline alongside hot and any downsampling steps. Click the timeline to open the data lifecycle flyout, set frozen_after, and see the timeline update before you save. Index Management's Data Streams page opens the same flyout.

How frozen tier conversion works in data stream lifecycle

When a backing index ages past frozen_after, DLM walks through five steps in order:

1. Clone. Mark the index read-only and clone it to a zero-replica copy so the original stays available while conversion runs.

2. Force merge. Merge the clone to a single segment. On completion a cluster state marker is written; duplicate force-merge requests (for instance after a master failover) are deduplicated, so a restart doesn't repeat the work.

3. Snapshot. Write the merged clone to the default repository, and record the snapshot name in cluster state on success. If a stalled snapshot from a previous attempt is detected, DLM deletes it and re-runs the step.

4. Mount. Create a partially-mounted searchable snapshot index from the snapshot.

5. Swap and delete.Once the mounted index's shards are fully allocated, atomically swap it in for the original in the data stream, then delete the original. The swap is atomic, so query results don't see a data volume dip during the transition.

On failure, DLM retries from the last successful step on the next run.

Each step is idempotent, and the cluster state markers make sure work already done isn't repeated after a master failover. Throttling caps concurrent conversions so a lifecycle change covering thousands of indices doesn't overwhelm the cluster. Errors surface in the data stream lifecycle status API and roll up to the lifecycle health indicator.

Scope and limitations of frozen_after

• Data indices only. frozen_after applies to a data stream's data indices. Failure store indices don't currently support the frozen tier and continue to be governed by their own data_retention setting.

• Enterprise license required. Frozen tier in DLM is implemented with searchable snapshots and requires an Enterprise license. You can write frozen_after on any license, but data won't move to frozen until the license is valid.

• Serverless ignores the field. In Elastic Cloud Serverless, frozen_after is accepted but ignored - Serverless manages tiering on your behalf. Built-in templates may include the field, so we don't reject it, but the step is skipped.

Getting started with frozen_after

1. In Kibana, open Streams or Index Management and choose a data stream backed by data stream lifecycle.

2. Open the Edit data lifecycle flyout, set a frozen-after value, and save. The lifecycle timeline shows the new phase.

3. On a self-managed cluster, set repositories.default_repository to a repository you control. On ECH, found-snapshots is already configured if you want zero setup.

4. For declarative workflows, write the same configuration into your index templates so new data streams pick up the lifecycle automatically.

Learn more

• Data stream lifecycle

• Frozen tier overview

• Searchable snapshots

• Manage data retention for Streams

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Why dense vectors create off-heap memory pressure that heap charts miss

Semantic search and kNN rely on dense_vector fields. Elasticsearch stores much of that data in off-heap memory. It’s related to how the Java Virtual Machine (JVM) operates, but it isn’t the same thing as heap usage.

In production, the heap versus off-heap split shows up in a familiar pattern:

• Heap looks fine for weeks, while the dense vector off-heap footprint quietly grows.

• Heap circuit breakers stay quiet or only spike late because the pressure sits outside the JVM.

• kNN search and bulk ingest slow down, queues build, and nothing on the dashboard points at vector RAM as the cause.

Heap limits protect Java allocations. They don’t tell you whether vector off-heap footprint still fits the RAM envelope that your deployment actually runs in. AutoOps already watches cluster health broadly; Vector memory pressure adds a focused read for vector-heavy nodes when memory and load signals line up.

How AutoOps measures vector RAM, heap, and headroom

AutoOps works from the same node stats metrics you already use for Stack Monitoring. For each node, it tracks three derived numbers:

Symbol

Meaning

Source (typical)

Chart (see below)

V

Vector off-heap footprint

indices.dense_vector.off_heap.total_size_bytes

First, green line

A

Available RAM in the product view

Delta between os.mem.total_in_bytes and os.mem.used_in_bytes

Second, green line

H

Headroom

A − V (headroom_bytes)

First, blue line

H > 0 means there’s a modeled runway: Vector use still fits comfortably in that accounting. H ≤ 0 means that you’re in a compression regime: Vector footprint (V) meets or exceeds the free RAM (A) picture that AutoOps can align in telemetry. On small tiers, that can be common under load. The insight emphasizes trends, growth in vector off-heap footprint, and corroborating stress, not a single negative snapshot.

AutoOps also tracks a compression regime flag (fraction of recent samples where H ≤ 0), so brief flickers don’t dominate the story (see third chart below):

How vector memory pressure detection works: Expansion and compression

Vector memory pressure is a single HIGH severity AutoOps event, which adapts to the compression regime:

• Expansion (H > 0): Emphasize shrinking headroom, hence a growing vector off-heap footprint.

• Compression (H ≤ 0): Emphasize ΔV, heap context, I/O, and latency. "Hours until H hits zero" isn’t the main narrative when headroom is already gone.

The detector requires three layers before it fires:

1. Memory carriers: Compression regime, shrinking headroom, or sustained growth in vector off-heap footprint.

2. Operational latch: Search or indexing latency versus rolling baselines, filesystem read stress (paired with latency or heap), indexing throttle, thread-pool queues or rejections, segment creep, or heap circuit breakers when paired with other stress, as circuit breakers alone don’t provide enough evidence to be escalated without corroborating stress.

3. Heap hot: Heap usage elevated versus a 24-hour rolling median on that node, so compression alone on a calm heap doesn’t fire the insight.

That pairing is intentional. Vector pressure without load might be capacity planning, and load without vector pressure might be a different root cause. Together, vector memory, operational stress, and heap heat surface the vector RAM story when the node is actually in trouble, not on every compressed mapping while the heap stays normal.

Validation: Memory pressure detection on a 4 GiB node under kNN load

We stress-tested vector memory pressure detection on 4 GiB Elastic Cloud Hosted deployments with throttled dense-vector ingest (~2,000 docs per minute) and steady kNN search (~8 queries per second). Across Hierarchical Navigable Small World (HNSW), Better Binary Quantization (BBQ) HNSW, and DiskBBQ mapping profiles over 24–48 hours:

• Vector off-heap footprint grew from near zero to about 6 GiB on the tightest runs (more than 4.7 million vectors indexed) in both HNSW test runs.

• Nodes spent most of each run in compression (H ≤ 0), which is expected when vector footprint exceeds total RAM in this model.

• Vector memory pressure stayed off while heap held near 50%, even with compression and pool stress building.

• On both HNSW and BBQ HNSW, the insight fired once heap climbed past ~75% with memory compression and thread-pool queue stress, roughly an hour before heap neared saturation. Node out of memory (OOM) and circuit breakers followed in the same window, as did slow search/indexing. As we can see on the dashboard below, performance drops drastically due to corroborating stress toward the end of the test run:

• On DiskBBQ, where compression was chronic but heap stayed normal, the insight didn’t fire,since storage rather than memory was the limiting factor. Disk and watermark signals are the right path to follow for that profile. As we can see on the screenshot below, all indicators stayed steady with constant performance throughout the test, even though we filled up the disk with more than 70 million vectors on the same instance type:

That timing is the point: Operators get a vector-first read tied to real RAM stress, with subsystem context, rather than an alert on every compressed index or only a red heap chart after the node is fighting on every front.

What to do when AutoOps raises vector memory pressure

Here’s the insight that AutoOps now raises when it detects vector memory pressure:

Recommendations in the product map to concrete actions:

1. Reduce vector footprint where quality allows: Fewer dimensions, quantized mappings, archive or split indices, reindex with a leaner mapping.

2. Tune kNN load: Lower num_candidates, reduce concurrent query rate, narrow filtered kNN where possible.

3. Consider DiskBBQ when HNSW in RAM is the bottleneck (evaluate recall/latency trade-offs for your use case). If you’re already on DiskBBQ and the heap memory is calm, treat disk and watermark insights as the primary signals. Note that DiskBBQ requires an Enterprise license.

4. Right-size RAM when vector off-heap footprint (V) trends up and headroom stays tight.

AutoOps links affected nodes and summarizes regime and stress in plain language. Treat it as “act now, rather than waiting for red on every chart.”

Where AutoOps vector memory pressure monitoring is available

Vector memory pressure is available wherever AutoOps runs against Elasticsearch 9.2+, including:

• Elastic Cloud Hosted (ECH).

• Elastic Cloud Serverless (coming soon).

• Self-managed via Cloud Connect.

AutoOps is included at all subscription levels for supported deployment types and doesn’t consume ECUs on ECH.

What is an Agent Builder OTel trace and what does it capture?

When your agent runs, Agent Builder records everything that happened as an OpenTelemetry (OTel) trace. Think of a trace as a receipt for a single conversation turn. Every LLM request, tool call, and agent action is recorded as an individual span in Elasticsearch, which is a unit of work or operation. When opted in, additional details like user prompts, LLM responses, tool outputs, and conversation IDs are captured as structured span attributes on the chat span. All of this is scoped to your Kibana space.

How to enable agent tracing and privacy controls in Kibana

To begin capturing trace data, ensure the following toggles under Agent Traces in Gen AI Settings are active within your environment:

• agentBuilder:tracing:enabled — This gen AI setting manages the collection of traces and is enabled by default.

Advanced privacy controls, located under the default tracing toggle, also let you collect message content. While prompts and tool outputs are masked by default, you may choose to enable them to support more robust traces:

• agentBuilder:tracing:includeUserPrompts

• agentBuilder:tracing:includeLlmResponses

• agentBuilder:tracing:includeToolDetails

• agentBuilder:tracing:includeSystemPrompt

• agentBuilder:tracing:includeRealNames: Retains real agent/tool names instead of anonymizing to custom

• agentBuilder:tracing:includeRealIds: Retains the actual conversation identifiers instead of the default hashed versions. This means trace data collects original IDs, which can link traces to specific user sessions (PII).

Only enable these if you understand what data your agents handle and have appropriate data governance in place.

How Agent Builder stores OTel trace data in Elasticsearch

The Agent Builder utilizes OpenTelemetry semantic conventions. This results in a structured hierarchy of spans that provides a granular view of the agent's internal logic:

Span

Type

What it captures

`invoke_agent <name>`

CHAIN

Full turn lifecycle, from user input to final reply

`invoke_agent <name>`

AGENT

Single agent execution: reasoning, tool calls, reply

`chat <model>`

LLM

One LLM request: model, latency, token counts

`execute_tool <toolName>`

TOOL

Tool invocation: arguments, duration, result

Trace data is written to a dedicated data stream per Kibana space, keeping conversation data cleanly isolated. To query your traces in Discover, target the index for your space directly:

For the default space, that’s traces-agent_builder.otel-default. If the advanced privacy controls are turned on, then those span attributes will also be shipped to the traces data stream with the original spans. This index lets you query the raw message content to see what's actually being said in conversations. It is best practice to avoid using wildcards to prevent mixing data from unrelated spaces.

Agent Builder ships with a built-in skill called agent-builder-traces, installed automatically whenagentBuilder:tracing:enabled is on. You can use it to ask questions directly about your trace data, making it easy to explore agent behavior without writing ES|QL from scratch.

How to debug agent behaviour with the OTel trace waterfall view

The trace waterfall shows every step of an Agent Builder session as a timeline. To open it, navigate to the specific turn in the conversation UI and select the trace icon.

This launches a waterfall timeline breaking down every step of your agent's execution. At the top level, you'll see the invoke_agent parent span with the full end-to-end duration of your agent run. Nested beneath it are chat spans, each representing a single LLM request and showing exactly how long the model took to respond. Alongside those are execute_tool spans, one per tool call, where you can see which tool was called, what arguments it received, and how long it ran. This allows you to trace the exact sequence your agent followed, pinpoint timing bottlenecks, and see where errors occurred.

How to build token cost dashboards from trace data

Discover gives you raw trace data, but most teams want answers to operational questions like "how many tokens did we burn today?", "which tool is called most often?", and "how many unique users interacted with the agent this week?". These would require a dashboard built directly against the trace data.

There is an Elastic-managed out-of-the-box dashboard called [Elastic] Agent Builder Overview that can be installed by clicking in the top-right corner of the Agent Traces section within GenAI Settings.

It contains basic details spanning Token Usage and Cost, Conversation Volume and Latency, Agent Execution, and Tool Call Frequency and Errors.

However, a custom dashboard may be more efficient. If you want something more tailored, build Lens panels directly against the OTel trace index. Some useful panels to start with:

1. Most active conversations by token spend

create a horizontal bar chart against traces-agent_builder.otel-<space-id>. Set the x-axis to gen_ai.conversation.id and sort descending and limit to the top 10. Set the y-axis to a sum of gen_ai.usage.input_tokensplusgen_ai.usage.output_tokens. Input the formula as: 

Conversations with the most LLM round-trips

Option to create this visualization with a simple ES|QL query that would look like:

There is also a dashboard-management skill that can be used to help create traces visualizations using natural language.

How to set token cost alerts for Agent Builder conversations

Token consumption is the most direct cost lever for LLM-based agents. A single runaway conversation can blow through your monthly budget before anyone notices.

Elastic alerting lets you define a threshold rule directly against the trace data. Navigate toObservability > Alerts > Manage Rules > Create Ruleand selectElasticsearch queryas the rule type.

A rule that fires when any single conversation exceeds 256,000 tokens looks like this as an ES|QL rule:

Set the schedule to run every 15 minutes and configure the action to send a Slack notification or open a PagerDuty incident. The gen_ai.conversation.id value in the alert payload gives you the exact conversation to inspect.

What’s coming next for Agent Builder observability

Agent traces give you visibility that goes far beyond debugging. Once you've built dashboards and configured alerts against Agent Builder trace data, you have a live pulse on how your agents are behaving in production. If you haven't already, spin up Agent Builder in your Kibana space, make sure tracing is enabled, and run a few conversations. Check Discover, pull up the waterfall view, and see what your agent is actually doing under the hood.

This is the first in a series of posts on Agent Builder observability. Coming up, we'll go deeper on using theagent-builder-traces skill to query your data conversationally, building custom evaluation pipelines from trace data, and using traces to feed conversation history back into your agents. Your agents have been keeping secrets. It's time to make them talk.

Why AutoOps for Elasticsearch needs clearer prioritization

Running Elasticsearch at scale requires administrators to monitor cluster health, performance, capacity, and configuration at the same time. AutoOps now provides a clearer way to distinguish conditions that threaten cluster functionality from significant but less urgent degradation. The redesigned interface also follows familiar Elastic Cloud Console patterns, making active issues easier to find and investigate.

What changed in AutoOps: severity, navigation, configuration, and page design

The monitoring engine remains the same. The redesigned layout, navigation, and workflows now follow familiar Elastic patterns.

A clearer severity model

We added Critical as a new severity level for conditions that pose an immediate threat to cluster functionality and require urgent intervention. Several events previously classified as High are now Critical. Others are now Medium because they represent potential risk rather than active, significant degradation. The reclassified events are:

• Promoted from High to Critical: Disk Watermark Flood Stage, Master Not Discovered, and Status Red.

• Demoted from High to Medium: Disk Watermark Low Threshold, Disk Watermark Low, and Disk Watermark Configuration Incorrect.

Severity

What it means

Critical

Immediate threat to cluster functionality. Urgent intervention required.

High

Significant degradation to usability, performance, or stability.

Medium

Potential risk that can escalate if left unaddressed.

Low

Minor anomalies with minimal operational impact.

Info

Routine operational updates and configuration changes. No action required. (Coming in a near-future update).

Every severity level ships with an updated icon set and color palette. Levels are fixed so teams can build consistent runbooks and notification filters: route Critical and High events to PagerDuty or Slack, keep Medium and Low in the console for periodic review, and when Info arrives, use it for awareness without alert fatigue.

Deployment view: open events and history, side by side

The redesigned deployment view presents the existing Open events and Event history tabs in a clearer layout.

Event flyout: a clearer view of what matters

The event detail flyout is redesigned around action. High-severity events include a notification callout and an interactive badge that shows whether alerts are configured and links directly to setup. Recommendations collapse by default so the core event stays in focus. Settings live in the flyout menu; share is a separate icon in the header. The Dismiss action appears only when your role has the required admin permissions and the event is dismissible.

AutoOps overview: triage active events across your Elasticsearch fleet

The Overview page is reorganized around how operators scan an estate. Elasticsearch context sits directly under the page header, and active events appear as event ribbons below the deployments table. Each ribbon shows the latest active event in your selected time range; if the same event type is open on other deployments, a new badge lets you expand the view without opening each resource individually. Event search moved to the left for quicker filtering.

The “Events over time” chart moved off Overview to keep this page focused on fleet-level triage; open a single deployment when you need that timeline.

Nodes, Shards, and Indices are designed with easier navigation and information hierarchy

Nodes view now uses updated chart components and the Elastic UI color scheme, with clear expansion indicators on accordion sections. Event and instance lists that duplicated deployment-level views were removed to reduce noise.

Shards view improves node selection and groups view controls in the upper-right corner. A horizontal scrollbar supports wider layouts, and the time slider now uses native Elastic UI components. Node selection in Shards view now works across larger clusters and presents up to 100 nodes at a time.

Index view keeps the Indices table experience you already use, including sorting, time-range brushing, and chart zoom behavior tuned for meaningful ranges.

Template Optimizer

The Template Optimizer now provides a searchable list of templates ordered by the most recently identified recommendations. You can open each recommendation directly or expand the JSON panel to inspect the complete template.

Configure notifications and event settings

Notification settings now include connector search, clearer filters, and a simpler connector editing flow. Event settings moved from a popup to a flyout, matching the pattern used across AutoOps. Notification reports retain the same 10-day history window with minor layout updates, and dismiss events use updated confirmation components aligned with Elastic UI.

Navigation and controls

The deployment picker now shows deployment ID and real-time cluster status, with copy actions for deployment name and ID in the dropdown sub-menu. Node selection supports select-all, select-by-tier grouping, and clear master node indication. The date picker follows the same relative-range and custom-range model used in Kibana and other Cloud Console monitoring views.

AutoOps roadmap: API, MCP, CLI, and agentic experience

Looking ahead, we are building toward a headless, agentic AutoOps experience. A forthcoming public AutoOps API will make insights and raw metrics available outside the AutoOps interface. Administrators and agents will be able to query the API directly or store its data in Elasticsearch. The API will also provide the foundation for integrations with MCP, Elastic Agent Builder, the Elastic CLI, Kibana, and native AutoOps chat.

• Hosted MCP server: Make AutoOps insights available to MCP clients such as Claude and Cursor.

• Native Elastic Agent Builder tool: Use AutoOps insights in Elastic Agent Builder.

• Elastic CLI support: Access the AutoOps API through the Elastic CLI.

• AutoOps in Kibana: Surface relevant insights and metrics within Kibana.

• Native AutoOps chat: Investigate cluster issues through an agentic chat experience within AutoOps UI in Elastic Cloud Console.

The application redesign is the foundation; these surfaces will meet operators where automation and AI already live. Read more about what is coming on the Elastic public roadmap.

How to start using the redesigned AutoOps in Elastic Cloud Console

Sign in to Elastic Cloud Console, open a deployment, project, or connected cluster, and select AutoOps from the navigation. Learn more in the AutoOps documentation.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

When dealing with dozens of users who actively connect and use your platform backed by an Elasticsearch cluster, it’s important to quickly grasp what potential bottlenecks are and how to overcome them. The main question is: Where to start? Here’s your potential decision path:

The path starts with AutoOps to detect cluster-level problems, like resource pressure or bad shard distribution. If the cluster is healthy, the Profile API helps identify slow queries; candidates can come from the slow query log or application-side logging. Next, ES Rally benchmarks index settings that may limit throughput. If all three come back clean, the bottleneck is on the client side.

How to detect cluster-level performance problems with AutoOps

The first question is always the same: Is the cluster itself the problem?

AutoOps is designed to provide real-time cluster diagnostics, deliver tailored advice, and help you improve the health of your clusters quickly. It comes by default in Elastic Cloud, and it has been added recently as a free option for self-hosted configurations.

It’s very easy to set it up, and it takes no more than five minutes to see data flowing into a comprehensive list of graphs that AutoOps offers you. The idea behind it is to install an Elastic Agent close to your cluster that reports back to the AutoOps platform which is connected with your Elastic Cloud account. Here’s the installation guide.

What’s nice about it is that it provides you with instant warnings and suggestions that are hard to spot through existing monitoring cluster technology. What’s really useful is that it also shows solutions to the issues. 

In our delivery logistics scenario, AutoOps surfaced one finding that explained the user-reported slowness: severe load imbalance across the cluster. Node es01 was handling nearly all traffic while the other three sat idle, with search latency spiking to 30 seconds, as we'll see in the graphs in the next section.

How AutoOps surfaces node hotspotting and shard imbalance

AutoOps node performance graphs revealed that es01 was handling nearly all indexing traffic (5.6 docs/sec) while es02, es03 and es04 were idle.

Below, we see that CPU usage was concentrated on a single node (es01) which was hosting the heavier index, while the other three nodes were mostly idle.

The next signal of unusual behavior appeared in the search latency graph. Using the AutoOps per-node latency view, we uncovered some unexpected results.

While other nodes show no latency, the heavy-loaded node shows symptoms of latency that affects search apps. Not only was it a write node, but also it was the one serving all search requests. During analysis, we found that a delivery index was stored on only one node, instead of being distributed across all four. That was the root cause of the issue. By reindexing the data (with increased number of primary shards), we balanced the query load and eliminated the primary issue.

This article focuses on a single AutoOps finding to keep the diagnostic flow clear. For deeper dives into the kinds of issues AutoOps surfaces, see the dedicated articles on hotspotting, high CPU usage, and long-running queries.

How to find slow Elasticsearch queries using the Profile API

Once the cluster is healthy, the next question is: Are there specific, expensive queries? Finding candidates is the first step. You can explore them either by using Elasticsearch’s slow query log or through application side logging, by measuring time required for every search request. In our case, infinite scroll on a tracking delivery screen was a problem while users would scroll deeper into results.

The Profile API is a powerful tool for analyzing long-running queries and finding search bottlenecks. Getting started is simple: Just set profile=”true” in any of your search queries, and the responses will contain a profile section with detailed timing breakdown. It highlights which phase dominates execution time: the query phase, aggregations, or data fetching.

Let’s take a closer look at the deep pagination case. In a case of infinite scroll in a mobile application, the user continues scrolling and the app responds by fetching 100 documents per request.

As users continues scrolling, the requests can reach very deep pagination levels, while the from parameter grows:

The profile data, summarized below and converted from nanoseconds to milliseconds, makes the bottleneck clear: Most of the time is spent outside the query and fetch phases on coordination work caused by the large from offset. Elasticsearch must collect 9,100 matching documents, sort them, discard the first 9,000, and return only the requested 100.

Component

Phase

Time

ConstantScoreQuery contains

Query

34.4ms

BooleanQuery

Query

31.1ms

QueryPhaseCollector contains

Collect

22.1ms

SimpleFieldCollector (9,100-doc priority queue)

Collect

19.1ms

FetchPhase

Fetch

12.1ms

Other steps, like request parsing and deserialization, queue wait time, response building

1,885.4ms

Total

2,004.2ms

Fixing deep pagination with search_after

In this case, the search_after approach is up to twice as fast because it avoids scanning and discarding earlier results. Instead, it resumes from the last document returned on page 90, using its delivery_pickup_datetime value (in epoch milliseconds) as a cursor and fetching only the next 100 records.

As shown in the comparison below, performances are much better with the search_after approach.

Component

Phase

Time

ConstantScoreQuery contains

Query

6.8ms

BooleanQuery

Query

6.2ms

QueryPhaseCollector contains

Collect

2.6ms

PagingFieldCollector

Collect

2.0ms

FetchPhase

Fetch

2.2ms

Other steps like request parsing and deserialization, queue wait time, response building

9.4ms

Total

29.2ms

Performance comparison

Component

Deep pagination (from:9000)

search_after

Time saved

% Saved

ConstantScoreQuery

34.4ms

6.8ms

27.6ms

80.2%

BooleanQuery

31.1ms

6.2ms

24.9ms

80.1%

QueryPhaseCollector (total)

22.1ms

2.6ms

19.5ms

88.2%

FieldCollect

19.1ms

2.0ms

17.1ms

89.5%

FetchPhase

12.1ms

2.2ms

9.9ms

81.8%

Total

2,004.2ms

29.2ms

1,975ms

98.5%

The total query time drops from 1,004 ms to 29 ms (a 98.5% improvement) almost entirely because Elasticsearch no longer has to build and discard a 9,000-document priority queue on every request.

How to benchmark Elasticsearch index settings with ES Rally

With the cluster healthy and queries optimized, the final question is: Are the index settings themselves a bottleneck?

ES Rally is the official benchmarking tool made by Elastic. Its key strength is reproducibility: You run the same workload against two configurations on the same cluster and hardware, so any difference in results is purely down to the settings you changed. ES Rally is able to measure improvements under identical conditions: same cluster, hardware, and dataset.

Why default Elasticsearch index settings limit bulk indexing throughput

In many cases, slow indexing is caused by default index settings that are suitable for development but not production. For instance, the default refresh_interval of 1 second increases resource usage because every refresh creates a new searchable segment, and a replica count of 1 doubles the number of write operations per indexing request.

Our delivery logistics platform, for example, ingests thousands of new records daily in bulk, exactly the kind of workload where these defaults hurt. Setting refresh_interval to -1 disables refreshing during the bulk loading phase, and temporarily dropping replicas to 0 halves the write operations. Both settings are restored after the import is complete.

In this demo, we won’t focus on how to prepare custom data for benchmarking, but it’s worth mentioning a nice article about it.

For benchmarking purposes, you set up two folders: one with the current settings and another with the contender settings. The sample dataset in this case contains 1 million records. All the files described below can be found in this repository.

Both track folders contain an index-settings.json file. These files let you adjust mappings, shard counts, replica settings, and field types; for example, converting a field from double to scaled_float or from text to keyword. Because ES Rally tracks can be rerun quickly, it’s easy to experiment with different configurations and evaluate new optimization ideas as you iterate.

The next step is to run race commands (for current and contender race) to gather stats.

Quick tip: Using meaningful --race-id values (rather than the auto-generated ones) makes the comparison command much easier to run.After both races are complete, compare the results:

Additionally, reports are generated into csv files that can be easily manipulated to extract important data.

Metric

Current settings

Contender

Change

Cumulative indexing time

0,826517

0,755967

+8.54%

Mean throughput

11,090 docs/s

50,921 docs/s

+359 %

Median throughput

10,821 docs/s

51,304 docs/s

+374 %

Min throughput

10,335 docs/s

41,387 docs/s

+300 %

Max throughput

13,006 docs/s

55,342 docs/s

+326%

The contender configuration delivers roughly 3–4× faster indexing throughput, purely from adjusting refresh_interval and replica count during the bulk load phase.

In practice, a single comparison is rarely enough. Common follow-up experiments include converting double fields to float or scaled_float, changing text fields to keyword, adjusting shard count, and tuning the refresh interval. Because ES Rally tracks rerun quickly, iterating through these options is straightforward.

What to do when Elasticsearch isn't the bottleneck

If AutoOps shows a healthy cluster, the Profile API shows fast queries, and ES Rally confirms that index settings are not the limiting factor, the bottleneck is on the client side. The Elasticsearch side is no longer the place to look. Common application-layer causes:

• Network latency between the client and the cluster, especially across regions, VPNs, or proxies.

• Client-side deserialization overhead on large response payloads. Use _source exclude or the fields parameter to return only what the client actually needs.

• Client-side queueing before requests reach the cluster. The Elasticsearch client uses an HTTP connection pool. Under concurrent load, requests wait in line for a free connection. The Profile API never sees this wait because the request hasn't left the client yet.

• Single search calls where _msearch would batch independent queries into one network round trip.

• Single-document indexing where the _bulk API would amortize per-request overhead across many documents.

The diagnostic value of AutoOps, the Profile API, and ES Rally is precisely that they let you definitively rule out the cluster, the query, and the index settings before turning to the application code. When the three tools come back clean, the investigation moves out of Elasticsearch and into the client.

For investigating these on the application side, Elastic APM is the natural next tool: It traces request paths through the client and surfaces exactly the kind of pre-cluster wait time the Profile API can't see.

Conclusion

Going back to the original problem, slow search on a delivery logistics platform, we traced the issue from the infrastructure level down to the query and settings level:

• AutoOps revealed an uneven shard distribution that concentrated all query load on a single node, causing latency spikes of up to 30 seconds for end users.

• The Profile API showed that deep pagination was the source of the slow queries. Switching from from/size to search_after eliminated 98.9% of the latency.

• ES Rally confirmed that optimizing index settings during bulk ingestion, specifically refresh_interval and replica count, can increase throughput by 3-4×.

Each tool answers a different question. AutoOps gives you the "Is something structurally wrong?" view. The Profile API answers "Why is this specific query slow?" And ES Rally validates "Do these changes actually improve things, and by how much?" Used together and in this order, they cover the full diagnostic surface for the Elasticsearch side of the application. When all three come back clean, the next step is to look at the client, as outlined in the previous section.

FIPS 140-3 support for Elasticsearch and Kibana is generally available in Elastic 9.4 for self-managed deployments. NIST has stopped accepting new FIPS 140-2 submissions, with existing certificates winding down in September 2026. Every layer of your infrastructure needs to catch up, and your search and analytics platform is no exception. Federal programs, defense integrators and regulated enterprises are actively moving procurement requirements to 140-3. With Elastic 9.4, your stack can answer "yes."

Two problems, one deadline: why FIPS 140-2 is no longer enough

If you've been running Elasticsearch in FIPS 140-2 mode, you already know the value of having a compliant Elastic Stack. But two pressures are converging:

• The standard is sunsetting. FIPS 140-2 certificates are being phased out. Procurement officers, auditors, and authorization bodies are shifting their requirements to 140-3. A stack that only supports the old standard is a stack with an expiration date on its compliance story.
• Auditors don't accept "close enough." It's not sufficient to run FIPS-approved algorithms. Your application layer needs to be explicitly configured for FIPS mode, with nonapproved algorithms disabled and cryptographic boundaries clearly documented. Partial compliance is noncompliance.

Can't I just put Elasticsearch behind a FIPS-compliant load balancer or run it on a FIPS-hardened OS? No, and here's exactly why: FedRAMP's network security requirements apply to cryptographic operations at the application layer, not just the network boundary. Intranode communication, keystore encryption, and credential hashing all need to happen inside a validated module. A compliant perimeter around a noncompliant application layer isn't a FIPS deployment; it's an audit finding waiting to happen.

What FIPS 140-3 mode does in Elasticsearch 9.4

When you flip FIPS mode on, Elasticsearch and Kibana restrict every cryptographic operation to FIPS-approved algorithms and delegate all of it to the validated module in your runtime. Here's what that covers and how.

• High-grade security Transport Layer Security (TLS) everywhere, no exceptions. Node-to-node transport, the REST API over HTTPS, Kibana talking to Elasticsearch: All of it uses only FIPS-approved cipher suites. Noncompliant suites aren't deprioritized. They're rejected.
• PBKDF2 replaces bcrypt for password hashing. Bcrypt isn't FIPS-approved, so in FIPS mode, Elasticsearch switches to PBKDF2 for the native realm, file realm, and any stored credentials. Your users and service accounts stay protected with an algorithm that the auditor won't flag.
• Keystore encryption stays inside the boundary. Secrets in the Elasticsearch keystore, API keys, repository credentials, and encryption keys for Kibana's saved objects are wrapped with FIPS-approved key derivation and encryption. No gaps between the cluster is compliant and the secrets are compliant.
• You supply the cryptographic module. FIPS 140-3 support in Elasticsearch is built on the Bouncy Castle FIPS Java API 2.0, a FIPS 140-3 validated cryptographic module that runs inside the Java Virtual Machine (JVM). Elasticsearch delegates all cryptographic operations to Bouncy Castle; it doesn't implement its own crypto functions. Elasticsearch uses your own FIPS 140-3 JVM and Bouncy Castle FIPS provider, giving you full control over your cryptographic boundary and module versioning.

Kibana takes a different path, running in a Node.js environment configured with a FIPS-compliant OpenSSL 3 provider. Together, both components operate within clearly defined cryptographic boundaries clean enough to diagram for an auditor.

Who needs FIPS 140-3 support in Elasticsearch

• Federal and defense teams operating Elastic inside FedRAMP boundaries, Cybersecurity Maturity Model Certification–scoped (CMMC-scoped) environments, or Defense Information Systems Agency (DISA) Security Technical Implementation Guide–hardened (STIG-hardened) infrastructure. You can now upgrade to 9.x without punching a hole in your authorization documentation. Your Authorization to Operate (ATO) package references FIPS 140-3, not a soon-to-expire 140-2 certificate.
• Financial services and healthcare organizations where Payment Card Industry Data Security Standard (PCI DSS), Sarbanes‑Oxley Act (SOX), or Health Insurance Portability and Accountability Act (HIPAA) audits ask how your search infrastructure handles cryptography. FIPS mode gives your compliance team a one-word answer instead of a three-paragraph explanation.
• Anyone fielding Do you support FIPS 140-3? in a procurement questionnaire. That question is showing up in enterprise requests for proposal (RFPs), partner security assessments, and insurance underwriting checklists. With 9.4, the answer is yes.

Migrating from FIPS 140-2 to FIPS 140-3

If you're running FIPS 140-2 on Elastic 8.x today, you don't have to jump to 9.x on day one. Elastic 8.19 continues to support FIPS 140-2, and FIPS 140-3 support is available starting in 8.19.15. That gives you two paths:

• Stay on 8.19.15+, and upgrade the standard in place. If you're not ready to move to 9.x, you can switch from FIPS 140-2 to 140-3 on your current major version. Your cluster stays put, and your compliance posture moves forward. FIPS 140-2 certificates remain valid through September 2026, so you have a window. But the earlier you transition, the less you're depending on a sunset timeline, and you can also benefit from new features!
• Move to 9.4, and land on 140-3 directly. If you're planning a major version upgrade anyway, 9.4 gives you FIPS 140-3 from the start, along with everything in the 9.x line: Elasticsearch Query Language (ES|QL) latest functionalities, improved ingest, updated security detections, and performance improvements. No compliance trade-off required.

Stay on 8.19.15+

Upgrade to 9.4

FIPS standard

140-3 (from 8.19.15)

140-3 (from the start)

Major version change

No

Yes

New 9.x features

No

Yes

FIPS 140-2 certificates valid until

September 2026

N/A (lands on 140-3 directly)

Kibana covered

Yes

Yes

Either way, the configuration model will feel familiar. You enable FIPS mode in your YAML config, point to a FIPS-validated JVM, and the stack handles the rest. And Kibana is covered on both paths: Your visualization and analytics layer operates within the same compliant boundary as Elasticsearch.

How to enable FIPS 140-3 in Elasticsearch 9.4

FIPS 140-3 support is available now in Elastic 9.4 for self-managed deployments with a Platinum or Enterprise subscription, the same licensing tier as FIPS 140-2. For setup instructions, supported JVM versions, configuration details, and known limitations, see the FIPS compliance documentation.

Download Elastic 9.4, and give your compliance team some good news.

Questions? Your Elastic account team can help scope a FIPS deployment, or drop into the Elastic community forums to compare notes with other operators running in regulated environments.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Frequently asked questions

Does Elasticsearch support FIPS 140-3?

Yes. FIPS 140-3 support for Elasticsearch and Kibana is generally available in Elastic 9.4 for self-managed deployments. All cryptographic operations are delegated to the Bouncy Castle FIPS Java API 2.0, a FIPS 140-3 validated module. A Platinum or Enterprise subscription is required.

When does FIPS 140-2 expire?

NIST stopped accepting new FIPS 140-2 module submissions. Existing FIPS 140-2 certificates remain valid through September 2026. Organizations running Elasticsearch in FIPS 140-2 mode should plan their migration before that deadline to avoid gaps in their compliance documentation.

Can I migrate from FIPS 140-2 to FIPS 140-3 without upgrading to Elasticsearch 9.x?

Yes. FIPS 140-3 support is available in Elastic 8.19.15 and later. You can switch from FIPS 140-2 to FIPS 140-3 on your current major version without moving to 9.x. Alternatively, upgrading directly to Elastic 9.4 lands you on FIPS 140-3 from the start.

What cryptographic changes does FIPS mode make in Elasticsearch?

When FIPS mode is enabled, Elasticsearch restricts all operations to FIPS-approved cipher suites, replaces bcrypt with PBKDF2 for password hashing, and delegates all cryptographic functions to the Bouncy Castle FIPS provider. Non-approved cipher suites are rejected, not just deprioritized.

Does Kibana support FIPS 140-3?

Yes. Kibana operates in a Node.js environment configured with a FIPS-compliant OpenSSL 3 provider. Both Elasticsearch and Kibana run within clearly defined cryptographic boundaries when FIPS mode is enabled.

Why isn't a FIPS-hardened OS or load balancer sufficient for FedRAMP compliance?

FedRAMP's network security requirements apply to cryptographic operations at the application layer, not just at the network boundary. Intranode communication, keystore encryption and credential hashing must all occur inside a validated cryptographic module. A compliant perimeter around a non-compliant application layer is an audit finding, not a FIPS deployment.

Is Elasticsearch FIPS 140-3 support available on Elastic Cloud?

Elastic 9.4 FIPS 140-3 support covers self-managed deployments. Cloud deployment availability is not covered in this announcement. Contact your Elastic account team or check the Elastic documentation for cloud FIPS roadmap details.

In most ecommerce deployments, economic signals, like profit margin and product popularity, are either ignored in search ranking or applied as static, global weights. A fixed margin boost might push high-margin products up across every query, which works for "chocolate" (where shoppers are open to suggestion) but backfires for "baby formula" (where shoppers want the trusted, popular brand).

The governed control plane makes it possible to treat economic optimization as a per-query decision, expressed as policy data and managed through the same admin UI as every other governance mechanism. A merchandiser can say "for chocolate queries, prioritize margin" and "for baby formula queries, prioritize popularity", without writing code, without deploying changes, and with full auditability.

For the mathematical foundation of margin and popularity boosting in Elasticsearch, including the logarithmic scaling formula and factor tuning explanation, see Boosting e-commerce search by profit and popularity with the function score query in Elasticsearch.

Two business signals: Margin and popularity

Every product document in our product catalog carries two numeric fields:

• margin: The product's profit margin as a percentage (0 to 200 in our dataset).

• popularity: A relative sales volume metric (0 to 10,000 in our dataset), such as weekly average units sold.

These fields represent two fundamentally different business objectives. Margin optimization pushes profit per transaction. Popularity optimization pushes conversion probability since products that many shoppers buy are products that the current shopper is likely to buy.

The baseline: Global boosting with business signals

Before introducing per-query policy overrides, the system applies a default boost for both margin and popularity. These are implemented using Elasticsearch's field_value_factor with logarithmic scaling inside a function_score query as described in Boosting e-commerce search by profit and popularity with the function score query in Elasticsearch.

The design has three properties worth noting:

• Calibrated range. Each signal's factor is calibrated so that it contributes at most approximately +1.0 to the boost multiplier at the top of its range. Combined with a baseline weight of 1, the final multiplier ranges from 1.0 (a product with zero margin and zero popularity) to approximately 3.0 (maximum margin plus maximum popularity). A product with strong business signals scores roughly 3x higher than an identical product with none, regardless of the BM25 score magnitude.

• Logarithmic scaling. The ln1p modifier grows fast at small values (rewarding incremental gains) but flattens at high values (preventing runaway scores from a single dominant product). This also makes the system resilient to data distribution changes: If the maximum popularity in a dataset shifts significantly, the boost curve stretches rather than breaking.

• Multiplicative, not additive. The business-signal boost is applied multiplicatively against BM25 (boost_mode: "multiply") rather than added to it. BM25 scores vary dramatically across queries, so an additive boost would have inconsistent impact depending on query specificity. Multiplicative scaling guarantees a consistent percentage uplift regardless of the absolute BM25 magnitude.

Per-query boosting overrides through policies

The default weights (1.0 for both margin and popularity) apply to every query. But the governed control plane makes it possible to override these weights on a per-query basis through the same policy engine described in Part 3 and Part 4.

Each policy document has two optional fields: margin_boost_weight and popularity_boost_weight. When a policy matches a query and includes weight overrides, those values flow through to the function_score construction, replacing the defaults.

Why per-query control matters

Consider two queries and why they demand different economic optimization strategies.

Margin boosting: Chocolate

A shopper searching for "chocolate" is browsing. They'll be satisfied by many chocolate-related products. The retailer's store-brand chocolate truffles at 60% margin might be just as appealing as the name-brand bar at 15% margin. Aggressive margin boosting pays for itself if the shopper doesn't care which chocolate they purchase and buys one of the margin-boosted hits.

Chocolate results without margin boosting

To isolate the effect of per-query margin boosting, we first disable margin boosting entirely for this query (margin boost weight: 0). Without any margin signal, the ranking is driven by text relevance. In our dataset, the first hit has a margin of 10 and the next one has a margin of 84 (out of a max of 200) as follows:

Setting a margin boost on queries for “chocolate”

A merchandiser who decides that “chocolate” queries should prioritize margin makes that change in the admin UI, tests it against representative queries, and promotes it to production. The change takes effect on the next query. No engineering ticket, no deployment, no code change. The following "chocolate" policy sets margin_boost_weight: 3.0, which ensures that searches for chocolate will aggressively promote high-margin items.

Chocolate results with margin boosting

With the above margin boost policy enabled, the higher-margin chocolates with a margin of 197 and 184 are boosted to the top of the results as follows (remember that the maximum margin in our dataset is 200):

Popularity boosting: Baby formula

A parent searching for baby formula is not experimenting. They want the product that other parents trust. Pushing a high-margin store-brand formula above the established brand that thousands of parents are buying would feel wrong and erode trust. Popularity is the right signal here because it functions as social proof for a high-stakes purchase.

Baby formula results without a popularity boost

To isolate the effect of per-query popularity boosting, we first disable popularity boosting entirely for this query (popularity_boost_weight: 0). Without any popularity signal, the ranking is driven by text relevance. In this example, the top hit has a popularity of 50 on a scale that goes up to 10,000.

Setting a popularity boost on queries for “baby formula”

A "baby formula" policy sets popularity_boost_weight: 5.0 and margin_boost_weight: 0; formula searches prioritize what's popular, completely ignoring margin.

Baby formula results with popularity boosting

If we enable the above rule, then the most popular baby formula (Lactogen 2 with a popularity of 9979) will be boosted to the top of the results, as shown below.

Disabling business signals: Clearance

Not every query benefits from economic boosting. A shopper searching for "clearance" is looking for deals; margin and popularity are both irrelevant to that intent. A high-margin product is the opposite of what the shopper wants, and a popular product may not be on clearance at all.

A "clearance" policy sets margin_boost_weight: 0 and popularity_boost_weight: 0, which disables both business signals entirely. Results are ranked on pure text relevance with no economic influence. This completes the design space: Policies can amplify either signal independently, rebalance them, or turn them off altogether.

How overrides flow through the control plane

When the percolator returns matching policies, the control plane checks for margin_boost_weight and popularity_boost_weight fields on the highest-priority matching policy. If present, those values replace the defaults in the RewriteState. If no matching policy includes weight overrides, the default values (1.0 for both) are used.

The weights then flow through to the function_score construction when the final Elasticsearch query is assembled. The structure of the function_score doesn't change; only the weight values on the margin and popularity functions.

Weight overrides participate in the same governance model as every other policy mechanism. They’re subject to priority ordering: A Christmas campaign policy with margin_boost_weight: 0.5 will override a product-category policy with margin_boost_weight: 3.0 if the campaign policy has higher priority. The cascading transformation model from Part 3 applies: Economic optimization parameters are just another field in the policy's execution plan.

Interaction with other policies

Per-query weight overrides compose naturally with the constraint enforcement, conflict resolution, and personalization mechanisms described in earlier parts of this series.

Consider a search for "cheap chocolate" during a Christmas campaign, with a shopper who has a purchase history and belongs to a vegan cohort. The control plane processes this query through the full governance stack:

1. The "cheap" policy extracts the price constraint and removes "cheap" from the query.

2. The "chocolate" policy sets margin_boost_weight: 3.0 and constrains results to chocolate categories.

3. The “Christmas campaign” policy (higher priority) overrides the category constraint with seasonal categories and adjusts the price ceiling.

4. The “vegan cohort” policy applies a soft boost to vegan-certified products.

5. The margin and popularity boosts are applied with the governed weights (margin at 3.0× from the “chocolate” policy, popularity at the default 1.0×).

6. The shopper's purchase history boosts are applied as the outermost scoring layer.

Every layer stacks multiplicatively. The economic optimization weights are governed by the same policy framework that controls category constraints, campaign overrides, and cohort-specific boosts. A merchandiser can tune all of these through the admin UI, all without code changes.

This example also illustrates where economic optimization sits in the scoring stack. The layers nest in a deliberate order: the base query (keyword or semantic match), then governance constraints (hard filters and soft boosts from Part 3 and Part 4), then business-signal boosts (margin and popularity with governed weights), and then purchase history personalization (Part 6). Each layer wraps the previous one, and the effects compound multiplicatively. Governance controls what appears. Economic optimization influences what ranks highest from the retailer's perspective. Personalization adjusts ranking further from the shopper's perspective.

Tuning guidance

The factor values in the baseline function_score are calibrated for the demo dataset's field ranges. A production deployment with substantially different ranges for margin or popularity should recalibrate the factors so that each signal contributes a consistent maximum boost. The logarithmic scaling provides built-in resilience to outliers and distribution shifts, but the factors are worth reviewing whenever the underlying data changes significantly. For the calibration methodology, see Boosting e-commerce search by profit and popularity with the function score query in Elasticsearch.

From economic optimization to agentic AI

The governed control plane now handles intent classification, constraint enforcement, conflict resolution, personalization, and economic optimization, all expressed as policy data, all managed through a business-editable admin UI, and all composable through a deterministic transformation framework.

The final post in this series asks what happens when the input to this system isn’t a search string typed by a human shopper, but an intent string extracted by an AI agent, and why the deterministic properties of the governed control plane become even more critical when the upstream decision-maker is probabilistic.

Put governed ecommerce search into practice

The per-query economic optimization described in this post (policy-governed margin and popularity weights composing with governance constraints, personalization, and campaign overrides) was designed and built by Elastic Services Engineering as part of our repeatable ecommerce search accelerators. Contact Elastic Professional Services.

Join the discussion

Have questions about search governance, retrieval strategies, or ecommerce search architecture? Join the broader Elastic community conversation.

From policy logic to retrieval architecture

Part 3 and Part 4 provided a technical deep dive into the governed control plane and its implementation using the Elasticsearch percolator. Once the logic layer has identified which policies to apply, the system must address the retrieval strategy used to execute the search.

Managing the transition from precision to recall is a critical function of any ecommerce search engine. For example, a basic search implementation often defaults to broad keyword matching. If a shopper searches for "organic Pink Lady apples", this can lead to irrelevant results, such as apple-scented dish soap, apple juice, or organic pink grapefruit, appearing at the top of the list simply because they share a common term. While these items are technically matches, they fail to satisfy the user's intent and typically lead to high bounce rates. However, a "No results" page is equally detrimental to conversion. This conflict is resolved by implementing a three-tier execution model, which uses the governed control plane to orchestrate a principled fallback strategy.

The three-tier execution model

This architecture executes up to three retrieval tiers in a sequence, each with a specific matching logic.

Highest tier: Strict matching

Strict matching is a lexical match that requires that all query terms appear in the product metadata.

• The logic: A search for "organic navel oranges" returns only products containing all three terms.

• Application: This tier provides the highest precision. When a customer types a precise product name, such as "organic navel oranges", they’re typically seeking that exact item rather than an alternative.

Mid-tier: Relaxed matching

If the strict tier fails to return sufficient results, the system expands the search parameters.

• The logic: This tier allows for a subset of terms to lexically match, using Elasticsearch's minimum_should_match logic.

• Application: Relaxed matching maintains lexical grounding. A search for "organic navel oranges" might surface "navel oranges" (missing the "organic" term) or "organic oranges" (missing the "navel" term). These represent intuitive, keyword-based alternatives for the shopper.

Lowest tier: Semantic matching

• The logic: This tier uses vector/semantic embeddings (such as Elastic Learned Sparse EncodeR [ELSER], E5, or Jina) to retrieve conceptually related products, regardless of direct keyword overlap.

• Application: A search for "organic navel oranges" might surface "mandarins" or "clementines”. This serves as the final retrieval tier, intended to provide relevant options when literal keyword matches are unavailable.

To see this multi-tier orchestration in action and how the Engine steps down from lexical to semantic matching, watch the video: Eliminating Zero-Result Pages: PRISM’s Multi-Tier Search Fallback.

Tier orchestration: The "bucket filling" logic

While the governed control plane provides the logic and the queries for each tier, the application layer is responsible for the execution. The application executes these tiers sequentially and excludes lower tiers once the accumulated result count on the first page reaches or exceeds 10 items (or whatever number of results you want to display on the first page). This threshold ensures a full first page of results while prioritizing the most accurate retrieval method.

Scenario 1: High-intent search ("oranges")

The first tier returns 15 hits. Since 15 is more than 10, the current result set is locked to only strict matches (which can be paged through) and subsequent tiers are not executed.

Scenario 2: Specific but limited results ("organic blood oranges")

The strict tier finds only four items. Since this is less than 10, the system triggers the relaxed tier, which finds 12 more relevant products. The combined total (16) meets the threshold of 10, so the current result set is locked to the strict and relaxed tiers. Subsequent paging will only surface results from these two tiers (preventing lower-quality semantic hits from appearing on later pages).

Scenario 3: Abstract or intent-based search ("high vitamin C snacks")

Keyword matches are limited (only five hits between tiers 1 and 2). The system triggers the semantic tier to find conceptually relevant items, such as kiwis, guavas, or red peppers, to fill the result set. The result set for this query includes products from all tiers.

This orchestration optimizes for latency, as the computational cost of the semantic tier is only incurred when the keyword-based tiers are insufficient. Additionally, this allows fast-responding keyword results to be displayed while semantic results are integrated shortly after, maintaining a responsive user interface.

Determining intent via tier activation

The logic used to fill the first page serves a critical secondary purpose: It acts as a diagnostic for user intent. The application uses the logic returned by the governed control plane to determine which tiers remain active for the current result set and paging.

If the strict and relaxed tiers together yield fewer than 10 results, the query is likely exploratory or abstract. In this case, activating the semantic tier is a benefit. Because the query is diagnosed as exploratory, the system allows the shopper to page through the entire depth of the semantic results. This provides access to conceptually related alternatives that lexical matching would have missed, which is appropriate for an abstract search.

Conversely, if the strict tier returns a robust set of results (for example, 30 hits), it confirms that the system has found high-precision matches. The user can page through those 30 hits and will likely find what they’re looking for. In this scenario, there’s no need to provide additional, less relevant exploratory hits. By disabling lower tiers for these high-precision queries, we ensure that a shopper deep diving into specific results isn’t distracted by irrelevant semantic fallback as they paginate through the current result set.

Governance across tiers

A critical component of this architecture is that policies apply globally across all tiers. If a user has a "vegan" preference profile, the governed control plane injects that constraint into the strict, relaxed, and semantic queries. This ensures that even when the system uses semantic fallback to return "mandarins" for an orange search, the results remain compliant with the user's broader dietary preferences or business constraints.

The problem of facet instability

A challenge with multi-tier search is maintaining consistent faceted navigation (sidebar filters). If a search for "chocolate" yields 12 strict results, the sidebar filters might show "dark" and "milk". If a user selects "dark" and the result count drops, a naive system might trigger the semantic tier to fill the page, which could suddenly introduce "red wine" into the filters due to a semantic relationship.

The governed control plane identifies which tiers contributed to the initial search and locks the facets to those tiers. This prevents the sidebar from changing unexpectedly during a filtered session, ensuring a stable user experience.

The pagination challenge: Seamless multi-tier paging

Pagination in a tiered system requires precise state management. As established, the first page determines the scope of the current result set. If the first page required semantic results, the user can page through all available results from all three tiers. On the other hand, if the first page was satisfied by high-intent keyword matches, the semantic tier is not retrieved for that specific result set.

The governed control plane manages this through:

• Tier locking: The response includes an array identifying the contributing tiers. The front end returns this on subsequent requests to keep the tier composition consistent across all pages.

• Dynamic offset calculation: The back end calculates an offset based on the requested page and the total products returned in preceding tiers.Example: If the first page has returned seven strict matches and three relaxed matches, a request for page 2 (starting at index 10) would execute a relaxed tier query with an offset of three.

• ID exclusion for lower tiers: The system retrieves IDs from the higher tiers (which, by definition, will always be fewer than the page size threshold) and explicitly excludes them from lower-tier results using an ID-only query (which avoids the overhead of a full fetch phase for excluded items).

Summary

The multi-tier approach ensures search results are precise when data is available and helpful when it is not. By providing a governed fallback sequence for the application to execute, the architecture maintains high relevance while eliminating "no results" scenarios.

What's next in this series

The next posts in this series extend the governed control plane into new territory. Part 6 explores personalization (using purchase history boosting and cohort-aware policies), and Part 7 demonstrates per-query economic optimization. Stay tuned!

Put governed ecommerce search into practice

The search architecture described in this post, where retrieval tiers, economic weights, and governance constraints compose into a single request, was designed and built by Elastic Services Engineering as part of our repeatable ecommerce search accelerators.

To learn more about applying these patterns to your business, Contact Elastic Professional Services.

From architecture to implementation

Part 3 described the control plane architecture: reverse matching as a lookup primitive, policy documents that separate match from action, and cascading transformations that compose multiple policies into a single execution plan. This post goes hands-on with the Elasticsearch feature that powers the policy lookup: the percolator query.

The percolator is a natural fit for governance because it inverts the direction of search in exactly the way a control plane needs. This post walks through the implementation step by step, starting with a clear explanation of what the percolator does and why it matters, and then moving through index design, policy storage, query-time evaluation, and multi-policy composition.

How normal search works

In an ecommerce system, you may have hundreds of thousands or millions of product documents containing fields such as title, category, and price. When a user searches for matching documents, you're asking Elasticsearch to compare the user’s search string against one or more stored fields in these product documents. Elasticsearch's default analyzer, the standard analyzer, lowercases text and splits it into tokens. A search for “oranges” matches “Oranges” because of lowercasing. With a language-aware analyzer that includes stemming, it also matches “orange” because both forms reduce to the same stem. For example, the following match query returns documents that have “orange” or “oranges” in their “title” field.

So for the above query, Elasticsearch returns the product documents whose title field matches “oranges”, which could include results such as “Orange Fruit Spread”, “Orange Juice”, “Juicy oranges”, “Orange Marmalade”, and so on. The key point to remember is that Elasticsearch is commonly used to compare a search string against documents and to return the documents that match the search string.

The governance problem: Finding relevant policies before searching for products

As established in Parts 1 through 3, a governed search system does not send the user's search string directly to the product catalog. First, it checks whether any policies apply to that search string.

A merchandiser has decided that when someone searches for exactly "oranges", results should be restricted to the Oranges category, eliminating orange juice, orange marmalade, and orange soda. That business decision is stored as a policy. When a user types "oranges", the control plane needs to find that policy, read its instructions, and modify the search against the product catalog accordingly. In order to do this, the control plane needs to figure out which stored policies are relevant for this search string.

An enterprise deployment might have hundreds or thousands such policies. Checking them one by one with if/else logic is the application-layer anti-pattern described in Part 2. What we need is a way to store all of those policies in an index and instantly find the ones that match a given search string. This is where the percolator comes in.

Flipping the direction: The percolator

We previously mentioned that in a normal search, Elasticsearch is commonly used to compare a search string against documents and to return the documents that contain that search string.

The percolator inverts this. With a percolator, you have an index where each document stores a query pattern, and then an incoming search string is checked against these stored queries to determine which of these stored query patterns has triggered.

For governance, the "stored query patterns" are policies. Each policy contains a pattern that describes the kind of search string it should match. For example, does the search string exactly match “oranges”, or does the search string contain “olive oil”? The incoming string is the user's search text, which arrives at query time and needs to be checked against all stored policy patterns. This is covered in a related PRISM video at 4:09.

Step by step: How a search for "oranges" finds its policy

The policy

A merchandiser has authored a policy that matches if a user searches for exactly "oranges" without any other words. Once the percolator matches, the remainder of the document includes the rules that the control plane will use to build the Product query; in this example, one of the rules is to restrict (filter) results to the Fruits category.

The percolator field contains the pattern that defines when this policy should fire. In this case, it matches the phrase "START oranges END". The rule_type and rule_args fields define what the policy should do when it fires. The START and END tokens are boundary markers, which we will explain shortly.

You can see how a policy is authored in the PRISM Studio UI at 2:52 of the related PRISM video.

The user searches

A shopper types "oranges" into the search bar.

The control plane checks for matching policies

Before searching the product catalog, the control plane intercepts the user search string, wraps it in boundary markers, and sends it to the percolator:

The string "START oranges END" is checked against all stored policy patterns. Internally, Elasticsearch runs the stored policy patterns against this string and returns the ones that match. That's the percolator. The user's search string was checked against all stored policy patterns, and the ones that matched were returned. No if/else chains. No sequential evaluation. The index handles the matching.

The control plane applies the policy

The control plane reads the matched policies’ actions. The above policy instructs the control plane to restrict results to the Fruits category. The control plane builds the final Elasticsearch query against the product catalog as follows:

The user searched for "oranges”. The product catalog receives a query for "oranges" constrained to the Fruits category. Because of this constraint, orange juice, orange marmalade, and orange soda are excluded.

Why "orange marmalade" does NOT trigger the oranges policy

Suppose a different user searches for "orange marmalade”. The control plane wraps the string and percolates: "START orange marmalade END". The oranges policy's pattern is match_phrase: "START oranges END". The oranges policy does not match and therefore the policy isn’t applied, and the results aren’t constrained to the Fruits category.

This is the purpose of the START and END boundary markers. Without them, a policy that matches on the word "oranges" could accidentally fire on a query like "orange marmalade". By wrapping the user's search string with START and END and including those markers in the policy's pattern, we ensure that the policy only fires when "oranges" is the complete search string, without any other words. This matches both the shoppers and the merchandiser's intent.

A second policy: "olive oil" on the stemmed field

Not every policy needs an exact string match. The “olive oil” policy matches on a stemmed field, so it fires regardless of minor word-form variations:

This policy's pattern matches against query.stemmed instead of query. When the user's search string arrives, it’s stored in both a query field (the exact text) and a query.stemmed field (analyzed with a stemming analyzer that reduces words to their stems, so "olives" and "olive" both reduce to the same stem, as do "oils" and "oil"). The policy's pattern is checked against the stemmed version of the string, so it fires regardless of minor word-form variations.

The START and END boundary markers work on the stemmed field, as well, ensuring this policy only fires when "olive oil" is the entire search string, not when it appears as part of something longer.

The rest of this post covers the implementation details that make this production-ready: the index mapping that supports both matching modes, how highlights drive phrase removal and consumed phrase tracking, and how multiple conflicting policies compose into a single execution plan.

The policy index mapping

The policy index needs a percolator field to hold stored query patterns and a text field that mirrors the structure of the incoming search string the percolator will match against. The mapping below is simplified for clarity. A production deployment is more complex, using custom analyzers to handle boundary markers, variable pattern matching (for example, recognizing that "under $4" contains a currency value), and other kinds of analysis.

The index is named policies because each document represents a complete governed policy as defined in Part 2. This includes match criteria, action, priority, and metadata. The rule_type and rule_args fields contain the action component of the policy, which contain the instructions that the control plane will use to compose the query for execution against the product catalog.

The query field is the string that the percolator matches against. It has two variants: an exact version and a stemmed version. When the user's search string arrives, it’s placed into this field in the temporary in-memory index. Policies that match on query see the exact string; policies that match on query.stemmed see the stemmed version.

Percolating with highlights, filtering, and sorting

The simple examples above showed minimal percolation requests. In practice, the control plane adds highlighting, filters disabled policies, and sorts by priority:

The highlight configuration uses "query" as the field key with "query.stemmed" in matched_fields. This tells Elasticsearch's unified highlighter to return highlights on the parent query field but to also consider matches from the query.stemmed subfield when determining which tokens to highlight. This is what allows a policy that matches on the stemmed field to still produce accurate highlight spans on the original text, which the control plane needs for phrase removal and consumed phrase tracking.

The enabled: true filter ensures that disabled policies are skipped. The sort on priority ensures that higher-priority policies are returned first, so the control plane can process them in the correct order for cascading transformations. The highlight field is the most important addition; it tells us exactly which words in the user's search string triggered each match.

The response for an "olive oil" search may look as follows:

Why highlights matter

Notice the highlight in the response: "<em>START olive oil END</em>". Elasticsearch is telling us exactly which words in the user's search string caused the policy to match. This isn’t cosmetic. The highlight metadata drives two critical downstream behaviors:

Phrase removal. Some policies need to remove the matched text from the search string before constructing the product catalog query. For example, a policy that matches on "cheap" removes that word and converts it into a price filter instead. The highlight identifies exactly which span of the search string the policy matched, so the system knows what to remove.

Consumed phrase tracking. As described in Part 3, when multiple policies match the same search string, a higher-priority policy might remove words that a lower-priority policy also matched on. By comparing each policy's highlight against the current (evolving) search string, the system can detect that a phrase has been consumed and skip the lower-priority policy. This prevents double-processing and ensures deterministic behavior.

You can learn more about how highlighting works in this article.

From percolation to execution plan

The percolator returns a set of matching policies. But as Part 3 described, the lookup is only half the story. The other half is composing those matches into a coherent execution plan. Here’s what that looks like for a concrete query.

Worked example: "Cheap chocolate" during a Christmas campaign

Suppose the system has two active policies: the "Cheap chocolate" policy (priority 210) and the "Christmas chocolates" policy (priority 300), both described in detail in Part 3.

Step 1: Percolate. The user searches for "cheap chocolate." The control plane wraps the search string as "START cheap chocolate END" and sends it to the percolator. Two policies match: The "Cheap chocolate" policy's pattern matches on the phrase "cheap chocolate"; and the "Christmas chocolates" policy's pattern matches on "chocolate" via the stemmed field.

Step 2: Sort by priority. The percolator returns both policies, sorted by priority in descending order. The “Christmas chocolates” policy (300) is processed first, followed by the “Cheap chocolate” policy (210).

Step 3: Apply the cascading transformation. This is the initial state → [Policy A] → state' → [Policy B] → state'' → execution plan model from Part 3.

The “Christmas chocolates” policy (priority 300) applies first:

• Adds a category hard filter: "Christmas foods and drinks," "Christmas sweets".

• Adds a price filter: less than $7.

• Adds a category soft boost: "Advent calendars" (3x).

The “Cheap chocolate” policy (priority 210) applies next against the modified state:

• Attempts to add a category hard filter: "Chocolates," "Milk chocolates"; but the Christmas policy already set this field with on_conflict: override, so the Cheap chocolate categories are dropped.

• Attempts to add a price filter: $2, the Christmas policy set on_conflict: restrict for price, and $2 is more restrictive than $7, so $2 wins.

• Removes "cheap" from the search string.

Step 4: Build the Elasticsearch query. The control plane assembles the execution plan into a single Elasticsearch query against the product catalog:

The original search string was "cheap chocolate”. The query that reaches the product catalog is a governed, intent-aware retrieval plan: The word "cheap" has been consumed and converted into a price constraint, results are restricted to Christmas seasonal categories, Advent calendar products receive a ranking boost, and the price ceiling reflects the more restrictive value from the lower-priority policy. Every transformation is deterministic, traceable, and explainable.

For a quick overview about how these multipliers interact with the base BM25 score, see 8:45 in the related PRISM video, where we briefly discuss multiplicative boosts.

Why this scales

The percolator is efficient for this use case because of the asymmetry: An enterprise ecommerce system might have millions of products but only hundreds or thousands of governance policies. The percolator is checking one incoming search string against that set of stored policy patterns, not scanning the full product catalog. The cost is proportional to the number of policies, and Elasticsearch applies internal optimizations (indexing terms from stored query patterns, short-circuiting Boolean logic) to keep matching fast.

Adding a new policy is just indexing a new document. Disabling one is a field update. No code changes, no deploys, no restarts.

From lookup to governed retrieval

The percolator provides the fast reverse-matching primitive that makes the control plane architecture from Part 3 practical at scale. Policies are data which are stored and indexed, and efficiently matched against incoming search strings. The control plane composes matching policies into a governed execution plan through the cascading transformation and per-field conflict resolution described in Part 3. And the retrieval engine executes the governed execution plan against the product catalog.

The result is a system where a merchandiser can author a new policy without touching application code, test it against representative queries, promote it to production, and immediately see the effect. The percolator makes the policy lookup fast; the control plane makes the policy composition deterministic; and the governed workflow makes the whole process safe.

What's next in this series

The next post in this series extends the governed control plane into new territory. It introduces a multi-tier search architecture, explaining how to orchestrate strict, relaxed, and semantic retrieval while maintaining stable pagination and facets.

Put governed ecommerce search into practice

The percolator-based control plane described in this post, from index mappings and boundary markers to highlight-driven phrase tracking and cascading policy composition, was built by Elastic Services Engineering as part of our repeatable ecommerce search accelerators. Every query example and policy structure shown here comes from a working system validated against enterprise-scale product catalogs.

If you want to implement a governed, policy-driven control plane on Elasticsearch, Elastic Services can get you there faster. Contact Elastic Professional Services.

Join the discussion

Have questions about search governance, retrieval strategies, or ecommerce search architecture? Join the broader Elastic community conversation.

Why query interpretation is often a challenge

Storing policies as code (if/else blocks in the application layer) produces tens of thousands of lines of brittle logic that lacks any indexing for efficient policy retrieval at query time. Iteration is slow (a single query behavior change may require a six-week deployment cycle), accountability is unclear (why did results change?), and business users cannot modify search behavior without engineering involvement. This is shown on the left side in the following image:

Storing policies as data in an Elasticsearch index is shown on the right side of the above image. This approach solves all of the issues associated with hard-coded query resolution logic. However, for this to work, you need a way to quickly determine which policies match the user query and how conflicts should be resolved. This is where the governed control plane comes in.

The control plane pattern

A governed control plane sits between the raw user query and an Elasticsearch retrieval. It receives user text as its input, and its output is an execution plan that includes filters, boosts, and retrieval routing decisions.

A control plane pipeline consists of:

1. User query: A user enters a string of what they’re looking for, such as “oranges” or “gift for grandpa”.

2. Policy lookup: Match the user query against the policy index.

3. Return matching policies: Policies that match the user query are returned from the policy index.

4. Policy application: The control plane analyzes these returned policies and composes matched policies into a single coherent execution plan that includes filters, boosts, overrides, and guardrails and that applies the appropriate retrieval method (for example, lexical versus semantic versus hybrid).

5. Execute: The modified intent-aware Elasticsearch query is passed to the application to be executed against a product catalog index.

6. Explain (optional): In addition to creating a query that provides business and intent-aligned results, the control plane provides an optional explainability payload to show which policies were triggered and how they were combined.

Finding which policies should be applied for a user’s search string requires a fast reverse-matching primitive, which we solve with the percolator query. After retrieving relevant policies, combining multiple matched policies into a unified execution plan requires a judgment framework: priorities, conflict strategies, consumed phrase tracking, and cascading transformations that apply policies in sequence rather than independently. Additionally, the most appropriate retrieval technology needs to be selected (for example, BM25 for “oranges” versus semantic search for “gift for grandpa”).

Policy lookup: Checking the query before searching for products

When a shopper types a query, a search system with a governed control plane doesn’t send that query directly to be executed against the product catalog. First, the query is checked against a set of stored policies and modified to reflect the intent of the query and business priorities.

Policy structure

Each policy is a simple document that defines two things:

• Match criteria: What query text should cause this policy to fire. This could be an exact phrase, a single word, a pattern, or a combination.

• Action: What to do when the policy fires. This could be applying a category filter, excluding products, extracting a price constraint, or changing the retrieval strategy.

The system finds all matching policies, composes them into an execution plan, and only then runs the product search. Taken together, policies act like a knowledgeable store associate who understands what you’re looking for and walks you to the right aisle.

The policy pattern

The first articles in this series introduced examples of policies in action: constraining "oranges" to the produce category, treating "without peanuts" as an exclusion, and routing "gift for grandpa" to semantic retrieval. The key architectural point is that in each case, the query is checked against stored policies before the product search begins. The policies determine what constraints to apply, which text to modify, and which retrieval strategy to use. The query against the product catalog comes after the policies have been applied and a new rewritten query has been created.

Why this is fast

An enterprise ecommerce system might have millions of products but only hundreds or thousands of policies. The policy lookup step is searching against a small curated index, not the full product catalog, and is therefore fast. And because policies are stored as data in their own index, a merchandiser adding a new policy doesn't touch the application code, and an engineer optimizing the product search doesn't touch the policy index. The two concerns evolve independently.

The examples above describe what happens conceptually. Under the hood, the policy lookup is implemented using the Elasticsearch percolator query type, which is purpose-built for this kind of pattern: matching incoming text against a set of stored queries. Part 4 in this series provides a hands-on deep dive into the percolator implementation, including index mappings, boundary markers, and highlight-driven phrase tracking. With the lookup mechanism covered in depth in Part 4, let's turn to what a policy document actually contains and how the control plane composes multiple policies into a single execution plan.

Example policies

Now that we've seen what policies do conceptually, let's look at what they actually contain. The two policies below have been designed to intentionally conflict, which will demonstrate the conflict resolution system described in subsequent sections.

Cheap chocolate

The policy shown below detects if a user has submitted a search containing the phrase “cheap chocolate”. If so, results are restricted to the “Chocolates” and “Milk chocolates” categories. This policy also applies a price filter of $2. Also, notice that this policy has a priority of 210; we’ll come back to this when we discuss conflict resolution in more detail.

The filter mode and conflict strategy settings shown here (hard_filter, soft_boost, restrict, override) are explained in detail in the conflict resolution section below.

When the above policy is activated, a search for “cheap chocolate” respects the price filter of $2 and restricts results to the “Chocolates” and “Milk chocolates” categories. Example results are shown below:

Christmas chocolate

The policy shown below is an example of a policy that one could imagine applying at Christmas. This example restricts results to “Christmas foods and drinks” and “Christmas sweets”, boosts any products that are also in the “Advent calendars” category, and applies a price filter of less than $7 to focus on affordable seasonal items. Additionally, notice that this policy has a priority of 300. We’ll come back to this when we discuss conflict resolution in more detail.

When the above policy is activated without any conflicting policies, a search for “chocolate” respects the price filter of $7, and restricts results to the “Christmas food and drinks” and “Christmas sweets” categories, and boosts any products tagged as “Advent calendars”. Example results are shown below:

Combining matched policies

The policy lookup described above is half the story. The other half is what happens when multiple policies match the same query.

In any nontrivial deployment, a single query will routinely trigger several policies at once. "Cheap chocolate" will match both of the policies that we demonstrated above. Each policy is correct in isolation. The challenge is composing them into a single, coherent execution plan without contradictions, without double-counting, and without one policy silently undoing the work of another.

This isn’t a lookup problem; it’s a judgment problem. The system must decide:

• Order of application: If a negation policy removes "without peanuts" from the query, does the price policy still see the original text or the modified text?

• Filter conflicts: If two policies set different price ceilings, which one wins? Is the loser silently dropped, or does it degrade gracefully into a soft boost?

• Phrase ownership: If two policies both matched on the same word and the first one already consumed it, should the second one still fire?

A naive implementation (apply all matched policies independently, merge the results) breaks as soon as policies interact. The architecture needs an explicit model for how policies compose. The next two sections describe that model: a priority and conflict resolution framework; and a cascading transformation model that makes policy interaction deterministic.

The key insight is that policy application isn’t a set of independent operations; it’s a cascading transformation. Each policy receives the rewrite state produced by all higher-priority policies and transforms it further:

initial state → [Policy A] → state' → [Policy B] → state'' → ... → execution plan

The state carries the rewritten query text, accumulated filters, current intent, and any synonym expansions. A high-priority policy can remove text from the query, and every subsequent policy sees the modified query, not the original. Context accumulates. Order matters.

Precedence and conflict resolution: Determinism matters

The specific conflict strategies are a design choice. Different organizations may resolve conflicts differently, depending on their business requirements. The following approach illustrates the kind of judgment framework a control plane needs. The important thing is not these specific strategies but that the system has explicit, deterministic strategies rather than letting conflicts resolve through unpredictable interactions.

Priority ordering

Policies are sorted by priority (highest first). When multiple policies match the same query, they’re applied in priority order. If two policies try to set the same filter field, the higher-priority policy's declared strategy for that field takes precedence. If there are multiple policies triggered that have the same priority, then the policy with the highest ID is given precedence (as if it were assigned a higher priority); this choice ensures deterministic behavior when conflicts arise.

Per-field resolution, not per policy

A critical design principle: Conflict resolution operates per field (for example, brand, category, or description), not per policy. When two policies produce filters that overlap on specific fields, only those specific fields are affected by the conflict resolution strategy, and the resolution strategy is defined by the highest-priority matching policy. Non-conflicting fields from both policies survive intact.

This matters because the alternative of a per-policy approach would force the system to either accept or reject an entire policy when only one of its fields conflicts.

Per-field resolution preserves the maximum amount of useful constraint information.

Three settings per filter field

Each filter field in a policy has three independent settings:

Filter mode: How the filter is applied when there’s no conflict.

• hard_filter (default): Applied as an Elasticsearch bool.filter clause. This is useful for excluding unrelated products entirely. For example, restricting a search for "oranges" to the produce category eliminates hits such as orange juice and orange marmalade. Non-matching documents are completely excluded from results.

• soft_boost: Applied as an Elasticsearch function_score weight with a configurable boost_weight. Documents that match get a ranking boost, but non-matching documents aren’t excluded. This is useful for something like boosting a brand, without excluding other brands.

Conflict strategy

What happens when a lower-priority policy sets the same field:

• override: This high-priority policy's value wins; the lower-priority value is dropped entirely. Valid for all field types.

• restrict: Take the more restrictive numeric value (for example, the lower ceiling for price__max, the higher floor for price__min). Valid for numeric range fields only.

• merge: Combine both values into a union. Valid for non-numeric fields only.

• soft_boost: Convert the conflicting filter to a function_score weight with a configurable boost_weight instead of a hard filter. For more details on function_score boosting, see Influencing BM25 ranking with multiplicative boosting in Elasticsearch. This is only valid for non-negation fields.

Value: The actual filter value (for example, a categories list, a price threshold).

Strategies by field type: Not all strategies make sense for all field types. For instance, an exclusion is inherently binary, so it cannot be soft-boosted. The following table shows which strategies are available for each field type:

Field type

Available strategies

Default

Negation fields (__not, __match__not)

override, merge

override

Numeric range fields (__max, __min, __gt, __lt)

restrict, override, soft_boost

restrict

All other fields (keyword, text)

soft_boost, override, merge

soft_boost

Negation fields cannot be soft-boosted because exclusions are binary. Converting "never show canned foods" to "slightly prefer not-canned-foods" fundamentally changes the semantics; a product from "canned foods" would still appear, just ranked slightly lower, which defeats the purpose of the exclusion.

A concrete example: Searching for "cheap chocolate" during a Christmas campaign

Suppose a merchandiser has created the two policies for chocolate that we previously demonstrated, a lower priority one for cheap chocolate and another higher-priority chocolate-related policy that will be enabled during Christmas. If both of these policies are enabled, then how these are combined depends on the filter mode and conflict strategy of the higher-precedence policy. If both of the previously discussed policies are enabled, they’ll be combined as follows:

This shows two conflicts, one on categories and one on price. It’s worth noting that the query that will be executed after this transformation has the following characteristics:

• Only products from the “Christmas foods and drinks” and “Christmas sweets” categories will be shown.

• Within those categories, if the products are also tagged as being in the “Advent calendars” category, they’ll be boosted up by 3x.

• A price filter for $2 is applied, which came from the lower-priority policy (because the higher-priority policy specified to “Restrict” on conflict).

• The word “cheap” is removed, only returning products matching “chocolate”.

With both of these policies enabled, “cheap chocolate” returns results similar to the image shown below:

Relaxing constraints

Perhaps the retailer doesn’t want to exclude products in the categories of “Chocolates” and “Milk chocolates” during Christmas. The settings on the Christmas policy might have overreached and inadvertently removed categories applied by the “cheap chocolate” policy. This is an example that shows why it might be more desirable to combine lower-priority policies with conflicting higher-priority policies. For example, we could modify the Christmas chocolates promotion so that instead of “Override” on conflict, we do a soft boost. The change to that policy would be as follows:

After this modification, the query rewriter transformation pipeline execution for “cheap chocolate” looks as follows:

With the soft boost on conflict, the conflicting filters are converted into soft boosts rather than being dropped. The query that will be executed on the product catalog after this transformation has the following characteristics:

• Because “On conflict” is specified as “Soft boost” on the higher-priority policy, the conflicts will be converted to boosts as follows:

  • Products from the “Christmas foods and drinks” and “Christmas sweets” categories will have a boost of 1x applied to them.

  • Products from the “Chocolates” and “Milk chocolates” categories will have a boost of 3x applied to them.

• As in the previous example, if the products are also tagged as being in the “Advent calendars” category, they’ll be boosted up by 3x.

• As in the previous example, a price filter for $2 is applied.

• The word “cheap” is removed, only returning products matching “chocolate”.

With relaxed filtering, results look as follows:

Overriding price from a high-priority policy

Or perhaps the retailer wants to allow slightly more expensive chocolates to be shown during Christmas by increasing the price max to $7. To ensure that the max price from the Christmas chocolates policy is not overridden if someone searches for “cheap chocolates”, we can set the conflict mode on the price to “override” rather than “restrict”, as follows:

With this override, the query for “cheap chocolate” ignores maximum price that is defined in the “cheap chocolate policy” and only applies the price specified in the “Christmas chocolates” policy, as follows:

This is similar to the previous example, with the difference being that the max price is set to the $7 value from the higher-priority policy because that policy specified “Override” on conflict. With the Christmas price filter taking precedence, the results look as follows:

These three variations (override, soft_boost, and override on price) demonstrate a key property of the system: A merchandiser can change how two policies interact by modifying a setting on a single field within a single policy, without deploying any code. The conflict strategy is the lever that controls business behavior.

Consumed phrase tracking

There’s a subtler form of conflict: two policies that match on the same phrase. If a higher-priority policy removes "without peanuts" from the query, a lower-priority policy that also matched on "without" has nothing left to act on. The system detects if the matched phrase is no longer present in the rewritten query and skips the lower-priority policy.

Intent policies are exempt from consumed phrase tracking: They set the retrieval strategy based on the original query match, regardless of what text has been removed by higher-priority policies.

Priority ordering, per-field conflict resolution, and consumed phrase tracking together give the control plane a deterministic composition model. With that foundation in place, the system can make a routing decision that would be risky without it.

Governance makes retrieval strategy safe

An important insight about routing to the correct retrieval method (text, semantic, or hybrid) is that it executes after governance. If your policies have already enforced "produce category”, then semantic retrieval becomes far less risky because the candidate set is constrained. A semantic search over 500 product items is a very different proposition from a semantic search over 500,000 SKUs. Governance narrows the blast radius before retrieval begins.

For example, without governance, a semantic query for “Fruit high in vitamin C under $4”, in addition to fruits, might return vitamin bottles, carrots, and green pepper. The control plane ensures that these undesired results aren’t even considered as part of the semantic expansion.

With that constraint in place, the control plane applies pragmatic routing logic:

• Lexical for navigational and head queries where deterministic precision matters.

• Semantic for descriptive discovery queries where concept matching helps.

• Hybrid selectively, when constraints have already been enforced and the business accepts broader recall.

From architecture to implementation

The governed control plane translates business intent into deterministic, composable execution plans, without embedding that logic in application code. Policies are data: matched at query time, resolved through explicit per-field conflict strategies, and applied as cascading transformations that produce explainable results. Elastic Services Engineering has built and deployed this architecture for enterprise ecommerce teams, using repeatable patterns and accelerators that compress the path from concept to production. You can see a demo of our implementation of a control plane on YouTube at: Fixing Search Relevance in Seconds: Introducing PRISM.

What's next in this series

The next post goes hands-on with the implementation: how the Elasticsearch percolator powers the policy lookup, including index mappings, boundary markers, highlight-driven phrase tracking, and concrete query examples.

Put governed ecommerce search into practice

The control plane architecture described in this post (per-field conflict resolution, cascading policy transformations, and governance-constrained retrieval routing) was designed and built by Elastic Services Engineering. Every pattern, screenshot, and transformation pipeline shown in this series comes from a working system built by Elastic Services Engineering and validated against enterprise-scale product catalogs.

If you want to implement a governed, policy-driven control plane on Elasticsearch, Elastic Services can get you there faster.

Join the discussion

Have questions about search governance, retrieval strategies, or ecommerce search architecture? Join the broader Elastic community conversation.

This blog's content was developed and verified using Elastic versions 9.2.8 and 8.19.14, along with Filestream Integration versions 2.3.0 and 1.2.0.

Important note: Depending on your environment, some steps may require specific modifications. Furthermore, be aware that dynamic templates were removed from the @package component template starting with Filestream Integration version 2.3.3.

Before starting the reindexing process, it’s important to consider the current storage allocation in your environment. The steps outlined below involve creating a copy of the existing backing index, which will temporarily reside in the hot tier.

Elasticsearch data tiers

• Hot: The hot tier is the Elasticsearch entry point for time series data, storing the most recent, frequently searched data. Hot tier nodes require fast reads and writes, necessitating more resources and faster storage (SSDs). This tier is mandatory, and new data stream indices are automatically allocated here.

• Warm: Time series data can move to the warm tier once it’s being queried less frequently than the recently indexed data in the hot tier. The warm tier typically holds data from recent weeks. Updates are still allowed but are likely infrequent. Nodes in the warm tier generally don’t need to be as fast as those in the hot tier. For resiliency, indices in the warm tier should be configured to use one or more replicas.

• Cold: Data that’s infrequently searched can move from the warm to the cold tier. The cold tier, while still searchable, prioritizes lower storage costs over search speed. Alternatively, the cold tier can store regular indices with replicas instead of searchable snapshots, allowing use of less expensive hardware for older data without reducing disk space requirements compared to the warm tier.

• Frozen: Data that’s queried infrequently or no longer queried moves from the cold to the frozen tier for its remaining lifecycle. This tier uses a snapshot repository and partially mounted indices to store and load data, reducing local storage and costs while still allowing search. Searches on the frozen tier are generally slower than on the cold tier because Elasticsearch may need to fetch frozen data from the snapshot repository. We recommend dedicated frozen tier nodes.

Prerequisites: Determine which fields have conflicts

To determine which fields have mapping conflicts, navigate to Stack Management -> Data Views -> logs-* (using the logs-* data view is the highest hierarchy of data present with the logs- prefix.) If there are any conflicts, there will be a yellow box stating that. You may either click View conflicts or, under the Field type box next to the Search box, select conflict.

Clicking the yellow Conflict button will reveal which indices are associated with which mapping types.

This situation (where the field is mapped as both a keyword and a long) typically occurs because data was ingested before a specific mapping type was defined in the component template for the relevant data stream. In such cases, Elasticsearch attempts to set the mapping based on its dynamic templates.

In order to determine which mapping is appropriate for the field, and if the field is an ECS field, verification with ECS field reference is needed. If the field in question is not an ECS field, its value must be reviewed to determine the correct mapping.

If a field, such as log.offset in this example, isn’t documented in the ECS, the next steps are to investigate the field's value, determine which conflicting mapping type has the most backing indices, and examine the component templates of the other indices.

Typically, the mapping type associated with the highest number of indices is the correct one, but we recommend you verify the value of the field in question to validate this. To confirm the validity of a mapping type (for example, long), you must also verify that the field's value is appropriate for that type. This verification can be done by using Discover to search for the field in question. Reviewing other data streams that contain the same field can provide additional confirmation also.

To review the values present for the field with the mapping issue, navigate back to the yellow Conflict button stated earlier, click the Conflict button, highlight one of the backing indices, and paste into a Discover session. Your Kibana Query Language (KQL) statement should look like the following screenshot, to include the _index: field delimiter.

Prepare the new backing index custom component template

To address the mapping conflict in the data stream, first examine the relevant @package component template. You can find this under Stack Management -> Index Management -> Component Template. Search for the data stream and select the corresponding @package link. This template contains mappings for the fields out of the box and, while it isn’t common to have a mapping mismatch, it’s possible for the more appropriate type to be overlooked.

Review the template to confirm it contains the necessary field nesting and mapping for the field in question. For example, if the template incorrectly lists log.offset as a keyword, this is the source of the issue.

Important: Because modifying @package/managed templates isn’t recommended, you must use or create an @custom component template to correct the mapping type (for example, for log.offset) for all future data.

• We don’t recommend modifying the @package/managed templates, since when you update the integration to a more recent version, any changes you make to the @package template will be overwritten. This is why we recommend using the @custom templates.

• If a data stream is experiencing mapping conflicts, you need to add any missing field (ECS and non-ECS) nestings or mappings to the data stream's @custom component template. Create this template if it doesn't exist yet, and make sure to specify the correct mapping type for the field.

• If you have multiple conflicts in your data view, apply all the necessary missing mappings for the data stream simultaneously so that the reindex is performed once versus multiple times. Having entries for proper data typing in the @custom component template will ensure any future data ingestion will follow the same mapping guideline.

To create the @custom component template (or verify it’s in use and populated), navigate to Index Templates, type in the name of the data stream in question, and click the appropriate @custom template being used by the data stream. If the template is not yet created, a yellow box will appear, allowing you to create the template through the UI.

The screenshot below shows the next page once Create component template is selected. Leave the defaults as is on the first page and click Mappings or Next until you reach the Mappings page.

To explicitly set the mapping for a new field coming in or to update a field that has a mapping conflict, when the data stream rolls over due to configuration set in the index lifecycle policy, an entry is needed for the field that the conflict exists in.

The below will set the mapping for the log.offset field in the @custom component template for the filestream data stream. Repeat the steps to add any custom fields or update necessary fields from the @package with the appropriate mappings, if needed, for this dataset. In this example, when setting offset to Long, the field type will be Numeric and the Numeric type will be Long. Click Add field and then outside of the area to continue.

Once all needed fields have been added, click through to review, and select Create component template when ready. All new data being ingested from this step forward will have log.offset set to long.

Creating the new backing index structure

The new backing index needs to have the existing mappings from the data stream’s component template, as well as the ECS ecs@mappings component template. The ecs@mappings component template is applied after the data stream’s component as a catchall for additional mappings that potentially weren’t captured in the previous component templates.

Navigate to the browser tab for the data stream's @package mappings. (Go to Stack Management -> Index Management -> Component Template -> logs-filestream.generic@package -> Manage -> Edit.) Once there, click on the Review section, then Request, and finally the Copy button on the right. The JSON contents of the component template copied will ensure the remaining field mappings and settings are retained while we update the log.offset field mapping. The JSON will form the backing structure for the newly reindexed backing index.

Important: If the template’s JSON was not copied and work was continued on with the reindex, the log.offset conflict would be resolved but there would be new conflicts with the integration, as the integrity of the current mappings were not upheld, creating double work to resolve the original issue.

Open a second browser tab, navigate to Dev Tools, and paste the copied content. Now, to clean up what was pasted:

Modifications to the request

1. Index name: Replace _component_template/logs-filestream.generic@package with the name of the backing index you intend to reindex, appending -1 to the end. For example, use PUT <backing index to reindex>-1.

• The appended -1 signifies a reindex and won’t conflict with the default ILM rollover settings, which are based on the index's creation date.

2. Settings: Remove the line "template" (line 3), as well as the very last closing brace for the entire JSON payload; Line 3 should start with "settings": {.

• Replace the inner contents of the settings section with "index.codec": "best_compression". This action will apply Elastic's best compression to the index upon creation.

• Add in "index.lifecycle.name": "logs", as well as a line for "index.lifecycle.rollover_alias": "".

  1. The "index.lifecycle.name": "logs" entry will apply the logs ILM policy to the new backing index. Modify the ILM policy name if you aren’t using logs.

  2. The "index.lifecycle.rollover_alias": "" is blank, since this backing index won’t be rolled over, yet the setting is required to avoid ILM rollover errors into the next ILM phase after hot.

3. Structure: The request should now include both a Settings section and a Mappings section. Inside "mappings": {, you should find "dynamic_templates" and a "properties" section containing hard-coded fields and their mappings.

4. Dynamic templates modification: The current dynamic templates section contains entries for fields that may be overwritten when the ecs@mappings dynamic templates are added next, causing redundancy and extra lines that aren’t needed.

• Remove all sections in "dynamic_templates" except for the second section titled "_embedded_ecs-data_stream_to_constant": {.

• Repeat the same process as described above, gathering the dynamic mappings for the @package component template, but this time the dynamic mappings for ecs@mappings component template.

  • It may be easier to copy the entire contents of the mappings from the UI for the ecs@mappings component template, paste into the working Dev Tools dynamic_templates section, and remove duplicate and unnecessary lines where appropriate. Include these dynamic template setting contents after the"_embedded_ecs-data_stream_to_constant": { entry. The dynamic_templates section should look very similar to the below sample contents in Dev Tools.

• If dynamic_templates are not included/removed altogether, other fields (review the screenshot below) will have double mappings: text and keyword versus the appropriate mappings, if the dynamic_templates section was left included. What’s left should be the "properties" section under "mappings". This will also create issues in the data view by having the fields be double mapped (if not already mapped this way) and will cause additional mapping conflicts.

5. Metadata removal: Delete the last section labeled "_meta", as well as the section labeled "version", if present.

6. Formatting: Auto-indent the remaining sections, and adjust or remove any unnecessary curly braces that would prevent a successful execution.

7. Mapping change: Navigate to the "properties" section, find "log", and then locate "offset" nested underneath. Change the type from keyword to long, and remove the line entry (comma included) labeled "ignore_above": 1024,. If more than one entry was added to the @custom component template created earlier, include them here.

Your Dev Tools console view should now be similar to the example provided below.

After your console resembles the example (with any additional custom fields included and custom values specific to your environment), execute the command to create the shell of the new backing index, pausing to resolve any errors that arise.

Begin reindex process

With the shell of the new backing index successfully created, the next step is to reindex and resolve the mapping conflicts.

Important: If the backing index that has the mapping conflict is the most recent index and is the current write index (for example, the ending number for the backing index is -000001), the data stream needs to be rolled over. Rolling over the data stream is needed since the current write index, which is having documents fed into it, is a live backing index and cannot be modified.

With the correct field mapping now applied to the newer write index via the previously created @custom component template, all new documents will reflect this change.

This is performed by executing the following:

For example:

Reindexing involves copying the data from an existing backing index to a new one within the same naming convention, typically to apply necessary changes. These modifications could include updates to a component template or the addition of a new ingest pipeline for the data to be processed through.

Next, the data will be copied from the backing index that has the incorrect mappings into a new backing index. The original backing index has been rolled over, meaning no new documents can be added to it. The new backing index will follow the same naming convention, which preserves data visibility and integrity while applying the correct ILM policy, but will include a -1 suffix to indicate that it has been reindexed.

Adjust the index names as needed and paste the following code into the console. By including wait_for_completion=false, you can track the progress of document copying, which helps estimate the remaining reindexing time. Without this setting, you cannot track the status using the GET _tasks command below and will only be able to check the document count in the newer backing index using GET <backing index name>-1/_count.

Important: If issues arise during the reindex process, don’t rerun the reindex command; doing so will restart the process and create duplicate records in the index ending with -1. If a restart is necessary, first delete the index with the trailing -1, and then execute the preceding PUT command to recreate the new backing index shell.

Upon execution, the response will include a task ID. You can monitor the reindex progress using this ID with the command: GET _tasks/<task ID>.

The duration of the reindex depends on the volume of data in the original index. The completion can be tracked by looking for "completed": true when executing the GET command, which should yield a similar output.

GET _tasks/<task ID>

With the reindexing process now finished for the document count, the next step is to verify that the mappings for the new backing index and the specific field in question are correct.

For example:

You can verify that the mapping for log.offset is as shown below. To confirm that other fields have only a single mapping entry (not both text and keyword), compare them to a field that was not part of the dynamic template section in the preceding PUT command.

If the backing index that’s being reindexed has a large number of documents, it’s helpful to check the status of those documents being copied to the new backing index; this can be done by the following two Dev Tools commands to compare the counts.

GET .ds-logs-filestream.generic-default-2026.04.14-000001/_count

GET .ds-logs-filestream.generic-default-2026.04.14-000001-1/_count

Once the counts are verified to match and the correct mappings are present, update the data stream to include the new backing index, preventing an orphaned backing index in index management, where the ILM policy will never occur on the backing index.

• The return should be an acknowledgment of true, if successful.

Verify the new backing index is added with the following command, making sure the ilm_policy is correct:

Check the ILM status of the backing index next with the following command:

• It’s normal to see that the index is in hot, as it was created very recently (review line 8 or 10).

Execute the following to transition the backing index from the hot tier to the next appropriate tier that’s after the hot phase for the ILM policy for this data stream. The specific values for phase, action, and name in the current_step below can be referenced from lines 11, 13, and 15, respectively, in the provided screenshot above.

The next_step value indicates the subsequent ILM phase or data tier to which the index will transition to.

For example:

• It isn’t necessary, but as a safety measure, you may execute the _ilm/explain command again to ensure the backing index has moved to the next phase and is no longer in hot.

Once the following conditions are met, you can safely delete the original backing index that had mapping conflicts:

1. A new backing index has been successfully created.

2. Documents have been moved to the new index, and the document counts match.

3. Mappings have been corrected (both data stream specific and ECS).

4. The data stream incorporates the new backing index.

5. The ILM policy has been applied and has moved the index out of the hot phase.

Important: Alternatively, before deleting the original index, you can check the Data Views page. Select logs-* and verify that the reindexed backing index (which ends in -1) now appears in the long section. The original backing index should still be present under keyword. If the reindexed backing index is not in the long section, go back and review the preceding steps and make any necessary corrections.

For example:

After resolving the conflicts, return to the Data Views page and select logs-*. If the conflict was solely related to log.offset, you should no longer see any conflicts listed. If there were other conflicts, the original backing index should no longer appear in the conflict list; instead, the new backing index should now be listed in the long section.

You can also verify in Discover that the log.offset field now displays the appropriate icons.

Continue this process, repeating the above steps for every backing index that has a mapping conflict until all are successfully resolved.

References:

• ECS field reference

• Reindex documents

Final thoughts

By following the steps in this blog, you will resolve mapping conflicts and ensure that all new data is correctly mapped. This is achieved by linking the necessary component templates to your data source. This workflow not only fixes the immediate issues but also establishes a secure and repeatable process for managing schema changes as your data and requirements evolve.

Lexical retrieval (text matching), semantic retrieval (matching concepts), and hybrid retrieval (combining lexical and semantic signals) don’t solve these issues on their own. Lexical retrieval may return anything containing the word “oranges”, while pure semantic retrieval on a high-intent query like “oranges” may broaden toward related items, such as lemons or grapefruits. Hybrid retrieval blends these lexical and semantic signals, but it still doesn’t determine if this query should be treated as navigational, which constraints should be enforced, or which business policies should apply. The gap isn’t the retrieval technology itself; it’s the absence of a governance layer that understands what kind of query this is and what constraints should be enforced before retrieval begins.

In this blog, we explore ecommerce search governance, why it matters, and how a control layer ensures predictable, accurate retrieval.

What governance means in ecommerce search

Governance, in this context, means introducing a decision layer between the user's query and the retrieval engine. This layer performs the following functions:

• Classifies query intent: Is this navigation ("oranges") or discovery ("gift for grandpa")?

• Applies business constraints: What category boundaries, eligibility rules, availability constraints, or merchandising policies apply?

• Routes to the appropriate strategy: Should this use lexical retrieval, semantic retrieval, or hybrid?

A governance layer determines which retrieval approach should be used for each query, which constraints must be enforced, and which business policies should apply before retrieval begins. It’s important not to conflate governance with hybrid retrieval: hybrid is one retrieval strategy that combines lexical and semantic signals, while governance is the upstream decision layer that determines whether lexical, semantic, or hybrid should be used.

The status quo: The application layer "spaghetti" implementation

Currently, many retailers attempt to solve this by adding logic directly into the application layer. This often results in spaghetti code, that is, thousands of lines of hard-coded if-then statements, regex, and complex search templates.

This approach can provide desired search results as shown above; however, it creates significant operational friction:

• Engineering dependency: Business users and merchandisers cannot modify search behavior without engineering tickets and long deployment cycles that often span several weeks.

• Fragmentation: Search logic becomes scattered between application code and search templates, and is difficult to explain or audit, making it risky to evolve.

Even when teams recognize the need for routing, the debate often focuses on the wrong question: which retrieval method to pick.

The false choice: Lexical vs. semantic vs. hybrid

Search teams often frame the challenge as a retrieval strategy choice: lexical/BM25 versus semantic/vectors versus hybrid. That framing is understandable (retrieval methods matter), but it misses the most common failure mode in real deployments, which is that using a single retrieval approach for all queries will give suboptimal results.

Commerce search is a mix of fundamentally different intents:

• Deterministic, high-intent navigation ( "oranges", “milk”, “chocolate without peanuts”, “cheap olive oil”).

• Exploratory discovery ("jacket for hiking in the mountains", "gift for a 12-year-old who likes robotics").

• Operational constraints (availability, size, price, color).

• Merchandising and campaigns (boost, bury, seasonal campaigns).

When the system routes all of these through the same retrieval strategy, the results are often systematically wrong in predictable ways because the operating model lacks governance. When teams don't recognize this as a governance gap, they respond with the only lever they have: more tuning.

Why "relevance tuning" can become cyclical

Without a routing layer, “relevance” often turns into a never-ending backlog:

• Why is this query showing accessories above the core product?

• Why did this head query suddenly start surfacing related items?

• Why did results change after we added synonyms, adjusted analyzers, or enabled hybrid?

• Why does the business team need an engineering release to fix a single query?

Teams respond with more tuning: more synonyms, more boosts, more reranking experiments, more exceptions in application code. This can work for a while, but it often produces brittle behavior because the system still lacks an explicit decision layer for determining query type and enforcing the right constraints before retrieval.

The anatomy of ecommerce intent: Head and tail

In this section, we use “head” and “tail” as practical shorthand for common navigational and exploratory query patterns in ecommerce. In the real world, many queries contain aspects of both:

Head queries (deterministic intent)

These are direct, navigational queries where the user knows exactly what they want:

• Single-item intent ("oranges", "milk", "bread").

• Exact brands or product families ("iPhone 15 Pro", "Diet Coke").

• SKUs, model numbers, sizes ("ABC123", "air max 270").

For these queries, lexical retrieval can handle token correspondence (matching words), but the business also expects to respect constraints, return predictable rankings, and have controllable outcomes. A merchandiser needs to ensure that a query resolves within the correct category boundaries, respects eligibility, and surfaces specific business priorities.

Governance is required to enforce the intended resolution. For example, “oranges” should map to the produce category, not to orange juice, orange marmalade, or orange soda.

Tail queries (exploratory discovery)

These are descriptive, intent-rich queries where shoppers are exploring:

• "Gift for grandpa who has a sweet tooth"

• "Jacket for hiking in the mountains"

• "Shoes for standing all day"

Lexical retrieval often struggles here. Semantic retrieval excels because it can connect the query concept to the product, even when wording does not match. But semantic retrieval alone is rarely sufficient either. Real queries often require constraints to be enforced, regardless of which retrieval method is used.

Constraints are orthogonal to retrieval method

Applying constraints to semantic retrieval doesn’t mean hybrid search. These are orthogonal concepts. Constraints, such as filters and boosts in Elasticsearch, can be applied to any lexical, semantic, or hybrid retrieval. The challenge is deciding how the query should be interpreted, which constraints must be enforced, and which retrieval strategy should be used.

Below are some examples of queries that combine retrieval with hard constraints:

• Oranges: Lexical retrieval for “oranges” plus a category constraint, such as “Fruits” or “Produce”, eliminating orange marmalade, orange juice, and orange soda.

• Fruits high in vitamin C under $4: Semantic retrieval for nutritional intent plus constraints limiting results to the fruit category and products under $4.

• Comfortable shoes for work: Semantic retrieval for contextual intent plus a category constraint limiting results to shoes.

These queries can't be handled by a single approach:

• Pure lexical retrieval is often insufficient here because phrases like “high in vitamin C” or “comfortable” may not exist as clean, structured attributes. They may need to be inferred from product descriptions, reviews, or specifications.

• Pure semantic retrieval is also not always sufficient because, without explicit constraints, a query like “fruits high in vitamin C” might broaden toward vitamin supplements, fruit-flavored drinks, or high-vitamin vegetables outside the intended category and price range.

A governance layer determines whether a query needs lexical retrieval, semantic understanding, constraint enforcement, or some combination of these. Without this layer, ecommerce teams may end up:

• Over-constraining: Using lexical retrieval for semantic requests (for example, "gift for grandpa").

• Under-constraining: Using semantic queries for high-intent head queries (for example, “oranges”).

The governance challenge is to build a system that can make the right judgment call for each class of query.

What happens without governance

The most common failure mode is straightforward: Teams take the raw user query and pass it directly into a single retrieval strategy (lexical, semantic, or hybrid), without an intermediate governance layer.

Lexical retrieval misses intended resolution

When a user searches for “oranges”, a lexical retrieval strategy may return anything containing that token: orange juice, orange marmalade, or orange soda. The system matched the term correctly, but without governance it may not resolve the intended shopping context (the fruit).

Semantic retrieval broadens beyond intended constraints

When a user searches for “oranges”, a semantic system may retrieve conceptually related items across nearby product concepts. The system may correctly understand the broader domain (fruit or produce), but without explicit governance it can still over-broaden beyond the user’s intended constraint (specifically oranges).

The gap is governance

What’s required is an upstream decision layer that determines query intent and enforces the right constraints before retrieval begins. This fixes issues such as the following:

• Similar or related items appearing alongside what the user actually wanted.

• Blurred category boundaries ("beverages" versus. "produce").

• Inability to implement seasonal boosts or campaigns.

• Unpredictable and unexplainable results.

Intent understanding and routing: The necessary control plane

A governed search system introduces a lightweight control plane in front of retrieval (prior to executing a query in Elasticsearch). The control will be discussed in detail in parts 3 and 4 of this blog series; for now, we just discuss what it can do but not how it works:

A control plane can detect intent, apply business policies, and ensure the appropriate retrieval strategy as follows:

1. Detect intent signals

• Is this query likely navigation versus discovery?

• Is it a known head query (milk, bread, bananas)?

• Is there a known product, brand, or category interpretation (for example, “oranges” should resolve to produce).

• Is the query an SKU-like pattern?

• Does the query fall under an active campaign or seasonal policy (for example, during Christmas, boost turkey-related results)?

• Does the query imply constraints (category, attributes, exclusions, price/size/color)?

2. Apply governance and business policies

• Enforce deterministic constraints first (category/attribute/negation/availability).

• Apply active merchandising policies (boost/bury/pin/override).

• Resolve conflicts with precedence rules (for example, campaign overrides versus global policies).

3. Route to the appropriate retrieval strategy

• Lexical (fast, deterministic) for navigational/high-intent head queries.

• Semantic retrieval for true discovery queries.

• Hybrid where combined lexical and semantic signals add value under explicit business constraints.

In practice, the output of the control plane is not simply “use hybrid” or “use semantic.” It’s a governed retrieval plan: an interpretation of the shopper’s intent, the constraints and policies that should apply, and the retrieval strategy that should be executed. A few simple examples make this concrete:

Shopper query

Governed interpretation

Example retrieval plan

“chocolate without peanuts”

Product-oriented query with a hard exclusion constraint

Lexical retrieval for chocolate plus an exclusion filter for products containing peanuts

“cheap olive oil”

Product/category query with a price constraint

Lexical retrieval for olive oil plus a price filter capped at the retailer’s threshold for cheap

“fruit high in vitamin C under $4”

Discovery query requiring semantic understanding plus hard constraints

Semantic retrieval for nutritional intent, constrained to the fruit category and filtered to products priced under $4

A control plane selects the right policy and retrieval strategy for each query consistently, predictably, and at scale. This makes advanced retrieval methods more predictable in production because intent-aligned constraints are enforced first and routing decisions are explicit rather than implicit.

How this relates to other approaches

Some teams use improved embedding models to better capture product semantics, which can materially improve semantic retrieval quality. Others use reranking approaches, such as Learning To Rank (LTR), to optimize result ordering based on engagement or business signals after retrieval. Both are valuable and often complementary. Better embeddings improve similarity matching. Reranking improves ordering among retrieved candidates.

Governance addresses a different layer of the problem: It sits upstream of retrieval. It decides which retrieval strategy to use (for example, lexical, semantic, or hybrid), what deterministic constraints are required, and which queries should combine multiple business policies.

What a governed control plane enables

Once a governance layer is in place, the operating model changes fundamentally. Revenue-critical queries become predictable. Business teams can update search behavior without waiting on engineering release cycles. And advanced retrieval methods, like semantic and hybrid, can be adopted incrementally, behind routing and guardrails, instead of as a global on/off switch.

The next post in this series explores what that operating model looks like in practice and why it may matter as much as the retrieval technology underneath it.

If a merchandiser has to open a Jira ticket and wait for a deploy to fix a revenue-critical query, the bottleneck isn't the engine; it's the operating model. Modern ecommerce search needs a way to translate business intent into controlled, auditable search behavior quickly and safely, while still using advanced retrieval where it adds measurable value.

What’s next in this series

The patterns explored in this series operate upstream of retrieval: translating business intent into the right query strategy before query generation begins. In the next post, we shift from the technical problem to the operational one: what happens when business teams can change search behavior without an engineering deployment, and why governance makes that safe.

Put governed ecommerce search into practice

Engineering bottlenecks, brittle application-layer logic, and unpredictable search results are problems that Elastic Services can help you solve in enterprise ecommerce services engagements. The governed control plane architecture described in this series was built by Elastic Services Engineering.

If your team is spending engineering cycles translating merchandising requests into code changes, or if your search relevance backlog never seems to shrink, we can help you assess your current architecture and build a path to governed, business-editable search. Contact Elastic Services.

Join the discussion

Have questions about search governance, retrieval strategies, or ecommerce search architecture? Join the broader Elastic community conversation.