How historical metrics were loaded before backfill

Even if loading historical metrics isn’t a very common use case, it’s an important step when teams are adopting TSDB. Two scenarios have been the most prominent: bootstrapping a new time series data stream and migrating data from a different system or data stream to a time series one.

Bootstrapping a new time series data stream

You want to start a new time series data stream with a week of historical data so you have something meaningful to query from the start. With existing tooling, you had to set index.look_back_time to the seven-day maximum in the index template, and all historical data would land in a single backing index. For anything beyond seven days, you needed to create past backing indices manually.

Migrating metrics from another system

You have months worth of metrics stored on a different system and want to move your full dataset to TSDB. You need to load months of metrics history alongside live ingestion. The workaround was to manually create all the necessary past backing indices with the right time_series.start_time and time_series.end_time and to index into it directly using the index name. You then attached it to the data stream via the modify data stream API. It worked, but it required understanding the index time semantics and repeating the steps for each time window, along with coordinating that process around ongoing writes.

We wanted both scenarios to feel as close to normal bulk indexing as possible.

What time series data backfill changes

In 9.5, Elasticsearch can create backing indices covering past time ranges, which extends the eligible write window backward.

The eligible write windowis the range of @timestamp values that a time series data stream accepts for new documents. 

In the past, the eligible write window was determined only by the existing writable backing indices at the moment the request was received by Elasticsearch.

In 9.5, Elasticsearch can expand the eligible write window in the past by creating backing indices. This converts the eligible write window to a sliding window extending from the present back to the first read-only or destructive lifecycle action. Common examples of these actions, which are typically defined within your lifecycle configuration, are downsampling or searchable snapshots. Examples also include retention configurations. 

So, given that loading historical data is enabled in a cluster, the eligible write window of the data stream with the following lifecycle configuration is determined by the downsampling action, because it’s the first action that makes backing indices read-only. So, for this data stream Elasticsearch accepts documents whose @timestamp is no older than three months.

Why loading historical data into TSDB is hard

TSDB consists of data streams optimized for timestamped measurements. It uses a columnar storage layout and enforces immutable dimensions. It also organizes data into time-bound backing indices; each index covers a specific time range and accepts only documents whose @timestamp falls within it.

As time passes, rollover creates new backing indices to cover upcoming ranges. Until this release, there was no corresponding mechanism for the past. Creating indices in the past is tricky because historical data might span over a long period of time and can arrive at Elasticsearch out of order. Consequently, Elasticsearch cannot determine the write timeframe that its backing index should cover. Our solution to this is to use a preconfigured interval and lazily create past backing indices.

How Elasticsearch creates past backing indices

When a document is detected whose timestamp isn't covered by any existing backing index, Elasticsearch determines the time boundaries for the missing indices and creates them. It then adds them to the data stream in a single atomic operation. 

Lazily creating the indices ensures that a single request in the past won’t overwhelm the cluster by requiring the creation of 300 indices all at once. It also doesn’t create indices before there are docs to write into them.

Proactive vs. reactive: How we chose the index creation approach

We explored two ways to detect when a past backing index needs to be created.

The first is proactive. Inspect each incoming document's timestamp before routing, and create any missing past backing indices up front. This keeps the write path clean. By the time a document is routed, the index it needs already exists. It does require the data stream to already exist with at least one time series backing index, since that's what we inspect to determine the eligible write window and the time boundaries of the new index. The downside is that it adds work to every bulk request targeting a time series data stream, even requests that contain no past timestamps and need no backfill at all.

The second is reactive. Let the document fail the normal indexing, intercept that failure, create the missing index, and retry. This avoids any overhead on the common case, since the extra work only happens when a mismatch actually occurs. The tradeoff is more complexity in the failure handling path and a retry on every backfill document.

We ran performance tests on the proactive approach against bulk requests with no past timestamps and found no measurable regression. The overhead of inspecting timestamps turned out to be negligible. That settled it. Proactive creation is simpler and consistent with how index auto-creation already works in Elasticsearch. Plus, it adds no measurable cost to the workloads that don't use backfill.

How Elasticsearch determines past index boundaries

Each new past backing index has three properties to compute: its duration, its start time, and its end time.

Property

How it's set

Constraint

Duration

Defaults to one day, configurable via the cluster setting data_streams.past_tsdb_index_interval

Minimum one hour. If the triggering timestamp falls in a gap up to 1.3 times the configured duration, Elasticsearch collapses it into a single bridging index rather than creating many tiny ones.

Start time

Anchored to the start of the next existing backing index, working backward in multiples of the configured duration

Increased to match the end time of the previous neighboring index, where they would otherwise overlap.

End time

Start time plus the configured duration

Reduced to match the start time of the next index, where they would otherwise overlap.

Handling concurrent writes

In a distributed setup, multiple nodes can receive bulk requests with overlapping past timestamps at the same time. Each node collects the timestamps that aren’t matching any of the existing indices and sends a request to the master node. 

The master node executes a cluster update that sorts them and then, one by one, checks whether the timestamp is covered by an existing or newly created index. Otherwise, it issues a new create index request with the time boundaries calculated as described above. The cluster updates are always sequential and guaranteed to produce valid cluster states, so new indices are guaranteed to not overlap with existing indices.

How lifecycle age works for backfilled indices

Past backing indices hold old data but are new indices. Without an adjustment, lifecycle features would apply downsampling and retention based on when the index was created rather than when the data is from. We account for this by using the index.time_series.end_time as the index.lifecycle.origination_date. As a result, the age of the index as perceived by both data stream lifecycle and index lifecycle management (ILM) is based on the age of its data and not its creation time.

How to use time series data backfill

How to enable time series data backfill

Backfill support ships disabled by default. Enable it at the cluster level:

Bootstrapping with historical metrics

To load historical data into a new time series data stream:

1. Create your index template. 

2. Initialize your data stream. (This is an important step because an existing data stream is a requirement for creating past backing indices.)

3. Start indexing. 

Past backing indices are created automatically as documents with historical timestamps arrive, each covering one day of data by default. No additional configuration is needed.

Data migration into an existing data stream

Migrating data within the eligible write window

For data that falls within the eligible write window of your data stream, point your migration pipeline at the data stream and let Elasticsearch manage the rest.

Migrating data beyond a read-only action

For data older than the write window (for example, you're migrating 18 months of metrics but downsampling kicks in after seven days), you need a separate data stream without read-only lifecycle actions. Retention isn’t an issue since the data would be deleted anyway. The pattern is:

1. Create an index template for the historical data stream, using the same mappings as the original but without a lifecycle:

2. Create the historical data stream. If this step isn’t executed, the first indexing request might fail. During the first indexing request, Elasticsearch can create the data stream but it cannot yet create any past backing indices, so indexing a historical document might fail. Creating the data stream explicitly ensures that all indexing requests will be accepted:

3. Index historical data into the historical data stream while current data continues flowing into the original.

4. When the load is complete, add lifecycle. This is only supported by data stream lifecycle since this feature functions on a data stream level:

5. Query across both data streams with a wildcard pattern (my-metrics*) or a data stream alias.

6. If retention is configured, delete the historical data streams when their data expires. Data stream lifecycle will delete the data but it won't clean up the data stream itself.

As you see, the historical data needs to fit on the target tier as a whole because lifecycle will be enabled after the data is loaded. If you have a large historical import, you might choose to split it into batches. Make sure each batch can fit on the target tier as a whole at the time of indexing, to avoid running your cluster out of disk space. Lifecycle will start processing the batch's indices as soon as it's enabled, but it will need time to process the whole backlog.

Protecting the cluster during large migrations: Downsampling floodgate

When data stream lifecycle runs against a data stream with many indices that all qualify for downsampling, it queues them simultaneously. Downsampling is CPU and I/O intensive; it reads and rewrites all data in an index. Queuing dozens of operations at once can overwhelm the master node with persistent task updates while it coordinates them.

The downsampling floodgate scenario could occur before backfill support (for example, when adding a lifecycle policy to an existing data stream with months of accumulated data). Backfill makes it more likely by design.

In 9.5 and serverless, we added flood protection to data stream lifecycle. It now tracks how many indices per data stream are actively being downsampled. If that count reaches a threshold, data stream lifecycle pauses queuing further operations for that data stream until the count drops. The threshold is configurable via the cluster setting data_streams.lifecycle.downsampling.max_indices_in_progress. Other data streams aren't affected.

Limitations and prerequisites of time series data backfill

• Backfill doesn’t apply to read-only indices. If downsampling or a searchable snapshot transition has already run on a time period, documents for that period are still rejected.

• The feature requires a preexisting time series data stream with at least one time series backing index.

• System data streams are excluded.

• Replicated data streams rely on the leader data stream, so no direct backfilling is possible.

• Scaling remains your responsibility. Loading months of data can trigger significant storage usage, force merge operations, and lifecycle activity in parallel. Check that your cluster has the headroom to manage it before starting.

Conclusion

Prior to the Elasticsearch 9.5 release, loading historical data into TSDB was a manual process. By automating the generation and management of past backing indices, we aim to transform historical data migration to a native capability of your standard ingest pipelines. The inherent complexity of managing time-bound indices remains, but it has transitioned from a user responsibility into an internal Elasticsearch function. Whether you’re bootstrapping a fresh data stream or migrating extensive historical datasets, the platform now handles the heavy lifting, allowing you to focus on analyzing your metrics. We look forward to seeing how these improvements streamline your adoption of TSDB.

Why snapshots pin primary shards in stateful Elasticsearch

In traditional stateful Elasticsearch, snapshots lock primary shards to their active nodes, completely preventing relocation. That works fine when cluster topology is stable and nodes stay online between maintenance windows, but stateless Elasticsearch works differently. Index data lives in an external object store, with local disk as a cache, and the cluster scales automatically based on CPU, memory, and data size, both vertically (upsizing nodes) and horizontally (adding nodes).

During vertical scale-up, existing nodes must vacate all shards and shut down before new hardware takes over. Since Elasticsearch version 8.13, shard snapshots can pause during node shutdowns and resume after relocation, so a long-running snapshot doesn't block an infrastructure update.

Horizontal scale-out is a different story: No nodes shut down, so pause logic never triggers. New nodes sit idle while existing nodes finish their snapshots, and as soon as a snapshot is queued, the primary shard is pinned to its node, significantly delaying relocation.

Clusters typically scale out because they're already under heavy load. Blocking shard relocations at that moment limits the cluster's ability to reduce pressure, which shows up as degraded indexing throughput and higher latency. The resource imbalance can also trigger unexpected autoscaling behavior. And even when overall topology stays the same, shard locking disrupts hotspot mitigation and workload distribution. These failures used to surface as server-side warnings: "undesired allocation due to snapshot."

Stateful Elasticsearch

Stateless Elasticsearch

Snapshot reads from

Local shard data on the node holding the primary

The object store, using file locations recorded in the commit

Primary shard during snapshot

Pinned to its node until the snapshot completes

Free to relocate at any time

Effect on horizontal scale-out

New nodes wait for in-flight snapshots before taking shards

New nodes take shards immediately, regardless of snapshot state

How stateless snapshots read directly from the object store

A shard snapshot pins primary shards because it needs to read local shard data. In stateless Elasticsearch, that data already lives in the object store, so reading from local disk is unnecessary. Letting snapshots read directly from the object store removes the requirement to lock primary shards. They can relocate freely, and backup is decoupled from cluster balancing.

Stateless commits include location information for each data file in the object store, so snapshots can read and stream directly to the snapshot repository (a separate object store bucket). In the future, we plan to look at server-side ranged copies, which object stores support natively, to skip the local copy step entirely.

Tracking commits when shards relocate mid-snapshot

A snapshot is bound to a specific commit point that determines which files to back up, and those files must remain accessible for the full duration of the operation. In stateful clusters, this is simple: The snapshotting node and the data node are the same, so the commit is managed locally and held until completion.

In a stateless model, the snapshotting node and the data node can be entirely separate, or they can diverge if a shard relocates mid-snapshot. To handle this, we added a transport action that acquires commits on remote data nodes over the network. The data node tracks which commit belongs to which snapshot and releases it once cluster state signals completion.

There's a wrinkle during relocation. A stationary shard relies on its commit point to preserve files. A relocating shard must release its commit so its local store can close cleanly. To keep files accessible through that transition, a newly recovered primary temporarily preserves all existing data files in the object store until notified of snapshot completion via cluster state. This handles both graceful relocations and ungraceful recovery from node or engine failures.

No more allocation delays and improved cache stats

After stateless snapshots shipped, the server-side "undesired allocation due to snapshot" warnings stopped. The chart below shows the before and after, with the release marked by the red arrow. Hotspot mitigation became more responsive because shard relocations no longer had to wait for backup operations.

Cache use is also improved. Snapshots that bypass local shard data stop competing with indexing for cache space. After the release (also marked in the chart), we observed the following two positive changes in cache metrics:

1. The median cache population throughput, defined as bytes per second for filling the local disk cache from the object store, increased about 50%.

2. Cache misses, where data must be retrieved from the object store to fill local disk cache, have dropped more than 60%. 

What comes after stateless snapshots

Object-store-native architectures are increasingly the standard for cloud-native data systems, and stateless snapshots are a step toward fully exploiting that model across Elasticsearch operations. Backups read from the object store, and shards move freely. Neither process waits on the other. Removing the local shard dependency is a step toward further modularizing the stateless architecture.

How dashboard activity logs differ from the Kibana audit log

Dashboard activity logs and the Kibana audit log both write structured logs about user actions, but they answer different questions.

Kibana audit log

Dashboard activity logs

Answers

Who accessed what, and when

Which dashboards are used, and how well they perform

Built for

Security and compliance teams

Kibana admins and dashboard owners

Tracks

Security-relevant events across Kibana

Five dashboard actions: create, update, delete, view, refresh

Enabled by

Its own setting in kibana.yml

user_activity.enabled: true in kibana.yml

What dashboard activity logs capture

The user activity service records structured events every time a user interacts with a dashboard. Each event captures what happened and to which dashboard, in addition to who did it. Five actions are tracked:

Action

Fires when

Includes duration

dashboard_create

A dashboard is created

No

dashboard_update

An edit is saved

No

dashboard_delete

A dashboard is removed

No

dashboard_view

A user opens a dashboard

Yes, time on the dashboard until they navigate away

dashboard_refresh

A user changes filters or time range, or auto-refresh runs

Yes, refresh duration

Very often, a dashboard_view event also triggers a refresh.

Every event carries the same core fields, with two that appear conditionally:

Field

What it holds

Present on

user.name

Name of the user who performed the action

Every event

user.email

Email address of the user

Every event

user.roles

Roles assigned to the user

Every event

object.name

Dashboard name

Every event

object.id

Dashboard ID

Every event

kibana.space

Kibana space the dashboard belongs to

Every event

client.ip

IP address the request came from

Every event

event.action

Which of the five actions occurred

Every event

event.outcome

Whether the action succeeded or failed

Every event

event.duration

Time taken, in nanoseconds

dashboard_view and dashboard_refresh

error.type / error.message

Error class and message when something fails

Events where event.outcome is failure

How Kibana records dashboard activity

Under the hood, Kibana plugins report events from the browser or server through a core client, and valid events are written to a dedicated logger. No data is sent to a third party; because events are standard JSON logs, you control where they go and how they’re ingested.

What you can do with dashboard usage data

Dashboard activity data answers five operational questions that would otherwise require further investigation:

• Clean up unused dashboards. Filter for dashboards with zero dashboard_view events. If nobody's looking at it, archive it. This is critical for customers who are managing thousands of dashboards. 

• Troubleshoot performance. The event.duration field tells you exactly how long each dashboard load or refresh takes. Sort by duration to find your slowest dashboards.

• Edit history. Every create, update, and delete is logged with the user who made the change. You no longer have to wonder who modified a critical dashboard or when it happened.

• Plan capacity. Identify users running heavyweight queries during peak hours. If one user's auto-refresh is hammering the cluster every 10 seconds, you'll see it.

• Monitor errors proactively. Dashboards throwing errors surface immediately through error.type and error.message fields, so you don’t need to wait for users to report them.

How to enable dashboard activity logs in Kibana

Add two lines to your kibana.yml ( the service is disabled by default):

Events will start flowing immediately using a default JSON console appender. You can customize the output appender and filter specific actions using the same logging configuration schema that Kibana already uses:

Ship these logs into an Elasticsearch index (for example, via Filebeat), and you have a fully queryable dataset of dashboard usage.

How to query dashboard activity in Discover

Once your activity logs are indexed, open Discover and point it at your user activity index pattern. You'll immediately see every dashboard interaction as a structured event, and they’re filterable by action type, user, dashboard name, and time range.

From here, you can quickly answer specific questions like the examples below. 

How many times was a dashboard viewed? 

Type your question in natural language in the Discover query editor, and press Cmd+J to automatically generate the Elasticsearch Query Language (ES|QL) query, as shown below:

Which dashboards had zero views in the last 30 days?

Dashboards with no activity simply don't appear in the logs, so you can't filter directly for zero views. Instead, this query works backward, pulling every dashboard created (and not deleted) in the past year and then checking which of those had zero views in the last 30 days.

Which dashboards took longer than 10 seconds to load?

Note that event.duration is recorded in nanoseconds, so the query converts to seconds before filtering:

Which dashboards are throwing errors, and what's failing?

This query shows dashboards with one or more panels throwing errors during dashboard_refresh events, so you can quickly spot recurring issues and prioritize fixes:

Build a dashboard usage overview with AI chat

We’re planning to add out-of-the-box dashboards along with the activity logs, but in the meantime, instead of manually building visualizations, open the AI chat in Kibana and ask it to create a dashboard from your user activity data. 

The generated dashboard gives you at-a-glance visibility into your most-viewed dashboards, heaviest users, slowest-performing panels, and recent errors; that is, exactly the operational view that large deployments need.

Get started with dashboard activity logs

User activity logs are available in Kibana 9.5. Begin by enabling the service. Then ship the logs into an index, and start building the operational visibility that your team has been asking for. For full configuration details and the complete event schema, see the user activity documentation.

What you'll discover

In this post, you'll learn how to:

• Use Elastic APM's built-in views to explore search latency and throughput and to explore errors.

• Define Service Level Objective (SLOs) for search, including latency targets and availability, along with search quality.

• Create SLOs in Kibana that track your search health over time, with burn rate alerting.

• Build operational dashboards with Elasticsearch Query Language (ES|QL) that show latency percentiles and time breakdowns and that include trends.

• Set up alerts for latency regressions and error spikes, along with zero-results rate increases.

• Establish an incident response pattern for search degradation.

What you'll need

• Search instrumentation from Blogs 2–4 (search spans with search.* attributes in Elastic).

• Kibana access with permissions to create SLOs and alert rules.

• Basic understanding of SLOs. (We'll explain the search-specific parts.)

• An Elastic cluster with an Enterprise subscription, an Elastic Cloud trial, or a local deployment with the trial activated.

Why search monitoring matters beyond cluster health

Search is the primary navigation path for a significant share of visitors, and search-initiated sessions tend to show stronger purchase intent than browse sessions. A search outage is a revenue event, rather than a minor feature degradation. A latency regression from 100ms to 500ms changes user behavior before anyone files a ticket.

Most platform teams monitor search at the infrastructure level, checking whether the Elasticsearch cluster is healthy and whether nodes are responding. They also determine whether the disk is full. These are all necessary but not sufficient. A cluster can be green while search quality silently degrades; for example, queries returning stale data after a bad index deployment or latency creeping up as the index grows. This could also include zero-results rates climbing because a synonym list wasn't updated.

The gap is between "search is up" and "search is working well."

A note on examples: As we have throughout this series, we use ecommerce search for concrete examples, but these reliability patterns apply equally to any search application, including content platforms, internal knowledge bases, job boards, and support portals.

OpenTelemetry search spans as monitoring signals

If your team followed Blogs 2–4 in this series, every search request already emits an OTel span with search.* attributes. These include the query text, result count, Elasticsearch execution time, and error status. Those spans land in traces-generic.otel-default in Elastic.

Following along with code? The reference project has all the instrumentation from Blogs 2–4. Generate traffic, and then follow along with the SLO and dashboard setup below. See queries/blog6_reliability.esql for ready-to-run queries.

The search team built that instrumentation for product analytics; that is, understanding what users search for and measuring click-through rates (CTRs) and conversion rates. They’ve prioritized relevance work, but the same spans contain everything you need for operational monitoring. Span duration gives you a latency signal, and search.result_count == 0 value reflects quality. Span errors point to availability signals.

This post shows how to put this operational value to work, beginning with what you can see right now in Kibana and then building SLOs and alerting on top of it, along with incident response.

Search monitoring out of the box with Elastic APM

Before building anything new, let's look at what Elastic APM already gives you out of the box.

If your search API is instrumented with Elastic Distribution of OpenTelemetry (EDOT) (as in Blog 2), it appears automatically as a service in the Elastic APM UI. Open Observability > APM > Services in Kibana, and select your search service (named search-analytics-demo if you're using the reference project). You'll immediately see the following:

Elastic APM service overview for search

The service overview page shows latency distribution and throughput over time, along with error rate, and doesn’t require configuration. You can see at a glance whether search is healthy, and the time-series charts make regressions obvious. If latency crept up after yesterday's deployment, you'll see it here.

Trace waterfall: breaking down search request latency

Click into any transaction, and you'll see the trace waterfall, which is a visual breakdown of every span in the request. For a search API call, this typically shows:

The waterfall makes the invisible visible. You can see that the 503ms API response breaks down into HTTP handling, a 241ms query rules lookup, and a 260ms Elasticsearch query,  plus the custom search span (36ms) carrying all of our search.* attributes. Click any span, and the metadata flyout shows exactly what was captured: search.query: "usb hub", search.result_count: 33, search.took_ms: 43, the index name, hit IDs, and more.

Blog 2 discussed the gap between search.took_ms and span duration. The waterfall shows you exactly where that gap lives, without writing any queries.

Automatic search error capture with OpenTelemetry

One of the most valuable things OTel auto-instrumentation gives you is automatic error capture. When an Elasticsearch query fails because of issues like a tripped circuit breaker or a timeout, or if an index isn’t found,  the span records the exception type and message, along with the stack trace. Blog 4 mentioned this as a side benefit of span-based conversion tracking; here it becomes an operational lifeline.

The errors tab on your service page automatically aggregates these, grouped by error type and frequency. The instrumentation captures the details for you, so you don't need custom error handling or logging. During an incident, this is often the fastest way to understand what's actually failing.

Service map: search API and Elasticsearch dependencies

The service map shows dependencies between your search API and Elasticsearch, making it easy to see whether a latency problem is in your service or in the cluster it depends on.

All of this is available the moment you deploy the instrumentation from Blog 2, without building any dashboards or writing any queries. This is the foundation everything else in this post builds on.

Defining service level objectives for search

An SLO defines good enough in measurable terms. You define what working means, measure it continuously, and alert when you're burning through your error budget too fast, instead of reacting when something breaks.

Elastic Observability has a built-in SLO framework that handles Service Level Indicator (SLI) calculation and budget tracking. It also takes care of burn rate alerting. You create SLOs directly in Kibana. No ES|QL or custom pipelines are required for the core indicators.

Three SLOs every search service needs

Navigate to Observability > SLOs in Kibana, and click Create SLO. The SLO creation workflow walks you through three steps: Define the SLI (what to measure), set the objective (the target), and describe the SLO.

1. Search latency SLO: 99% of queries under 250ms

Indicator type: Elastic APM latency target: 99% of searches complete in under 250ms.

The Elastic APM latency indicator is purpose-built for this. Select your search service (search-analytics-demo), and set the threshold to 250ms.  Elastic handles the rest, including calculating the percentage of transactions below the threshold and tracking your error budget over time.

Note: The Elastic APM latency indicator measures all HTTP transactions for the search-analytics-demo service, including health checks and click/cart/checkout endpoints, along with static asset requests, not just the POST /api/search endpoint. For a search-only latency SLO, use a custom Kibana Query Language (KQL) indicator with name: "search" AND attributes.search.query: * on the traces-generic.otel-default index. The Elastic APM indicator is still valuable for whole-service health. For full coverage, combine both.

This measures end-to-end span duration; that is, what the user actually experiences. If your Elasticsearch query takes 50ms but the user waits 300ms because of network overhead or slow application logic, this SLO catches it. Use search.took_ms in the Elastic APM waterfall to diagnose where the latency lives when the SLO starts burning.

2. Search availability SLO: 99.9% success rate

Indicator type: Elastic APM availability target: 99.9% of searches succeed.

The Elastic APM availability indicator calculates the percentage of successful transactions for your service. When the Elasticsearch client throws an exception or the search endpoint returns a 5xx, the span's status records an error and this SLO counts it.

Note: Like the latency SLO, the Elastic APM availability indicator covers all HTTP transactions on search-analytics-demo, not just POST /api/search. Click/cart/checkout errors will consume this budget. For a search-only availability SLO, use a custom KQL indicator with name: "search" AND attributes.search.query: * for good events and name: "search" as the total query.

An 0.1% error budget on 100,000 daily searches means that you can tolerate 100 errors per day. That's tight, but search errors are hard failures and the user gets nothing. Availability SLOs should be stricter than latency SLOs.

3. Search quality SLO: tracking zero-results rate

Indicator type: Custom KQL target: 85% of searches return at least one result (zero-results rate < 15%); index: traces-generic.otel-default; good query: name: "search" AND attributes.search.result_count > 0; total query:name: "search" AND attributes.search.query: *.

Note on KQL versus ES|QL: The SLO framework uses KQL for its indicator filters rather than ES|QL. KQL uses field: valuesyntax and is the same language you see in the Kibana search bar. The ES|QL queries throughout this series are for ad hoc analysis and dashboards; KQL here is the SLO indicator's document filter. Both query the same traces-generic.otel-default index.

This is the SLO that surprises most teams. A search that returns an empty result set isn't an error; HTTP status is 200 and the span status is OK. Plus, no exception was thrown. But from the user's perspective, it failed. They asked for something and got nothing.

The quality SLO uses the custom KQL indicator type because it relies on our custom search.result_count attribute, which the built-in Elastic APM indicators don't know about. But the SLO framework handles everything else, including budget tracking and burn rate calculation, along with alerting.

A sudden spike in zero-results rate, such as from 12% to 40% over an hour, is almost always an infrastructure event, like a failed index deployment or a mapping change that broke queries. It could also be a synonym list misconfiguration. That's an operational problem, not a relevance problem.

Reading your search health in the SLO overview

Once the latency, availability and quality SLOs are created, the SLO overview page shows your search health at a glance:

Each SLO shows the current value, the target, the remaining error budget, and the burn rate. Green means healthy:  Search availability is at 100%, and Search quality is just above its 85% target. Red means violated: Search latency is at 50% against a 99% objective, with the burn rate breached at 200x the sustainable rate. When a budget bar starts shrinking faster than expected, you know something changed, even before users complain.

Clicking into the SLO detail shows burn rate across multiple time windows (1h, 6h, 24h, 72h) and the historical SLI trend. It also shows remaining error budget. For the latency SLO, the Elastic APM latency indicator tracks the percentage of transactions below your 250ms threshold. For the quality SLO, the custom KQL indicator uses traces-generic.otel-default with the good query filtering for attributes.search.result_count > 0. This is where the custom search.* attributes from Blog 2 pay off, since they're the foundation of meaningful SLOs.

Burn rate alerting for search SLOs

When you create an SLO through the Kibana UI, a default burn rate alert rule is automatically created. This is where the real operational value lives.

Burn rate alerts improve on threshold alerts ("error rate > 1%"), which are noisy and miss slow degradation. : Burn rate alerts measure how fast you're consuming your error budget relative to the SLO window.

A burn rate of 1.0 means that you're spending budget at exactly the sustainable rate, but a burn rate of 10.0 means that you're burning 10x too fast and you'll exhaust the budget in 1/10th of the window.

The default burn rate rule uses a multi-window approach, with four severity levels:

Severity

Burn rate

Long window

Short window

What it means

Critical (page)

> 14.4x

1 hour

5 minutes

Exhausts budget in ~50 hours

High (ticket)

> 6.0x

6 hours

30 minutes

Exhausts budget in ~5 days

Medium (review)

> 3.0x

24 hours

120 minutes

Exhausts budget in ~10 days

Low (awareness)

> 1.0x

72 hours

360 minutes

Trending toward exhaustion

The short window prevents alerting on brief spikes that self-resolve, and the long window catches sustained degradation. Together, they balance responsiveness with alert fatigue.

Routing search alerts to PagerDuty, Slack and Jira

Alerts are only useful if they reach the right people in the right tools. Elastic's alerting framework supports a wide range of connectors out of the box, including:

• Incident management: PagerDuty, Opsgenie, xMatters for on-call routing.

• Chat: Slack, Microsoft Teams for team notifications.

• Case management: Jira, ServiceNow for automatic ticket creation when SLOs breach.

• Custom: Webhooks for integrating with any system via HTTP.

You can also use Elastic's built-in cases to track incidents directly within Kibana, linking alerts and traces in one place, along with investigation notes, with push to Jira or ServiceNow when escalation is needed.

A typical routing setup:

Alert

Severity

Channel

Latency SLO burn rate > 14.4

Page

PagerDuty

Availability SLO burn rate > 14.4

Page

PagerDuty + Slack

Quality SLO burn rate > 6

Ticket

Jira (auto-create) + Slack

CTR anomaly (machine learning [ML] job)

Notification

Slack (search team)

Anomaly detection for search quality

Some search degradations are gradual shifts that slip past threshold-based alerts, rather than sudden spikes. A relevance regression after a model update might reduce CTR by 15% over a week, and latency might creep up by 5ms per day as the index grows. These are real problems, but they don't trigger burn rate alerts until it's too late.

Elastic's anomaly detection is built for exactly this. It learns normal patterns in your search metrics and flags deviations automatically, and you don’t have to configure any thresholds. 

Detecting search quality degradation with ML anomaly detection

• Latency anomalies:

 Elastic APM anomaly detection can be enabled directly from the Elastic APM UI for your search service. It learns the typical latency distribution, including daily and weekly patterns, and alerts when behavior deviates. A gradual 5ms/day creep will eventually register as anomalous before it hits your SLO threshold.

• CTR drops:

A relevance regression is invisible to traditional monitoring; latency is fine and errors are zero, plus the result counts are normal, but the ranking changed and users aren't clicking. Anomaly detection on click volume per query is a practical proxy: When a query that normally receives 20 first-click events per hour drops to 5, something likely changed.

To set this up: In Kibana → Machine Learning → Anomaly Detection, create a new job. Use the Multi-metric wizard, and select traces-generic.otel-default as the index. Configure a count detector on attributes.search.first_click split by attributes.search.query. This creates a per-query click-volume baseline and alerts when individual query engagement drops outside the expected range.

Note: This job detects click-volume anomalies per query, not CTR (which requires dividing clicks by searches). Click volume is a useful proxy (a CTR regression usually manifests as a drop in absolute click count), but be aware that a traffic surge with flat click volume would show as a CTR drop without triggering this alert. For true CTR anomaly detection, use a scheduled ES|QL transform to materialize hourly CTR values and run anomaly detection on the computed ratio.

Route the resulting ML alert rule to your Slack search channel.

• Throughput shifts:

A sudden drop or unexpected surge in search volume can indicate upstream problems (like load balancer changes or traffic shifts) or downstream issues (such as search becoming unresponsive or users retrying).

Configure ML anomaly alert rules to route these to your notification channels. These complement your SLO burn rate alerts; burn rates catch budget consumption, and anomaly detection catches pattern changes.

Building a search monitoring dashboard with ES|QL

SLOs tell you whether search is healthy. When they indicate a problem, you need a dashboard that tells you why.

The search team and the on-call team need different views of the same data. A search engineer wants query-level detail, such as which queries have low CTR and which ones return nothing. They’re also interested in where to invest in relevance. But an on-call SRE wants the operational picture, including whether search is fast and whether it’s up. It also wants to know whether search is degrading, and if so, since when.

Search monitoring panels for the on-call dashboard

Build it in Kibana dashboards using Kibana Lens panels. Lens supports ES|QL as a data source, so the queries from Blogs 2–4 can power dashboard panels directly. The key panels include:

• Search throughput over time:  A sudden drop is often the first sign of a problem.

• Latency percentiles (p50, p95, p99) over time: When they diverge (p50 flat, p99 spikes), you have a subset of slow queries.

• Error rate over time: Spikes here mean hard failures.

• Zero-results rate over time: A step change upward, especially correlated with a deployment, means something changed in the index or query pipeline.

The ES|QL for each panel follows the patterns from earlier blogs. For example, a latency percentile panel:

This includes three lines on one chart. When they diverge, such as when p50 stays flat but p99 spikes, you likely have a subset of queries that are slow while the majority are fine. That's a different diagnosis than all queries slowing down (cluster-level pressure).

Drill-down panels: slowest queries and top zero-result queries

For investigation, add a few detail panels, such as:

Slowest queries: A table showing the queries with the highest p95 latency and their search volume. During an incident, this narrows the problem from "search is slow" to "these specific queries are slow."

• Top zero-result queries: A table showing which queries most frequently return nothing. When zero-results rate spikes, this panel immediately shows which queries are responsible.

These drill-down panels use the same ES|QL patterns as Blogs 2 and 3, just surfaced on a persistent dashboard instead of run ad hoc.

Search incident response using OpenTelemetry traces

As an example, an alert fires, noting that search latency has spiked. What happens now?

The trace data from Blog 2's instrumentation gives you a structured path from symptom to root cause.

Step 1: Assess scope

Start at the on-call dashboard, and get answers to the basics:

• When did it start? Narrow the time range to the degradation window.

• How bad is it? Is p50 affected (all queries slow) or just p99 (a subset)?

• Is it just search? Check the Elastic APM service map to determine whether the Elasticsearch dependency is also degraded.

Step 2: Find the problem queries

If the problem is a subset of queries (p99 spike but p50 is fine), use the slowest queries panel or run:

Adjust the 100 ms threshold to match your environment's normal range. It can be lower for a fast cluster or higher if your data volume makes 100ms typical. This narrows the problem from "search is slow" to "these specific queries are slow." That's the difference between restarting the cluster and investigating a specific query pattern.

Step 3: Drill into the trace waterfall

Pick a slow query, and open it in the Elastic APM trace view. The waterfall shows exactly where time was spent. (Refer back to the trace waterfall GIF above to see a real example of a POST /api/search trace broken down into its component spans.)

The overhead gap between search.took_ms (Elasticsearch time) and span duration (end-to-end time) is your diagnostic tool:

Scenario

search.took_ms

Span duration

Diagnosis

Elasticsearch slow

400ms

430ms

Elasticsearch problem: Check slow log, cluster metrics.

App slow

50ms

350ms

Application / network overhead: Check serialization, network.

Both slow

400ms

700ms

Multiple issues: Investigate both.

If the problem is in Elasticsearch, drill into the Search Profile API or cluster monitoring. If it's application overhead, look at the spans around the search span in the waterfall.

Step 4: Correlate with events

Check whether the degradation correlates with:

• Deployments: Did someone deploy a new version of the search service or push a new index?

• Cluster events: Is Elasticsearch under memory pressure, or are there long garbage collection pauses? Or maybe the disk is I/O saturated?

• Network: Is latency between the search service and Elasticsearch elevated?

Elastic Observability's unified platform makes this correlation straightforward because traces, logs, metrics, and infrastructure data all live in the same Kibana instance. You're adding filters in the same interface, rather than switching between tools.

Going further: Infrastructure metrics and cost attribution

This post focuses on what you get from trace data; that is, the spans your search API already emits. But OTel and Elastic Observability support a wider instrumentation picture that becomes valuable as your search infrastructure matures.

• Infrastructure metrics. Adding host and container metrics (like CPU, memory, disk I/O, and network) alongside your traces lets you correlate search performance with infrastructure use. When p99 latency spikes, you can immediately see whether the Elasticsearch nodes are under memory pressure and whether garbage collection  pauses are increasing. You can also check whether disk I/O is saturated, and you can do all this in the same Kibana interface. The Elastic Agent collects these automatically for your infrastructure, and the infrastructure monitoring UI surfaces them alongside your Elastic APM data.

• Total cost attribution (TCA). With infrastructure metrics flowing alongside traces, you can start attributing infrastructure costs to specific services and operations. How much compute does your search service consume? How does that correlate with query volume? If a new ranking model doubles CPU usage per query, you can see the cost impact directly. This is particularly valuable for teams running search on cloud infrastructure where costs scale with resource consumption; understanding the cost per search helps justify infrastructure investment and identify optimization opportunities.

• Logs correlation. OTel auto-instrumentation injects trace context (such as trace ID and span ID) into your application logs. This means that when you're investigating a slow search in the trace waterfall, you can click through to the exact log lines from that request, including Elasticsearch slow log entries and application debug output. It also includes error details that don't fit in span attributes. The logs correlation feature automatically ties them together.

These are natural next steps once you have traces working. Each one extends the same unified platform, without new tools or separate pipelines.

How search analytics and search monitoring share one data pipeline

Here's how it all fits together:

The data flows from the instrumentation you built in Blog 2. This one investment supports two audiences: The search team gets product analytics (Blogs 2–5), and the SRE team gets operational monitoring (this post). Neither team needs separate data pipelines.

Getting started with search monitoring in Elastic

This is the last post in the series, and it brings us full circle. Blog 1 describes the vision: Instrument search once with OTel, and send spans to Elastic. Then use ES|QL to answer any question about search behavior. Blogs 2–4 build the instrumentation and analytics, and Blog 5 shows how to feed that data back into relevance improvements. This post shows how the same data powers operational monitoring, including SLOs, alerting, anomaly detection, and incident response.

The key takeaway for search engineers is that the instrumentation you built for analytics already generates the signals. The SLOs and alerts are built-in capabilities of Elastic Observability, as are the dashboards. You're closer to production-grade search monitoring than you might think. Observability isn't a separate discipline you need to learn from scratch. 

If you've been following along and built the instrumentation from Blogs 2–4, start here:

1. Open Elastic APM: Look at your search service, and explore a trace waterfall. You can also check the errors tab.

2. Create three SLOs: Latency (Elastic APM latency), availability (Elastic APM availability), and quality (custom KQL for zero-results).

3. Enable anomaly detection: One click in the Elastic APM UI for latency anomalies.

4. Build the on-call dashboard: Four Lens panels with the queries from this post.

By the end of an afternoon of work, your search service can have the same observability coverage as any other critical production system.

Get started

Working code

• Reference project: Working code for the entire blog series; clone, configure, and run.

Elastic APM and traces

• Elastic APM Overview: Elastic APM concepts and trace analysis.

• Elastic APM UI: Service overview, transactions, dependencies, errors.

• Service Maps: Dependency visualization and health.

• OpenTelemetry Integration: OTel ingestion in Elastic.

SLOs and alerting

• SLOs in Elastic Observability: Creating and managing SLOs.

• Create an SLO: SLI types (like Elastic APM latency, Elastic APM availability, or custom KQL), time windows, budgeting.

• SLO Burn Rate Alerts: Multi-window burn rate alerting.

• Alert Connectors: Slack, PagerDuty, webhook, email integrations.

• Alerting Framework: Rule types and configuration.

• Jira Connector: Automatic ticket creation from alerts.

• ServiceNow Connector: ITSM integration.

• Elastic Cases: Built-in incident tracking with external push.

Anomaly detection

• Anomaly Detection Overview: Unsupervised time series anomaly detection.

• Elastic APM Anomaly Detection: Enable ML for latency, throughput, error rate.

• ML Anomaly Alert Rules: Alerting on detected anomalies.

Dashboards and ES|QL

• Kibana dashboards: Building operational dashboards.

• Lens: Visualization editor.

• ES|QL in Lens: ES|QL-powered dashboard panels.

• ES|QL Overview: Language reference.

Elasticsearch operations

• Slow Log Configuration: Threshold-based query slow logging.

• Search Profiling: Profile API for query execution analysis.

• Monitoring Elasticsearch: Cluster stats, search rate, latency.

From this series

• Modern search analytics with OpenTelemetry: The vision.

• Instrument your search API : Search spans and search.* attributes.

• Measuring search quality : CTR, MRR, click distribution.

• From clicks to conversions: Conversion tracking and revenue attribution.

• Personalizing search from behavior: Judgment lists, rank features, Learning To Rank (LTR).

This is the final post in a six-part series on search analytics with OpenTelemetry and Elastic. Start from the beginning: Modern search analytics with OpenTelemetry, or to start building, jump to Instrument your search API.

Observability workloads are storage-intensive by nature, and the composition of that storage determines both cost and query performance. ES95 introduces adaptive compression: rather than applying the same encoding to every numeric field, it automatically selects the encoding that best matches each field's structure. The result is a 33.6% reduction in total doc-values storage, 19% to 74% reduction on floating-point gauge metrics and a 92% reduction in @timestamp. No configuration or migration required.

Observability data is storage-intensive

Data processing systems are rarely limited by how fast they can compute. They’re limited by how fast they can move bytes: off disk, across the network, and through the memory hierarchy. Compression is how a storage engine trades CPU time for memory bandwidth, spending comparatively cheap CPU cycles so fewer bytes have to travel through the parts of the system that are usually constrained. In a read-heavy system like Elasticsearch, that trade-off pays back every time data is queried, often long after it was written.

Storage size and query performance move together; fewer bytes on disk means fewer bytes to read on every range query, every aggregation and every dashboard load. Compression is not just about saving storage. Every byte that is never written is also a byte that never has to be read.

The right encoding depends on the structure of the values themselves, and the largest wins come from exploiting the structure already present in the data rather than squeezing an opaque stream of bytes. Few workloads expose that structure more clearly than observability metrics.

A single host reports hundreds of metrics every few seconds, including CPU utilization, memory ratios, request latencies, and network throughput. Multiply that by thousands of hosts across weeks of retention, and the bytes accumulate fast. Most of that volume is structured but not uniform: timestamps arrive at near-constant intervals from thousands of concurrent series, counters increase monotonically, while gauges like 23.47 or1.15 are short decimal measurements.

A fixed compression approach cannot adapt to that variety. A timestamp column and a floating-point gauge column compress through fundamentally different techniques, but a codec that applies the same approach to both will necessarily handle one of them poorly. For most of Elasticsearch's time-series codec history, gauges were on the losing end of that trade-off.

The structure the old codec was not built to exploit

Elasticsearch stores time-series numeric values in doc values: a column-oriented structure where all values for the same field sit adjacent on disk. That adjacency makes compression possible: the codec compares consecutive values of the same field, finds patterns, and exploits them.

The time-series codec before ES95 applied the same fixed encoding to every numeric field: delta encoding followed by normalization, GCD (greatest common divisor) reduction, and bit-packing. Each encoding technique activated where it helped and skipped where it would not. For timestamps and integer counters, this approach was remarkably effective. For floating-point gauges, it could find almost nothing to work with.

The reason is how they’re stored. To support range queries, Elasticsearch stores floating-point values as integers that preserve numeric ordering. A change of 0.01 in a CPU percentage reading translates to a jump of trillions in that integer space. The codec sees those large jumps and has no strategy to further reduce their footprint. Storage stays near the original eight bytes per value.

The codec was doing the right thing with the representation it had, but the latter was chosen for querying, not compression, and the goals conflict at the bit level.

The cost of a fixed format

A compression stage for floating-point values was already on the roadmap, so the interesting part wasn’t the algorithm. The obstacle was architectural.

The previous codec baked its compression approach into the storage format. Adding a new encoding meant changing the meaning of existing bytes on disk, which forced a format migration, a rollout that can last weeks or months in large production clusters. Over time, that migration burden constrains codec development itself. The question stops being Is this a good compression idea? and becomes Is it worth another format migration? That rigidity limits the cadence of codec evolution and leads to missed compression improvements.

The right encoding without configuration

ES95 solves this at the architecture level for time-series indices. Each field's encoding is no longer baked into the format. It is selected automatically at write time, based on what the field mapping already declares: the field's name, its data type, and its metric role. Timestamps are encoded differently than counters. Counters are encoded differently than gauges. ES95 encodes all of them, and it chooses the right strategy for each.

Users already tell Elasticsearch everything the codec needs to know. The mapping describes the data; the codec chooses the compression strategy.

Compression strategy is a codec concern, not a user concern.

The alternative would have been to expose per-field encoding selection as a configuration parameter, letting users opt in to better compression for specific fields. That would shift the burden of knowing which encoding fits which data type onto those least equipped to make that call and would guarantee that most deployments never see the benefit. ES95 keeps that decision inside the codec, where it belongs. This matters most in managed and serverless deployments, where users expect the system to automatically make optimal storage decisions.

The timestamp result nobody planned for

With the adaptive architecture in place, the team set out to ship the planned float-compression algorithm. Before it arrived, the architecture proved itself by substantially improving compression for timestamps.

A time-series index is sorted first by its time-series identifier (_tsid constructed by the metric’s dimensions) and then by timestamp within each series. Timestamps on disk aren’t one smooth sequence; there are many smooth sequences laid end to end, one per series, with a large jump at every boundary where one series ends and the next begins.

The codec compresses data in fixed-size blocks without regard to those series boundaries. A block straddling a series boundary holds timestamps from two different series. The jump between them breaks monotonicity, reducing delta encoding effectiveness on blocks spanning different time series. A block that would otherwise compress to near-zero bits per value ended up needing nine or more, because bit-packing encodes every value in a block using the same fixed number of bits, so one large jump sets the cost for all of them.

In the ideal case, every block belongs to a single series: timestamps increase at near-constant intervals, delta encoding captures the regularity, and bit-packing compresses the result to near-zero bits per value. With few series, boundary blocks are rare and the overhead barely registers. On an observability cluster ingesting millions of documents across thousands of series, that changes. The cost scales along two dimensions: series count and data density. More series means more boundary events; sparser series means multiple jumps packed into single blocks. In high-churn environments, both compound, and boundary blocks accumulate into a standing tax on the most-read field in any time-series workload. It’s why @timestamp storage grew faster than the data that produced it.

The fix was to detect series boundaries and treat each run as its own independent sequence. Instead of trying to encode across the jump between two series, which forces every value in the block to pay the storage cost of that one large jump, each run is compressed on its own terms. The boundary simply becomes a seam: the jump is never seen by the encoder on either side.

That encoding is called SplitDelta. @timestamp and monotonic long counters now use it by default. No format change. No migration. Existing segments retain legacy encoding.

On the high-cardinality Elasticsearch Rally benchmark, that single unplanned encoding cut counters storage by 20%–30% and @timestamp storage by 92.4%, from 1.03 GB to 79 MB. Gigabytes to megabytes, and no, that isn’t a typo.

The pluggable pipeline had already paid for itself. SplitDelta, which wasn’t part of the original plan, slotted in without a format change or migration before ALP even shipped.

ALP: recovering the decimal that was always there

Most floating-point metrics can be expressed as short decimals with no loss of accuracy: CPU utilization at 23.47, load average at 1.15. ALP (Adaptive Lossless floating-Point compression) recovers that decimal structure from the floating-point representation, converting values into integers that the existing pipeline already handles well. ES95 feeds ALP's output into the same mature integer compression pipeline used for timestamps and counters, extracting additional savings rather than treating ALP as a standalone encoding. Values that don’t fit ALP's model (such as irregular high-precision floats or special values) fall back to direct bit-packing or the original representation without degrading the rest of the block.

ALP lets Elasticsearch treat floating-point metrics according to the structure they actually contain rather than the binary representation they happen to use. It’s applied automatically to double-valued gauge fields, selected by field type and metric role, through exactly the door the architecture had built for it.

What the numbers say

Here’s what the Elasticsearch Rally benchmark looks like on a high-cardinality workload containing 2.26 billion data points. Results are from an internal tsdb-metricsgen benchmark.

Field or metric

Storage reduction (%)

@timestamp

−92.4%

cpu.load_average.5m

−74.3%

system.cpu.utilization

−63%

memory.utilization

-19%

Total doc values

−33.6%

That overall 33.6% reduction deserves context.

A time-series index contains more than metrics. Every data point also carries the labels that identify the series: host names, IP addresses, regions, container IDs. Those dimension fields are stored as keywords. ES95 doesn’t target dimensions.

On this benchmark, two dimension fields, host.ip and host.mac, accounted for 44% of doc-values storage after ES95 ran. The 33.6% total reflects that mix. The per-field breakdown is the honest picture. Compression for dimension fields is an active area of work.

The per-field variation is the most convincing result. Some gauges shrank by nearly three quarters, while others moved by less than a fifth. That spread is direct evidence that ES95 matches compression to the structure actually present in each field. A fixed encoding treats every field identically and misses most of those wins.

Better compression without extra configuration

The storage reductions from SplitDelta and ALP are the most visible results of ES95. The more consequential result is the architecture that produced them.

Before ES95, every new compression technique required a format evolution. That reality shaped which ideas were practical to pursue. Today, new encodings become implementation decisions inside the codec rather than migration projects. Existing data never needs to move, and users gain better compression on newly written data simply by upgrading Elasticsearch. SplitDelta and ALP are the first encodings to benefit from this architecture. They will not be the last.

Asking users to choose compression algorithms would only duplicate information Elasticsearch already has. There are no per-field compression parameters to tune, and no expert knowledge is required to get good storage efficiency. Different fields get different strategies because ES95 understands what kind of data each field contains, not because a user configured it. As the codec evolves, those decisions evolve with it. The API does not.

In Elasticsearch Serverless, good defaults are part of the product. Users expect the system, not configuration, to make storage decisions. ES95 is designed to honor that expectation: encoding that starts right and gets better over time.

The compression was always there

ES95 establishes a new standard for how time-series codec evolution works. New encodings become implementation decisions, not migration projects. Users get better compression on newly written data with every Elasticsearch upgrade.

The compression was already in the data. ES95 just removed what was hiding it.

How to configure frozen_after in data stream lifecycle

frozen_after sits at the top level of the lifecycle, next to data_retention and downsampling.

That's the whole feature, at the API level. Indices in my-data-stream stay on hot for 30 days, then move to frozen for the remaining 60. After 90 days they're deleted, and the backing snapshot goes with them.

It composes with the rest of the lifecycle, including downsampling:

Same options in an index template:

The order of values is enforced: frozen_after has to be less than data_retention and greater than any downsampling.after. The API rejects configurations that don't make physical sense.

Where frozen tier data is stored: the default snapshot repository

Frozen tier data is held as partially-mounted searchable snapshots, which means DLM needs a snapshot repository to write into. Rather than make you choose a repository per lifecycle, 9.5 introduces a cluster-level default snapshot repository.

DLM uses this repository for every frozen tier index in the cluster. On Elastic Cloud Hosted (ECH), the default is pre-populated with found-snapshots so existing clusters work out of the box. You can change it to a repository you control if you'd rather keep your frozen data in a bucket you own (useful if you want object versioning, lifecycle backups to Glacier, or anything else that needs bucket-level access). Wherever you can set frozen_after in Kibana, the UI shows the current default repository inline and links to the place to change it, so you can see where frozen data will be written.

If the cluster doesn't have a default repository configured, you can still write a lifecycle with frozen_after. The API accepts it but returns a warning:

The data stays on hot until a default repository is configured and exists. The same logic applies if the cluster lacks a valid Enterprise license. Errors are visible in the data stream lifecycle status API for that stream.

Configuring frozen_after in Kibana

Kibana in 9.5 lets you set frozen_after and the default snapshot repository from the UI. In Streams, the Retention tab shows the frozen phase on the lifecycle timeline alongside hot and any downsampling steps. Click the timeline to open the data lifecycle flyout, set frozen_after, and see the timeline update before you save. Index Management's Data Streams page opens the same flyout.

How frozen tier conversion works in data stream lifecycle

When a backing index ages past frozen_after, DLM walks through five steps in order:

1. Clone. Mark the index read-only and clone it to a zero-replica copy so the original stays available while conversion runs.

2. Force merge. Merge the clone to a single segment. On completion a cluster state marker is written; duplicate force-merge requests (for instance after a master failover) are deduplicated, so a restart doesn't repeat the work.

3. Snapshot. Write the merged clone to the default repository, and record the snapshot name in cluster state on success. If a stalled snapshot from a previous attempt is detected, DLM deletes it and re-runs the step.

4. Mount. Create a partially-mounted searchable snapshot index from the snapshot.

5. Swap and delete.Once the mounted index's shards are fully allocated, atomically swap it in for the original in the data stream, then delete the original. The swap is atomic, so query results don't see a data volume dip during the transition.

On failure, DLM retries from the last successful step on the next run.

Each step is idempotent, and the cluster state markers make sure work already done isn't repeated after a master failover. Throttling caps concurrent conversions so a lifecycle change covering thousands of indices doesn't overwhelm the cluster. Errors surface in the data stream lifecycle status API and roll up to the lifecycle health indicator.

Scope and limitations of frozen_after

• Data indices only. frozen_after applies to a data stream's data indices. Failure store indices don't currently support the frozen tier and continue to be governed by their own data_retention setting.

• Enterprise license required. Frozen tier in DLM is implemented with searchable snapshots and requires an Enterprise license. You can write frozen_after on any license, but data won't move to frozen until the license is valid.

• Serverless ignores the field. In Elastic Cloud Serverless, frozen_after is accepted but ignored - Serverless manages tiering on your behalf. Built-in templates may include the field, so we don't reject it, but the step is skipped.

Getting started with frozen_after

1. In Kibana, open Streams or Index Management and choose a data stream backed by data stream lifecycle.

2. Open the Edit data lifecycle flyout, set a frozen-after value, and save. The lifecycle timeline shows the new phase.

3. On a self-managed cluster, set repositories.default_repository to a repository you control. On ECH, found-snapshots is already configured if you want zero setup.

4. For declarative workflows, write the same configuration into your index templates so new data streams pick up the lifecycle automatically.

Learn more

• Data stream lifecycle

• Frozen tier overview

• Searchable snapshots

• Manage data retention for Streams

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Why dense vectors create off-heap memory pressure that heap charts miss

Semantic search and kNN rely on dense_vector fields. Elasticsearch stores much of that data in off-heap memory. It’s related to how the Java Virtual Machine (JVM) operates, but it isn’t the same thing as heap usage.

In production, the heap versus off-heap split shows up in a familiar pattern:

• Heap looks fine for weeks, while the dense vector off-heap footprint quietly grows.

• Heap circuit breakers stay quiet or only spike late because the pressure sits outside the JVM.

• kNN search and bulk ingest slow down, queues build, and nothing on the dashboard points at vector RAM as the cause.

Heap limits protect Java allocations. They don’t tell you whether vector off-heap footprint still fits the RAM envelope that your deployment actually runs in. AutoOps already watches cluster health broadly; Vector memory pressure adds a focused read for vector-heavy nodes when memory and load signals line up.

How AutoOps measures vector RAM, heap, and headroom

AutoOps works from the same node stats metrics you already use for Stack Monitoring. For each node, it tracks three derived numbers:

Symbol

Meaning

Source (typical)

Chart (see below)

V

Vector off-heap footprint

indices.dense_vector.off_heap.total_size_bytes

First, green line

A

Available RAM in the product view

Delta between os.mem.total_in_bytes and os.mem.used_in_bytes

Second, green line

H

Headroom

A − V (headroom_bytes)

First, blue line

H > 0 means there’s a modeled runway: Vector use still fits comfortably in that accounting. H ≤ 0 means that you’re in a compression regime: Vector footprint (V) meets or exceeds the free RAM (A) picture that AutoOps can align in telemetry. On small tiers, that can be common under load. The insight emphasizes trends, growth in vector off-heap footprint, and corroborating stress, not a single negative snapshot.

AutoOps also tracks a compression regime flag (fraction of recent samples where H ≤ 0), so brief flickers don’t dominate the story (see third chart below):

How vector memory pressure detection works: Expansion and compression

Vector memory pressure is a single HIGH severity AutoOps event, which adapts to the compression regime:

• Expansion (H > 0): Emphasize shrinking headroom, hence a growing vector off-heap footprint.

• Compression (H ≤ 0): Emphasize ΔV, heap context, I/O, and latency. "Hours until H hits zero" isn’t the main narrative when headroom is already gone.

The detector requires three layers before it fires:

1. Memory carriers: Compression regime, shrinking headroom, or sustained growth in vector off-heap footprint.

2. Operational latch: Search or indexing latency versus rolling baselines, filesystem read stress (paired with latency or heap), indexing throttle, thread-pool queues or rejections, segment creep, or heap circuit breakers when paired with other stress, as circuit breakers alone don’t provide enough evidence to be escalated without corroborating stress.

3. Heap hot: Heap usage elevated versus a 24-hour rolling median on that node, so compression alone on a calm heap doesn’t fire the insight.

That pairing is intentional. Vector pressure without load might be capacity planning, and load without vector pressure might be a different root cause. Together, vector memory, operational stress, and heap heat surface the vector RAM story when the node is actually in trouble, not on every compressed mapping while the heap stays normal.

Validation: Memory pressure detection on a 4 GiB node under kNN load

We stress-tested vector memory pressure detection on 4 GiB Elastic Cloud Hosted deployments with throttled dense-vector ingest (~2,000 docs per minute) and steady kNN search (~8 queries per second). Across Hierarchical Navigable Small World (HNSW), Better Binary Quantization (BBQ) HNSW, and DiskBBQ mapping profiles over 24–48 hours:

• Vector off-heap footprint grew from near zero to about 6 GiB on the tightest runs (more than 4.7 million vectors indexed) in both HNSW test runs.

• Nodes spent most of each run in compression (H ≤ 0), which is expected when vector footprint exceeds total RAM in this model.

• Vector memory pressure stayed off while heap held near 50%, even with compression and pool stress building.

• On both HNSW and BBQ HNSW, the insight fired once heap climbed past ~75% with memory compression and thread-pool queue stress, roughly an hour before heap neared saturation. Node out of memory (OOM) and circuit breakers followed in the same window, as did slow search/indexing. As we can see on the dashboard below, performance drops drastically due to corroborating stress toward the end of the test run:

• On DiskBBQ, where compression was chronic but heap stayed normal, the insight didn’t fire,since storage rather than memory was the limiting factor. Disk and watermark signals are the right path to follow for that profile. As we can see on the screenshot below, all indicators stayed steady with constant performance throughout the test, even though we filled up the disk with more than 70 million vectors on the same instance type:

That timing is the point: Operators get a vector-first read tied to real RAM stress, with subsystem context, rather than an alert on every compressed index or only a red heap chart after the node is fighting on every front.

What to do when AutoOps raises vector memory pressure

Here’s the insight that AutoOps now raises when it detects vector memory pressure:

Recommendations in the product map to concrete actions:

1. Reduce vector footprint where quality allows: Fewer dimensions, quantized mappings, archive or split indices, reindex with a leaner mapping.

2. Tune kNN load: Lower num_candidates, reduce concurrent query rate, narrow filtered kNN where possible.

3. Consider DiskBBQ when HNSW in RAM is the bottleneck (evaluate recall/latency trade-offs for your use case). If you’re already on DiskBBQ and the heap memory is calm, treat disk and watermark insights as the primary signals. Note that DiskBBQ requires an Enterprise license.

4. Right-size RAM when vector off-heap footprint (V) trends up and headroom stays tight.

AutoOps links affected nodes and summarizes regime and stress in plain language. Treat it as “act now, rather than waiting for red on every chart.”

Where AutoOps vector memory pressure monitoring is available

Vector memory pressure is available wherever AutoOps runs against Elasticsearch 9.2+, including:

• Elastic Cloud Hosted (ECH).

• Elastic Cloud Serverless (coming soon).

• Self-managed via Cloud Connect.

AutoOps is included at all subscription levels for supported deployment types and doesn’t consume ECUs on ECH.

What is an Agent Builder OTel trace and what does it capture?

When your agent runs, Agent Builder records everything that happened as an OpenTelemetry (OTel) trace. Think of a trace as a receipt for a single conversation turn. Every LLM request, tool call, and agent action is recorded as an individual span in Elasticsearch, which is a unit of work or operation. When opted in, additional details like user prompts, LLM responses, tool outputs, and conversation IDs are captured as structured span attributes on the chat span. All of this is scoped to your Kibana space.

How to enable agent tracing and privacy controls in Kibana

To begin capturing trace data, ensure the following toggles under Agent Traces in Gen AI Settings are active within your environment:

• agentBuilder:tracing:enabled — This gen AI setting manages the collection of traces and is enabled by default.

Advanced privacy controls, located under the default tracing toggle, also let you collect message content. While prompts and tool outputs are masked by default, you may choose to enable them to support more robust traces:

• agentBuilder:tracing:includeUserPrompts

• agentBuilder:tracing:includeLlmResponses

• agentBuilder:tracing:includeToolDetails

• agentBuilder:tracing:includeSystemPrompt

• agentBuilder:tracing:includeRealNames: Retains real agent/tool names instead of anonymizing to custom

• agentBuilder:tracing:includeRealIds: Retains the actual conversation identifiers instead of the default hashed versions. This means trace data collects original IDs, which can link traces to specific user sessions (PII).

Only enable these if you understand what data your agents handle and have appropriate data governance in place.

How Agent Builder stores OTel trace data in Elasticsearch

The Agent Builder utilizes OpenTelemetry semantic conventions. This results in a structured hierarchy of spans that provides a granular view of the agent's internal logic:

Span

Type

What it captures

`invoke_agent <name>`

CHAIN

Full turn lifecycle, from user input to final reply

`invoke_agent <name>`

AGENT

Single agent execution: reasoning, tool calls, reply

`chat <model>`

LLM

One LLM request: model, latency, token counts

`execute_tool <toolName>`

TOOL

Tool invocation: arguments, duration, result

Trace data is written to a dedicated data stream per Kibana space, keeping conversation data cleanly isolated. To query your traces in Discover, target the index for your space directly:

For the default space, that’s traces-agent_builder.otel-default. If the advanced privacy controls are turned on, then those span attributes will also be shipped to the traces data stream with the original spans. This index lets you query the raw message content to see what's actually being said in conversations. It is best practice to avoid using wildcards to prevent mixing data from unrelated spaces.

Agent Builder ships with a built-in skill called agent-builder-traces, installed automatically whenagentBuilder:tracing:enabled is on. You can use it to ask questions directly about your trace data, making it easy to explore agent behavior without writing ES|QL from scratch.

How to debug agent behaviour with the OTel trace waterfall view

The trace waterfall shows every step of an Agent Builder session as a timeline. To open it, navigate to the specific turn in the conversation UI and select the trace icon.

This launches a waterfall timeline breaking down every step of your agent's execution. At the top level, you'll see the invoke_agent parent span with the full end-to-end duration of your agent run. Nested beneath it are chat spans, each representing a single LLM request and showing exactly how long the model took to respond. Alongside those are execute_tool spans, one per tool call, where you can see which tool was called, what arguments it received, and how long it ran. This allows you to trace the exact sequence your agent followed, pinpoint timing bottlenecks, and see where errors occurred.

How to build token cost dashboards from trace data

Discover gives you raw trace data, but most teams want answers to operational questions like "how many tokens did we burn today?", "which tool is called most often?", and "how many unique users interacted with the agent this week?". These would require a dashboard built directly against the trace data.

There is an Elastic-managed out-of-the-box dashboard called [Elastic] Agent Builder Overview that can be installed by clicking in the top-right corner of the Agent Traces section within GenAI Settings.

It contains basic details spanning Token Usage and Cost, Conversation Volume and Latency, Agent Execution, and Tool Call Frequency and Errors.

However, a custom dashboard may be more efficient. If you want something more tailored, build Lens panels directly against the OTel trace index. Some useful panels to start with:

1. Most active conversations by token spend

create a horizontal bar chart against traces-agent_builder.otel-<space-id>. Set the x-axis to gen_ai.conversation.id and sort descending and limit to the top 10. Set the y-axis to a sum of gen_ai.usage.input_tokensplusgen_ai.usage.output_tokens. Input the formula as: 

Conversations with the most LLM round-trips

Option to create this visualization with a simple ES|QL query that would look like:

There is also a dashboard-management skill that can be used to help create traces visualizations using natural language.

How to set token cost alerts for Agent Builder conversations

Token consumption is the most direct cost lever for LLM-based agents. A single runaway conversation can blow through your monthly budget before anyone notices.

Elastic alerting lets you define a threshold rule directly against the trace data. Navigate toObservability > Alerts > Manage Rules > Create Ruleand selectElasticsearch queryas the rule type.

A rule that fires when any single conversation exceeds 256,000 tokens looks like this as an ES|QL rule:

Set the schedule to run every 15 minutes and configure the action to send a Slack notification or open a PagerDuty incident. The gen_ai.conversation.id value in the alert payload gives you the exact conversation to inspect.

What’s coming next for Agent Builder observability

Agent traces give you visibility that goes far beyond debugging. Once you've built dashboards and configured alerts against Agent Builder trace data, you have a live pulse on how your agents are behaving in production. If you haven't already, spin up Agent Builder in your Kibana space, make sure tracing is enabled, and run a few conversations. Check Discover, pull up the waterfall view, and see what your agent is actually doing under the hood.

This is the first in a series of posts on Agent Builder observability. Coming up, we'll go deeper on using theagent-builder-traces skill to query your data conversationally, building custom evaluation pipelines from trace data, and using traces to feed conversation history back into your agents. Your agents have been keeping secrets. It's time to make them talk.

Why AutoOps for Elasticsearch needs clearer prioritization

Running Elasticsearch at scale requires administrators to monitor cluster health, performance, capacity, and configuration at the same time. AutoOps now provides a clearer way to distinguish conditions that threaten cluster functionality from significant but less urgent degradation. The redesigned interface also follows familiar Elastic Cloud Console patterns, making active issues easier to find and investigate.

What changed in AutoOps: severity, navigation, configuration, and page design

The monitoring engine remains the same. The redesigned layout, navigation, and workflows now follow familiar Elastic patterns.

A clearer severity model

We added Critical as a new severity level for conditions that pose an immediate threat to cluster functionality and require urgent intervention. Several events previously classified as High are now Critical. Others are now Medium because they represent potential risk rather than active, significant degradation. The reclassified events are:

• Promoted from High to Critical: Disk Watermark Flood Stage, Master Not Discovered, and Status Red.

• Demoted from High to Medium: Disk Watermark Low Threshold, Disk Watermark Low, and Disk Watermark Configuration Incorrect.

Severity

What it means

Critical

Immediate threat to cluster functionality. Urgent intervention required.

High

Significant degradation to usability, performance, or stability.

Medium

Potential risk that can escalate if left unaddressed.

Low

Minor anomalies with minimal operational impact.

Info

Routine operational updates and configuration changes. No action required. (Coming in a near-future update).

Every severity level ships with an updated icon set and color palette. Levels are fixed so teams can build consistent runbooks and notification filters: route Critical and High events to PagerDuty or Slack, keep Medium and Low in the console for periodic review, and when Info arrives, use it for awareness without alert fatigue.

Deployment view: open events and history, side by side

The redesigned deployment view presents the existing Open events and Event history tabs in a clearer layout.

Event flyout: a clearer view of what matters

The event detail flyout is redesigned around action. High-severity events include a notification callout and an interactive badge that shows whether alerts are configured and links directly to setup. Recommendations collapse by default so the core event stays in focus. Settings live in the flyout menu; share is a separate icon in the header. The Dismiss action appears only when your role has the required admin permissions and the event is dismissible.

AutoOps overview: triage active events across your Elasticsearch fleet

The Overview page is reorganized around how operators scan an estate. Elasticsearch context sits directly under the page header, and active events appear as event ribbons below the deployments table. Each ribbon shows the latest active event in your selected time range; if the same event type is open on other deployments, a new badge lets you expand the view without opening each resource individually. Event search moved to the left for quicker filtering.

The “Events over time” chart moved off Overview to keep this page focused on fleet-level triage; open a single deployment when you need that timeline.

Nodes, Shards, and Indices are designed with easier navigation and information hierarchy

Nodes view now uses updated chart components and the Elastic UI color scheme, with clear expansion indicators on accordion sections. Event and instance lists that duplicated deployment-level views were removed to reduce noise.

Shards view improves node selection and groups view controls in the upper-right corner. A horizontal scrollbar supports wider layouts, and the time slider now uses native Elastic UI components. Node selection in Shards view now works across larger clusters and presents up to 100 nodes at a time.

Index view keeps the Indices table experience you already use, including sorting, time-range brushing, and chart zoom behavior tuned for meaningful ranges.

Template Optimizer

The Template Optimizer now provides a searchable list of templates ordered by the most recently identified recommendations. You can open each recommendation directly or expand the JSON panel to inspect the complete template.

Configure notifications and event settings

Notification settings now include connector search, clearer filters, and a simpler connector editing flow. Event settings moved from a popup to a flyout, matching the pattern used across AutoOps. Notification reports retain the same 10-day history window with minor layout updates, and dismiss events use updated confirmation components aligned with Elastic UI.

Navigation and controls

The deployment picker now shows deployment ID and real-time cluster status, with copy actions for deployment name and ID in the dropdown sub-menu. Node selection supports select-all, select-by-tier grouping, and clear master node indication. The date picker follows the same relative-range and custom-range model used in Kibana and other Cloud Console monitoring views.

AutoOps roadmap: API, MCP, CLI, and agentic experience

Looking ahead, we are building toward a headless, agentic AutoOps experience. A forthcoming public AutoOps API will make insights and raw metrics available outside the AutoOps interface. Administrators and agents will be able to query the API directly or store its data in Elasticsearch. The API will also provide the foundation for integrations with MCP, Elastic Agent Builder, the Elastic CLI, Kibana, and native AutoOps chat.

• Hosted MCP server: Make AutoOps insights available to MCP clients such as Claude and Cursor.

• Native Elastic Agent Builder tool: Use AutoOps insights in Elastic Agent Builder.

• Elastic CLI support: Access the AutoOps API through the Elastic CLI.

• AutoOps in Kibana: Surface relevant insights and metrics within Kibana.

• Native AutoOps chat: Investigate cluster issues through an agentic chat experience within AutoOps UI in Elastic Cloud Console.

The application redesign is the foundation; these surfaces will meet operators where automation and AI already live. Read more about what is coming on the Elastic public roadmap.

How to start using the redesigned AutoOps in Elastic Cloud Console

Sign in to Elastic Cloud Console, open a deployment, project, or connected cluster, and select AutoOps from the navigation. Learn more in the AutoOps documentation.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

When dealing with dozens of users who actively connect and use your platform backed by an Elasticsearch cluster, it’s important to quickly grasp what potential bottlenecks are and how to overcome them. The main question is: Where to start? Here’s your potential decision path:

The path starts with AutoOps to detect cluster-level problems, like resource pressure or bad shard distribution. If the cluster is healthy, the Profile API helps identify slow queries; candidates can come from the slow query log or application-side logging. Next, ES Rally benchmarks index settings that may limit throughput. If all three come back clean, the bottleneck is on the client side.

How to detect cluster-level performance problems with AutoOps

The first question is always the same: Is the cluster itself the problem?

AutoOps is designed to provide real-time cluster diagnostics, deliver tailored advice, and help you improve the health of your clusters quickly. It comes by default in Elastic Cloud, and it has been added recently as a free option for self-hosted configurations.

It’s very easy to set it up, and it takes no more than five minutes to see data flowing into a comprehensive list of graphs that AutoOps offers you. The idea behind it is to install an Elastic Agent close to your cluster that reports back to the AutoOps platform which is connected with your Elastic Cloud account. Here’s the installation guide.

What’s nice about it is that it provides you with instant warnings and suggestions that are hard to spot through existing monitoring cluster technology. What’s really useful is that it also shows solutions to the issues. 

In our delivery logistics scenario, AutoOps surfaced one finding that explained the user-reported slowness: severe load imbalance across the cluster. Node es01 was handling nearly all traffic while the other three sat idle, with search latency spiking to 30 seconds, as we'll see in the graphs in the next section.

How AutoOps surfaces node hotspotting and shard imbalance

AutoOps node performance graphs revealed that es01 was handling nearly all indexing traffic (5.6 docs/sec) while es02, es03 and es04 were idle.

Below, we see that CPU usage was concentrated on a single node (es01) which was hosting the heavier index, while the other three nodes were mostly idle.

The next signal of unusual behavior appeared in the search latency graph. Using the AutoOps per-node latency view, we uncovered some unexpected results.

While other nodes show no latency, the heavy-loaded node shows symptoms of latency that affects search apps. Not only was it a write node, but also it was the one serving all search requests. During analysis, we found that a delivery index was stored on only one node, instead of being distributed across all four. That was the root cause of the issue. By reindexing the data (with increased number of primary shards), we balanced the query load and eliminated the primary issue.

This article focuses on a single AutoOps finding to keep the diagnostic flow clear. For deeper dives into the kinds of issues AutoOps surfaces, see the dedicated articles on hotspotting, high CPU usage, and long-running queries.

How to find slow Elasticsearch queries using the Profile API

Once the cluster is healthy, the next question is: Are there specific, expensive queries? Finding candidates is the first step. You can explore them either by using Elasticsearch’s slow query log or through application side logging, by measuring time required for every search request. In our case, infinite scroll on a tracking delivery screen was a problem while users would scroll deeper into results.

The Profile API is a powerful tool for analyzing long-running queries and finding search bottlenecks. Getting started is simple: Just set profile=”true” in any of your search queries, and the responses will contain a profile section with detailed timing breakdown. It highlights which phase dominates execution time: the query phase, aggregations, or data fetching.

Let’s take a closer look at the deep pagination case. In a case of infinite scroll in a mobile application, the user continues scrolling and the app responds by fetching 100 documents per request.

As users continues scrolling, the requests can reach very deep pagination levels, while the from parameter grows:

The profile data, summarized below and converted from nanoseconds to milliseconds, makes the bottleneck clear: Most of the time is spent outside the query and fetch phases on coordination work caused by the large from offset. Elasticsearch must collect 9,100 matching documents, sort them, discard the first 9,000, and return only the requested 100.

Component

Phase

Time

ConstantScoreQuery contains

Query

34.4ms

BooleanQuery

Query

31.1ms

QueryPhaseCollector contains

Collect

22.1ms

SimpleFieldCollector (9,100-doc priority queue)

Collect

19.1ms

FetchPhase

Fetch

12.1ms

Other steps, like request parsing and deserialization, queue wait time, response building

1,885.4ms

Total

2,004.2ms

Fixing deep pagination with search_after

In this case, the search_after approach is up to twice as fast because it avoids scanning and discarding earlier results. Instead, it resumes from the last document returned on page 90, using its delivery_pickup_datetime value (in epoch milliseconds) as a cursor and fetching only the next 100 records.

As shown in the comparison below, performances are much better with the search_after approach.

Component

Phase

Time

ConstantScoreQuery contains

Query

6.8ms

BooleanQuery

Query

6.2ms

QueryPhaseCollector contains

Collect

2.6ms

PagingFieldCollector

Collect

2.0ms

FetchPhase

Fetch

2.2ms

Other steps like request parsing and deserialization, queue wait time, response building

9.4ms

Total

29.2ms

Performance comparison

Component

Deep pagination (from:9000)

search_after

Time saved

% Saved

ConstantScoreQuery

34.4ms

6.8ms

27.6ms

80.2%

BooleanQuery

31.1ms

6.2ms

24.9ms

80.1%

QueryPhaseCollector (total)

22.1ms

2.6ms

19.5ms

88.2%

FieldCollect

19.1ms

2.0ms

17.1ms

89.5%

FetchPhase

12.1ms

2.2ms

9.9ms

81.8%

Total

2,004.2ms

29.2ms

1,975ms

98.5%

The total query time drops from 1,004 ms to 29 ms (a 98.5% improvement) almost entirely because Elasticsearch no longer has to build and discard a 9,000-document priority queue on every request.

How to benchmark Elasticsearch index settings with ES Rally

With the cluster healthy and queries optimized, the final question is: Are the index settings themselves a bottleneck?

ES Rally is the official benchmarking tool made by Elastic. Its key strength is reproducibility: You run the same workload against two configurations on the same cluster and hardware, so any difference in results is purely down to the settings you changed. ES Rally is able to measure improvements under identical conditions: same cluster, hardware, and dataset.

Why default Elasticsearch index settings limit bulk indexing throughput

In many cases, slow indexing is caused by default index settings that are suitable for development but not production. For instance, the default refresh_interval of 1 second increases resource usage because every refresh creates a new searchable segment, and a replica count of 1 doubles the number of write operations per indexing request.

Our delivery logistics platform, for example, ingests thousands of new records daily in bulk, exactly the kind of workload where these defaults hurt. Setting refresh_interval to -1 disables refreshing during the bulk loading phase, and temporarily dropping replicas to 0 halves the write operations. Both settings are restored after the import is complete.

In this demo, we won’t focus on how to prepare custom data for benchmarking, but it’s worth mentioning a nice article about it.

For benchmarking purposes, you set up two folders: one with the current settings and another with the contender settings. The sample dataset in this case contains 1 million records. All the files described below can be found in this repository.

Both track folders contain an index-settings.json file. These files let you adjust mappings, shard counts, replica settings, and field types; for example, converting a field from double to scaled_float or from text to keyword. Because ES Rally tracks can be rerun quickly, it’s easy to experiment with different configurations and evaluate new optimization ideas as you iterate.

The next step is to run race commands (for current and contender race) to gather stats.

Quick tip: Using meaningful --race-id values (rather than the auto-generated ones) makes the comparison command much easier to run.After both races are complete, compare the results:

Additionally, reports are generated into csv files that can be easily manipulated to extract important data.

Metric

Current settings

Contender

Change

Cumulative indexing time

0,826517

0,755967

+8.54%

Mean throughput

11,090 docs/s

50,921 docs/s

+359 %

Median throughput

10,821 docs/s

51,304 docs/s

+374 %

Min throughput

10,335 docs/s

41,387 docs/s

+300 %

Max throughput

13,006 docs/s

55,342 docs/s

+326%

The contender configuration delivers roughly 3–4× faster indexing throughput, purely from adjusting refresh_interval and replica count during the bulk load phase.

In practice, a single comparison is rarely enough. Common follow-up experiments include converting double fields to float or scaled_float, changing text fields to keyword, adjusting shard count, and tuning the refresh interval. Because ES Rally tracks rerun quickly, iterating through these options is straightforward.

What to do when Elasticsearch isn't the bottleneck

If AutoOps shows a healthy cluster, the Profile API shows fast queries, and ES Rally confirms that index settings are not the limiting factor, the bottleneck is on the client side. The Elasticsearch side is no longer the place to look. Common application-layer causes:

• Network latency between the client and the cluster, especially across regions, VPNs, or proxies.

• Client-side deserialization overhead on large response payloads. Use _source exclude or the fields parameter to return only what the client actually needs.

• Client-side queueing before requests reach the cluster. The Elasticsearch client uses an HTTP connection pool. Under concurrent load, requests wait in line for a free connection. The Profile API never sees this wait because the request hasn't left the client yet.

• Single search calls where _msearch would batch independent queries into one network round trip.

• Single-document indexing where the _bulk API would amortize per-request overhead across many documents.

The diagnostic value of AutoOps, the Profile API, and ES Rally is precisely that they let you definitively rule out the cluster, the query, and the index settings before turning to the application code. When the three tools come back clean, the investigation moves out of Elasticsearch and into the client.

For investigating these on the application side, Elastic APM is the natural next tool: It traces request paths through the client and surfaces exactly the kind of pre-cluster wait time the Profile API can't see.

Conclusion

Going back to the original problem, slow search on a delivery logistics platform, we traced the issue from the infrastructure level down to the query and settings level:

• AutoOps revealed an uneven shard distribution that concentrated all query load on a single node, causing latency spikes of up to 30 seconds for end users.

• The Profile API showed that deep pagination was the source of the slow queries. Switching from from/size to search_after eliminated 98.9% of the latency.

• ES Rally confirmed that optimizing index settings during bulk ingestion, specifically refresh_interval and replica count, can increase throughput by 3-4×.

Each tool answers a different question. AutoOps gives you the "Is something structurally wrong?" view. The Profile API answers "Why is this specific query slow?" And ES Rally validates "Do these changes actually improve things, and by how much?" Used together and in this order, they cover the full diagnostic surface for the Elasticsearch side of the application. When all three come back clean, the next step is to look at the client, as outlined in the previous section.

FIPS 140-3 support for Elasticsearch and Kibana is generally available in Elastic 9.4 for self-managed deployments. NIST has stopped accepting new FIPS 140-2 submissions, with existing certificates winding down in September 2026. Every layer of your infrastructure needs to catch up, and your search and analytics platform is no exception. Federal programs, defense integrators and regulated enterprises are actively moving procurement requirements to 140-3. With Elastic 9.4, your stack can answer "yes."

Two problems, one deadline: why FIPS 140-2 is no longer enough

If you've been running Elasticsearch in FIPS 140-2 mode, you already know the value of having a compliant Elastic Stack. But two pressures are converging:

• The standard is sunsetting. FIPS 140-2 certificates are being phased out. Procurement officers, auditors, and authorization bodies are shifting their requirements to 140-3. A stack that only supports the old standard is a stack with an expiration date on its compliance story.
• Auditors don't accept "close enough." It's not sufficient to run FIPS-approved algorithms. Your application layer needs to be explicitly configured for FIPS mode, with nonapproved algorithms disabled and cryptographic boundaries clearly documented. Partial compliance is noncompliance.

Can't I just put Elasticsearch behind a FIPS-compliant load balancer or run it on a FIPS-hardened OS? No, and here's exactly why: FedRAMP's network security requirements apply to cryptographic operations at the application layer, not just the network boundary. Intranode communication, keystore encryption, and credential hashing all need to happen inside a validated module. A compliant perimeter around a noncompliant application layer isn't a FIPS deployment; it's an audit finding waiting to happen.

What FIPS 140-3 mode does in Elasticsearch 9.4

When you flip FIPS mode on, Elasticsearch and Kibana restrict every cryptographic operation to FIPS-approved algorithms and delegate all of it to the validated module in your runtime. Here's what that covers and how.

• High-grade security Transport Layer Security (TLS) everywhere, no exceptions. Node-to-node transport, the REST API over HTTPS, Kibana talking to Elasticsearch: All of it uses only FIPS-approved cipher suites. Noncompliant suites aren't deprioritized. They're rejected.
• PBKDF2 replaces bcrypt for password hashing. Bcrypt isn't FIPS-approved, so in FIPS mode, Elasticsearch switches to PBKDF2 for the native realm, file realm, and any stored credentials. Your users and service accounts stay protected with an algorithm that the auditor won't flag.
• Keystore encryption stays inside the boundary. Secrets in the Elasticsearch keystore, API keys, repository credentials, and encryption keys for Kibana's saved objects are wrapped with FIPS-approved key derivation and encryption. No gaps between the cluster is compliant and the secrets are compliant.
• You supply the cryptographic module. FIPS 140-3 support in Elasticsearch is built on the Bouncy Castle FIPS Java API 2.0, a FIPS 140-3 validated cryptographic module that runs inside the Java Virtual Machine (JVM). Elasticsearch delegates all cryptographic operations to Bouncy Castle; it doesn't implement its own crypto functions. Elasticsearch uses your own FIPS 140-3 JVM and Bouncy Castle FIPS provider, giving you full control over your cryptographic boundary and module versioning.

Kibana takes a different path, running in a Node.js environment configured with a FIPS-compliant OpenSSL 3 provider. Together, both components operate within clearly defined cryptographic boundaries clean enough to diagram for an auditor.

Who needs FIPS 140-3 support in Elasticsearch

• Federal and defense teams operating Elastic inside FedRAMP boundaries, Cybersecurity Maturity Model Certification–scoped (CMMC-scoped) environments, or Defense Information Systems Agency (DISA) Security Technical Implementation Guide–hardened (STIG-hardened) infrastructure. You can now upgrade to 9.x without punching a hole in your authorization documentation. Your Authorization to Operate (ATO) package references FIPS 140-3, not a soon-to-expire 140-2 certificate.
• Financial services and healthcare organizations where Payment Card Industry Data Security Standard (PCI DSS), Sarbanes‑Oxley Act (SOX), or Health Insurance Portability and Accountability Act (HIPAA) audits ask how your search infrastructure handles cryptography. FIPS mode gives your compliance team a one-word answer instead of a three-paragraph explanation.
• Anyone fielding Do you support FIPS 140-3? in a procurement questionnaire. That question is showing up in enterprise requests for proposal (RFPs), partner security assessments, and insurance underwriting checklists. With 9.4, the answer is yes.

Migrating from FIPS 140-2 to FIPS 140-3

If you're running FIPS 140-2 on Elastic 8.x today, you don't have to jump to 9.x on day one. Elastic 8.19 continues to support FIPS 140-2, and FIPS 140-3 support is available starting in 8.19.15. That gives you two paths:

• Stay on 8.19.15+, and upgrade the standard in place. If you're not ready to move to 9.x, you can switch from FIPS 140-2 to 140-3 on your current major version. Your cluster stays put, and your compliance posture moves forward. FIPS 140-2 certificates remain valid through September 2026, so you have a window. But the earlier you transition, the less you're depending on a sunset timeline, and you can also benefit from new features!
• Move to 9.4, and land on 140-3 directly. If you're planning a major version upgrade anyway, 9.4 gives you FIPS 140-3 from the start, along with everything in the 9.x line: Elasticsearch Query Language (ES|QL) latest functionalities, improved ingest, updated security detections, and performance improvements. No compliance trade-off required.

Stay on 8.19.15+

Upgrade to 9.4

FIPS standard

140-3 (from 8.19.15)

140-3 (from the start)

Major version change

No

Yes

New 9.x features

No

Yes

FIPS 140-2 certificates valid until

September 2026

N/A (lands on 140-3 directly)

Kibana covered

Yes

Yes

Either way, the configuration model will feel familiar. You enable FIPS mode in your YAML config, point to a FIPS-validated JVM, and the stack handles the rest. And Kibana is covered on both paths: Your visualization and analytics layer operates within the same compliant boundary as Elasticsearch.

How to enable FIPS 140-3 in Elasticsearch 9.4

FIPS 140-3 support is available now in Elastic 9.4 for self-managed deployments with a Platinum or Enterprise subscription, the same licensing tier as FIPS 140-2. For setup instructions, supported JVM versions, configuration details, and known limitations, see the FIPS compliance documentation.

Download Elastic 9.4, and give your compliance team some good news.

Questions? Your Elastic account team can help scope a FIPS deployment, or drop into the Elastic community forums to compare notes with other operators running in regulated environments.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Frequently asked questions

Does Elasticsearch support FIPS 140-3?

Yes. FIPS 140-3 support for Elasticsearch and Kibana is generally available in Elastic 9.4 for self-managed deployments. All cryptographic operations are delegated to the Bouncy Castle FIPS Java API 2.0, a FIPS 140-3 validated module. A Platinum or Enterprise subscription is required.

When does FIPS 140-2 expire?

NIST stopped accepting new FIPS 140-2 module submissions. Existing FIPS 140-2 certificates remain valid through September 2026. Organizations running Elasticsearch in FIPS 140-2 mode should plan their migration before that deadline to avoid gaps in their compliance documentation.

Can I migrate from FIPS 140-2 to FIPS 140-3 without upgrading to Elasticsearch 9.x?

Yes. FIPS 140-3 support is available in Elastic 8.19.15 and later. You can switch from FIPS 140-2 to FIPS 140-3 on your current major version without moving to 9.x. Alternatively, upgrading directly to Elastic 9.4 lands you on FIPS 140-3 from the start.

What cryptographic changes does FIPS mode make in Elasticsearch?

When FIPS mode is enabled, Elasticsearch restricts all operations to FIPS-approved cipher suites, replaces bcrypt with PBKDF2 for password hashing, and delegates all cryptographic functions to the Bouncy Castle FIPS provider. Non-approved cipher suites are rejected, not just deprioritized.

Does Kibana support FIPS 140-3?

Yes. Kibana operates in a Node.js environment configured with a FIPS-compliant OpenSSL 3 provider. Both Elasticsearch and Kibana run within clearly defined cryptographic boundaries when FIPS mode is enabled.

Why isn't a FIPS-hardened OS or load balancer sufficient for FedRAMP compliance?

FedRAMP's network security requirements apply to cryptographic operations at the application layer, not just at the network boundary. Intranode communication, keystore encryption and credential hashing must all occur inside a validated cryptographic module. A compliant perimeter around a non-compliant application layer is an audit finding, not a FIPS deployment.

Is Elasticsearch FIPS 140-3 support available on Elastic Cloud?

Elastic 9.4 FIPS 140-3 support covers self-managed deployments. Cloud deployment availability is not covered in this announcement. Contact your Elastic account team or check the Elastic documentation for cloud FIPS roadmap details.

In most ecommerce deployments, economic signals, like profit margin and product popularity, are either ignored in search ranking or applied as static, global weights. A fixed margin boost might push high-margin products up across every query, which works for "chocolate" (where shoppers are open to suggestion) but backfires for "baby formula" (where shoppers want the trusted, popular brand).

The governed control plane makes it possible to treat economic optimization as a per-query decision, expressed as policy data and managed through the same admin UI as every other governance mechanism. A merchandiser can say "for chocolate queries, prioritize margin" and "for baby formula queries, prioritize popularity", without writing code, without deploying changes, and with full auditability.

For the mathematical foundation of margin and popularity boosting in Elasticsearch, including the logarithmic scaling formula and factor tuning explanation, see Boosting e-commerce search by profit and popularity with the function score query in Elasticsearch.

Two business signals: Margin and popularity

Every product document in our product catalog carries two numeric fields:

• margin: The product's profit margin as a percentage (0 to 200 in our dataset).

• popularity: A relative sales volume metric (0 to 10,000 in our dataset), such as weekly average units sold.

These fields represent two fundamentally different business objectives. Margin optimization pushes profit per transaction. Popularity optimization pushes conversion probability since products that many shoppers buy are products that the current shopper is likely to buy.

The baseline: Global boosting with business signals

Before introducing per-query policy overrides, the system applies a default boost for both margin and popularity. These are implemented using Elasticsearch's field_value_factor with logarithmic scaling inside a function_score query as described in Boosting e-commerce search by profit and popularity with the function score query in Elasticsearch.

The design has three properties worth noting:

• Calibrated range. Each signal's factor is calibrated so that it contributes at most approximately +1.0 to the boost multiplier at the top of its range. Combined with a baseline weight of 1, the final multiplier ranges from 1.0 (a product with zero margin and zero popularity) to approximately 3.0 (maximum margin plus maximum popularity). A product with strong business signals scores roughly 3x higher than an identical product with none, regardless of the BM25 score magnitude.

• Logarithmic scaling. The ln1p modifier grows fast at small values (rewarding incremental gains) but flattens at high values (preventing runaway scores from a single dominant product). This also makes the system resilient to data distribution changes: If the maximum popularity in a dataset shifts significantly, the boost curve stretches rather than breaking.

• Multiplicative, not additive. The business-signal boost is applied multiplicatively against BM25 (boost_mode: "multiply") rather than added to it. BM25 scores vary dramatically across queries, so an additive boost would have inconsistent impact depending on query specificity. Multiplicative scaling guarantees a consistent percentage uplift regardless of the absolute BM25 magnitude.

Per-query boosting overrides through policies

The default weights (1.0 for both margin and popularity) apply to every query. But the governed control plane makes it possible to override these weights on a per-query basis through the same policy engine described in Part 3 and Part 4.

Each policy document has two optional fields: margin_boost_weight and popularity_boost_weight. When a policy matches a query and includes weight overrides, those values flow through to the function_score construction, replacing the defaults.

Why per-query control matters

Consider two queries and why they demand different economic optimization strategies.

Margin boosting: Chocolate

A shopper searching for "chocolate" is browsing. They'll be satisfied by many chocolate-related products. The retailer's store-brand chocolate truffles at 60% margin might be just as appealing as the name-brand bar at 15% margin. Aggressive margin boosting pays for itself if the shopper doesn't care which chocolate they purchase and buys one of the margin-boosted hits.

Chocolate results without margin boosting

To isolate the effect of per-query margin boosting, we first disable margin boosting entirely for this query (margin boost weight: 0). Without any margin signal, the ranking is driven by text relevance. In our dataset, the first hit has a margin of 10 and the next one has a margin of 84 (out of a max of 200) as follows:

Setting a margin boost on queries for “chocolate”

A merchandiser who decides that “chocolate” queries should prioritize margin makes that change in the admin UI, tests it against representative queries, and promotes it to production. The change takes effect on the next query. No engineering ticket, no deployment, no code change. The following "chocolate" policy sets margin_boost_weight: 3.0, which ensures that searches for chocolate will aggressively promote high-margin items.

Chocolate results with margin boosting

With the above margin boost policy enabled, the higher-margin chocolates with a margin of 197 and 184 are boosted to the top of the results as follows (remember that the maximum margin in our dataset is 200):

Popularity boosting: Baby formula

A parent searching for baby formula is not experimenting. They want the product that other parents trust. Pushing a high-margin store-brand formula above the established brand that thousands of parents are buying would feel wrong and erode trust. Popularity is the right signal here because it functions as social proof for a high-stakes purchase.

Baby formula results without a popularity boost

To isolate the effect of per-query popularity boosting, we first disable popularity boosting entirely for this query (popularity_boost_weight: 0). Without any popularity signal, the ranking is driven by text relevance. In this example, the top hit has a popularity of 50 on a scale that goes up to 10,000.

Setting a popularity boost on queries for “baby formula”

A "baby formula" policy sets popularity_boost_weight: 5.0 and margin_boost_weight: 0; formula searches prioritize what's popular, completely ignoring margin.

Baby formula results with popularity boosting

If we enable the above rule, then the most popular baby formula (Lactogen 2 with a popularity of 9979) will be boosted to the top of the results, as shown below.

Disabling business signals: Clearance

Not every query benefits from economic boosting. A shopper searching for "clearance" is looking for deals; margin and popularity are both irrelevant to that intent. A high-margin product is the opposite of what the shopper wants, and a popular product may not be on clearance at all.

A "clearance" policy sets margin_boost_weight: 0 and popularity_boost_weight: 0, which disables both business signals entirely. Results are ranked on pure text relevance with no economic influence. This completes the design space: Policies can amplify either signal independently, rebalance them, or turn them off altogether.

How overrides flow through the control plane

When the percolator returns matching policies, the control plane checks for margin_boost_weight and popularity_boost_weight fields on the highest-priority matching policy. If present, those values replace the defaults in the RewriteState. If no matching policy includes weight overrides, the default values (1.0 for both) are used.

The weights then flow through to the function_score construction when the final Elasticsearch query is assembled. The structure of the function_score doesn't change; only the weight values on the margin and popularity functions.

Weight overrides participate in the same governance model as every other policy mechanism. They’re subject to priority ordering: A Christmas campaign policy with margin_boost_weight: 0.5 will override a product-category policy with margin_boost_weight: 3.0 if the campaign policy has higher priority. The cascading transformation model from Part 3 applies: Economic optimization parameters are just another field in the policy's execution plan.

Interaction with other policies

Per-query weight overrides compose naturally with the constraint enforcement, conflict resolution, and personalization mechanisms described in earlier parts of this series.

Consider a search for "cheap chocolate" during a Christmas campaign, with a shopper who has a purchase history and belongs to a vegan cohort. The control plane processes this query through the full governance stack:

1. The "cheap" policy extracts the price constraint and removes "cheap" from the query.

2. The "chocolate" policy sets margin_boost_weight: 3.0 and constrains results to chocolate categories.

3. The “Christmas campaign” policy (higher priority) overrides the category constraint with seasonal categories and adjusts the price ceiling.

4. The “vegan cohort” policy applies a soft boost to vegan-certified products.

5. The margin and popularity boosts are applied with the governed weights (margin at 3.0× from the “chocolate” policy, popularity at the default 1.0×).

6. The shopper's purchase history boosts are applied as the outermost scoring layer.

Every layer stacks multiplicatively. The economic optimization weights are governed by the same policy framework that controls category constraints, campaign overrides, and cohort-specific boosts. A merchandiser can tune all of these through the admin UI, all without code changes.

This example also illustrates where economic optimization sits in the scoring stack. The layers nest in a deliberate order: the base query (keyword or semantic match), then governance constraints (hard filters and soft boosts from Part 3 and Part 4), then business-signal boosts (margin and popularity with governed weights), and then purchase history personalization (Part 6). Each layer wraps the previous one, and the effects compound multiplicatively. Governance controls what appears. Economic optimization influences what ranks highest from the retailer's perspective. Personalization adjusts ranking further from the shopper's perspective.

Tuning guidance

The factor values in the baseline function_score are calibrated for the demo dataset's field ranges. A production deployment with substantially different ranges for margin or popularity should recalibrate the factors so that each signal contributes a consistent maximum boost. The logarithmic scaling provides built-in resilience to outliers and distribution shifts, but the factors are worth reviewing whenever the underlying data changes significantly. For the calibration methodology, see Boosting e-commerce search by profit and popularity with the function score query in Elasticsearch.

From economic optimization to agentic AI

The governed control plane now handles intent classification, constraint enforcement, conflict resolution, personalization, and economic optimization, all expressed as policy data, all managed through a business-editable admin UI, and all composable through a deterministic transformation framework.

The final post in this series asks what happens when the input to this system isn’t a search string typed by a human shopper, but an intent string extracted by an AI agent, and why the deterministic properties of the governed control plane become even more critical when the upstream decision-maker is probabilistic.

Put governed ecommerce search into practice

The per-query economic optimization described in this post (policy-governed margin and popularity weights composing with governance constraints, personalization, and campaign overrides) was designed and built by Elastic Services Engineering as part of our repeatable ecommerce search accelerators. Contact Elastic Professional Services.

Join the discussion

Have questions about search governance, retrieval strategies, or ecommerce search architecture? Join the broader Elastic community conversation.

From policy logic to retrieval architecture

Part 3 and Part 4 provided a technical deep dive into the governed control plane and its implementation using the Elasticsearch percolator. Once the logic layer has identified which policies to apply, the system must address the retrieval strategy used to execute the search.

Managing the transition from precision to recall is a critical function of any ecommerce search engine. For example, a basic search implementation often defaults to broad keyword matching. If a shopper searches for "organic Pink Lady apples", this can lead to irrelevant results, such as apple-scented dish soap, apple juice, or organic pink grapefruit, appearing at the top of the list simply because they share a common term. While these items are technically matches, they fail to satisfy the user's intent and typically lead to high bounce rates. However, a "No results" page is equally detrimental to conversion. This conflict is resolved by implementing a three-tier execution model, which uses the governed control plane to orchestrate a principled fallback strategy.

The three-tier execution model

This architecture executes up to three retrieval tiers in a sequence, each with a specific matching logic.

Highest tier: Strict matching

Strict matching is a lexical match that requires that all query terms appear in the product metadata.

• The logic: A search for "organic navel oranges" returns only products containing all three terms.

• Application: This tier provides the highest precision. When a customer types a precise product name, such as "organic navel oranges", they’re typically seeking that exact item rather than an alternative.

Mid-tier: Relaxed matching

If the strict tier fails to return sufficient results, the system expands the search parameters.

• The logic: This tier allows for a subset of terms to lexically match, using Elasticsearch's minimum_should_match logic.

• Application: Relaxed matching maintains lexical grounding. A search for "organic navel oranges" might surface "navel oranges" (missing the "organic" term) or "organic oranges" (missing the "navel" term). These represent intuitive, keyword-based alternatives for the shopper.

Lowest tier: Semantic matching

• The logic: This tier uses vector/semantic embeddings (such as Elastic Learned Sparse EncodeR [ELSER], E5, or Jina) to retrieve conceptually related products, regardless of direct keyword overlap.

• Application: A search for "organic navel oranges" might surface "mandarins" or "clementines”. This serves as the final retrieval tier, intended to provide relevant options when literal keyword matches are unavailable.

To see this multi-tier orchestration in action and how the Engine steps down from lexical to semantic matching, watch the video: Eliminating Zero-Result Pages: PRISM’s Multi-Tier Search Fallback.

Tier orchestration: The "bucket filling" logic

While the governed control plane provides the logic and the queries for each tier, the application layer is responsible for the execution. The application executes these tiers sequentially and excludes lower tiers once the accumulated result count on the first page reaches or exceeds 10 items (or whatever number of results you want to display on the first page). This threshold ensures a full first page of results while prioritizing the most accurate retrieval method.

Scenario 1: High-intent search ("oranges")

The first tier returns 15 hits. Since 15 is more than 10, the current result set is locked to only strict matches (which can be paged through) and subsequent tiers are not executed.

Scenario 2: Specific but limited results ("organic blood oranges")

The strict tier finds only four items. Since this is less than 10, the system triggers the relaxed tier, which finds 12 more relevant products. The combined total (16) meets the threshold of 10, so the current result set is locked to the strict and relaxed tiers. Subsequent paging will only surface results from these two tiers (preventing lower-quality semantic hits from appearing on later pages).

Scenario 3: Abstract or intent-based search ("high vitamin C snacks")

Keyword matches are limited (only five hits between tiers 1 and 2). The system triggers the semantic tier to find conceptually relevant items, such as kiwis, guavas, or red peppers, to fill the result set. The result set for this query includes products from all tiers.

This orchestration optimizes for latency, as the computational cost of the semantic tier is only incurred when the keyword-based tiers are insufficient. Additionally, this allows fast-responding keyword results to be displayed while semantic results are integrated shortly after, maintaining a responsive user interface.

Determining intent via tier activation

The logic used to fill the first page serves a critical secondary purpose: It acts as a diagnostic for user intent. The application uses the logic returned by the governed control plane to determine which tiers remain active for the current result set and paging.

If the strict and relaxed tiers together yield fewer than 10 results, the query is likely exploratory or abstract. In this case, activating the semantic tier is a benefit. Because the query is diagnosed as exploratory, the system allows the shopper to page through the entire depth of the semantic results. This provides access to conceptually related alternatives that lexical matching would have missed, which is appropriate for an abstract search.

Conversely, if the strict tier returns a robust set of results (for example, 30 hits), it confirms that the system has found high-precision matches. The user can page through those 30 hits and will likely find what they’re looking for. In this scenario, there’s no need to provide additional, less relevant exploratory hits. By disabling lower tiers for these high-precision queries, we ensure that a shopper deep diving into specific results isn’t distracted by irrelevant semantic fallback as they paginate through the current result set.

Governance across tiers

A critical component of this architecture is that policies apply globally across all tiers. If a user has a "vegan" preference profile, the governed control plane injects that constraint into the strict, relaxed, and semantic queries. This ensures that even when the system uses semantic fallback to return "mandarins" for an orange search, the results remain compliant with the user's broader dietary preferences or business constraints.

The problem of facet instability

A challenge with multi-tier search is maintaining consistent faceted navigation (sidebar filters). If a search for "chocolate" yields 12 strict results, the sidebar filters might show "dark" and "milk". If a user selects "dark" and the result count drops, a naive system might trigger the semantic tier to fill the page, which could suddenly introduce "red wine" into the filters due to a semantic relationship.

The governed control plane identifies which tiers contributed to the initial search and locks the facets to those tiers. This prevents the sidebar from changing unexpectedly during a filtered session, ensuring a stable user experience.

The pagination challenge: Seamless multi-tier paging

Pagination in a tiered system requires precise state management. As established, the first page determines the scope of the current result set. If the first page required semantic results, the user can page through all available results from all three tiers. On the other hand, if the first page was satisfied by high-intent keyword matches, the semantic tier is not retrieved for that specific result set.

The governed control plane manages this through:

• Tier locking: The response includes an array identifying the contributing tiers. The front end returns this on subsequent requests to keep the tier composition consistent across all pages.

• Dynamic offset calculation: The back end calculates an offset based on the requested page and the total products returned in preceding tiers.Example: If the first page has returned seven strict matches and three relaxed matches, a request for page 2 (starting at index 10) would execute a relaxed tier query with an offset of three.

• ID exclusion for lower tiers: The system retrieves IDs from the higher tiers (which, by definition, will always be fewer than the page size threshold) and explicitly excludes them from lower-tier results using an ID-only query (which avoids the overhead of a full fetch phase for excluded items).

Summary

The multi-tier approach ensures search results are precise when data is available and helpful when it is not. By providing a governed fallback sequence for the application to execute, the architecture maintains high relevance while eliminating "no results" scenarios.

What's next in this series

The next posts in this series extend the governed control plane into new territory. Part 6 explores personalization (using purchase history boosting and cohort-aware policies), and Part 7 demonstrates per-query economic optimization. Stay tuned!

Put governed ecommerce search into practice

The search architecture described in this post, where retrieval tiers, economic weights, and governance constraints compose into a single request, was designed and built by Elastic Services Engineering as part of our repeatable ecommerce search accelerators.

To learn more about applying these patterns to your business, Contact Elastic Professional Services.

From architecture to implementation

Part 3 described the control plane architecture: reverse matching as a lookup primitive, policy documents that separate match from action, and cascading transformations that compose multiple policies into a single execution plan. This post goes hands-on with the Elasticsearch feature that powers the policy lookup: the percolator query.

The percolator is a natural fit for governance because it inverts the direction of search in exactly the way a control plane needs. This post walks through the implementation step by step, starting with a clear explanation of what the percolator does and why it matters, and then moving through index design, policy storage, query-time evaluation, and multi-policy composition.

How normal search works

In an ecommerce system, you may have hundreds of thousands or millions of product documents containing fields such as title, category, and price. When a user searches for matching documents, you're asking Elasticsearch to compare the user’s search string against one or more stored fields in these product documents. Elasticsearch's default analyzer, the standard analyzer, lowercases text and splits it into tokens. A search for “oranges” matches “Oranges” because of lowercasing. With a language-aware analyzer that includes stemming, it also matches “orange” because both forms reduce to the same stem. For example, the following match query returns documents that have “orange” or “oranges” in their “title” field.

So for the above query, Elasticsearch returns the product documents whose title field matches “oranges”, which could include results such as “Orange Fruit Spread”, “Orange Juice”, “Juicy oranges”, “Orange Marmalade”, and so on. The key point to remember is that Elasticsearch is commonly used to compare a search string against documents and to return the documents that match the search string.

The governance problem: Finding relevant policies before searching for products

As established in Parts 1 through 3, a governed search system does not send the user's search string directly to the product catalog. First, it checks whether any policies apply to that search string.

A merchandiser has decided that when someone searches for exactly "oranges", results should be restricted to the Oranges category, eliminating orange juice, orange marmalade, and orange soda. That business decision is stored as a policy. When a user types "oranges", the control plane needs to find that policy, read its instructions, and modify the search against the product catalog accordingly. In order to do this, the control plane needs to figure out which stored policies are relevant for this search string.

An enterprise deployment might have hundreds or thousands such policies. Checking them one by one with if/else logic is the application-layer anti-pattern described in Part 2. What we need is a way to store all of those policies in an index and instantly find the ones that match a given search string. This is where the percolator comes in.

Flipping the direction: The percolator

We previously mentioned that in a normal search, Elasticsearch is commonly used to compare a search string against documents and to return the documents that contain that search string.

The percolator inverts this. With a percolator, you have an index where each document stores a query pattern, and then an incoming search string is checked against these stored queries to determine which of these stored query patterns has triggered.

For governance, the "stored query patterns" are policies. Each policy contains a pattern that describes the kind of search string it should match. For example, does the search string exactly match “oranges”, or does the search string contain “olive oil”? The incoming string is the user's search text, which arrives at query time and needs to be checked against all stored policy patterns. This is covered in a related PRISM video at 4:09.

Step by step: How a search for "oranges" finds its policy

The policy

A merchandiser has authored a policy that matches if a user searches for exactly "oranges" without any other words. Once the percolator matches, the remainder of the document includes the rules that the control plane will use to build the Product query; in this example, one of the rules is to restrict (filter) results to the Fruits category.

The percolator field contains the pattern that defines when this policy should fire. In this case, it matches the phrase "START oranges END". The rule_type and rule_args fields define what the policy should do when it fires. The START and END tokens are boundary markers, which we will explain shortly.

You can see how a policy is authored in the PRISM Studio UI at 2:52 of the related PRISM video.

The user searches

A shopper types "oranges" into the search bar.

The control plane checks for matching policies

Before searching the product catalog, the control plane intercepts the user search string, wraps it in boundary markers, and sends it to the percolator:

The string "START oranges END" is checked against all stored policy patterns. Internally, Elasticsearch runs the stored policy patterns against this string and returns the ones that match. That's the percolator. The user's search string was checked against all stored policy patterns, and the ones that matched were returned. No if/else chains. No sequential evaluation. The index handles the matching.

The control plane applies the policy

The control plane reads the matched policies’ actions. The above policy instructs the control plane to restrict results to the Fruits category. The control plane builds the final Elasticsearch query against the product catalog as follows:

The user searched for "oranges”. The product catalog receives a query for "oranges" constrained to the Fruits category. Because of this constraint, orange juice, orange marmalade, and orange soda are excluded.

Why "orange marmalade" does NOT trigger the oranges policy

Suppose a different user searches for "orange marmalade”. The control plane wraps the string and percolates: "START orange marmalade END". The oranges policy's pattern is match_phrase: "START oranges END". The oranges policy does not match and therefore the policy isn’t applied, and the results aren’t constrained to the Fruits category.

This is the purpose of the START and END boundary markers. Without them, a policy that matches on the word "oranges" could accidentally fire on a query like "orange marmalade". By wrapping the user's search string with START and END and including those markers in the policy's pattern, we ensure that the policy only fires when "oranges" is the complete search string, without any other words. This matches both the shoppers and the merchandiser's intent.

A second policy: "olive oil" on the stemmed field

Not every policy needs an exact string match. The “olive oil” policy matches on a stemmed field, so it fires regardless of minor word-form variations:

This policy's pattern matches against query.stemmed instead of query. When the user's search string arrives, it’s stored in both a query field (the exact text) and a query.stemmed field (analyzed with a stemming analyzer that reduces words to their stems, so "olives" and "olive" both reduce to the same stem, as do "oils" and "oil"). The policy's pattern is checked against the stemmed version of the string, so it fires regardless of minor word-form variations.

The START and END boundary markers work on the stemmed field, as well, ensuring this policy only fires when "olive oil" is the entire search string, not when it appears as part of something longer.

The rest of this post covers the implementation details that make this production-ready: the index mapping that supports both matching modes, how highlights drive phrase removal and consumed phrase tracking, and how multiple conflicting policies compose into a single execution plan.

The policy index mapping

The policy index needs a percolator field to hold stored query patterns and a text field that mirrors the structure of the incoming search string the percolator will match against. The mapping below is simplified for clarity. A production deployment is more complex, using custom analyzers to handle boundary markers, variable pattern matching (for example, recognizing that "under $4" contains a currency value), and other kinds of analysis.

The index is named policies because each document represents a complete governed policy as defined in Part 2. This includes match criteria, action, priority, and metadata. The rule_type and rule_args fields contain the action component of the policy, which contain the instructions that the control plane will use to compose the query for execution against the product catalog.

The query field is the string that the percolator matches against. It has two variants: an exact version and a stemmed version. When the user's search string arrives, it’s placed into this field in the temporary in-memory index. Policies that match on query see the exact string; policies that match on query.stemmed see the stemmed version.

Percolating with highlights, filtering, and sorting

The simple examples above showed minimal percolation requests. In practice, the control plane adds highlighting, filters disabled policies, and sorts by priority:

The highlight configuration uses "query" as the field key with "query.stemmed" in matched_fields. This tells Elasticsearch's unified highlighter to return highlights on the parent query field but to also consider matches from the query.stemmed subfield when determining which tokens to highlight. This is what allows a policy that matches on the stemmed field to still produce accurate highlight spans on the original text, which the control plane needs for phrase removal and consumed phrase tracking.

The enabled: true filter ensures that disabled policies are skipped. The sort on priority ensures that higher-priority policies are returned first, so the control plane can process them in the correct order for cascading transformations. The highlight field is the most important addition; it tells us exactly which words in the user's search string triggered each match.

The response for an "olive oil" search may look as follows:

Why highlights matter

Notice the highlight in the response: "<em>START olive oil END</em>". Elasticsearch is telling us exactly which words in the user's search string caused the policy to match. This isn’t cosmetic. The highlight metadata drives two critical downstream behaviors:

Phrase removal. Some policies need to remove the matched text from the search string before constructing the product catalog query. For example, a policy that matches on "cheap" removes that word and converts it into a price filter instead. The highlight identifies exactly which span of the search string the policy matched, so the system knows what to remove.

Consumed phrase tracking. As described in Part 3, when multiple policies match the same search string, a higher-priority policy might remove words that a lower-priority policy also matched on. By comparing each policy's highlight against the current (evolving) search string, the system can detect that a phrase has been consumed and skip the lower-priority policy. This prevents double-processing and ensures deterministic behavior.

You can learn more about how highlighting works in this article.

From percolation to execution plan

The percolator returns a set of matching policies. But as Part 3 described, the lookup is only half the story. The other half is composing those matches into a coherent execution plan. Here’s what that looks like for a concrete query.

Worked example: "Cheap chocolate" during a Christmas campaign

Suppose the system has two active policies: the "Cheap chocolate" policy (priority 210) and the "Christmas chocolates" policy (priority 300), both described in detail in Part 3.

Step 1: Percolate. The user searches for "cheap chocolate." The control plane wraps the search string as "START cheap chocolate END" and sends it to the percolator. Two policies match: The "Cheap chocolate" policy's pattern matches on the phrase "cheap chocolate"; and the "Christmas chocolates" policy's pattern matches on "chocolate" via the stemmed field.

Step 2: Sort by priority. The percolator returns both policies, sorted by priority in descending order. The “Christmas chocolates” policy (300) is processed first, followed by the “Cheap chocolate” policy (210).

Step 3: Apply the cascading transformation. This is the initial state → [Policy A] → state' → [Policy B] → state'' → execution plan model from Part 3.

The “Christmas chocolates” policy (priority 300) applies first:

• Adds a category hard filter: "Christmas foods and drinks," "Christmas sweets".

• Adds a price filter: less than $7.

• Adds a category soft boost: "Advent calendars" (3x).

The “Cheap chocolate” policy (priority 210) applies next against the modified state:

• Attempts to add a category hard filter: "Chocolates," "Milk chocolates"; but the Christmas policy already set this field with on_conflict: override, so the Cheap chocolate categories are dropped.

• Attempts to add a price filter: $2, the Christmas policy set on_conflict: restrict for price, and $2 is more restrictive than $7, so $2 wins.

• Removes "cheap" from the search string.

Step 4: Build the Elasticsearch query. The control plane assembles the execution plan into a single Elasticsearch query against the product catalog:

The original search string was "cheap chocolate”. The query that reaches the product catalog is a governed, intent-aware retrieval plan: The word "cheap" has been consumed and converted into a price constraint, results are restricted to Christmas seasonal categories, Advent calendar products receive a ranking boost, and the price ceiling reflects the more restrictive value from the lower-priority policy. Every transformation is deterministic, traceable, and explainable.

For a quick overview about how these multipliers interact with the base BM25 score, see 8:45 in the related PRISM video, where we briefly discuss multiplicative boosts.

Why this scales

The percolator is efficient for this use case because of the asymmetry: An enterprise ecommerce system might have millions of products but only hundreds or thousands of governance policies. The percolator is checking one incoming search string against that set of stored policy patterns, not scanning the full product catalog. The cost is proportional to the number of policies, and Elasticsearch applies internal optimizations (indexing terms from stored query patterns, short-circuiting Boolean logic) to keep matching fast.

Adding a new policy is just indexing a new document. Disabling one is a field update. No code changes, no deploys, no restarts.

From lookup to governed retrieval

The percolator provides the fast reverse-matching primitive that makes the control plane architecture from Part 3 practical at scale. Policies are data which are stored and indexed, and efficiently matched against incoming search strings. The control plane composes matching policies into a governed execution plan through the cascading transformation and per-field conflict resolution described in Part 3. And the retrieval engine executes the governed execution plan against the product catalog.

The result is a system where a merchandiser can author a new policy without touching application code, test it against representative queries, promote it to production, and immediately see the effect. The percolator makes the policy lookup fast; the control plane makes the policy composition deterministic; and the governed workflow makes the whole process safe.

What's next in this series

The next post in this series extends the governed control plane into new territory. It introduces a multi-tier search architecture, explaining how to orchestrate strict, relaxed, and semantic retrieval while maintaining stable pagination and facets.

Put governed ecommerce search into practice

The percolator-based control plane described in this post, from index mappings and boundary markers to highlight-driven phrase tracking and cascading policy composition, was built by Elastic Services Engineering as part of our repeatable ecommerce search accelerators. Every query example and policy structure shown here comes from a working system validated against enterprise-scale product catalogs.

If you want to implement a governed, policy-driven control plane on Elasticsearch, Elastic Services can get you there faster. Contact Elastic Professional Services.

Join the discussion

Have questions about search governance, retrieval strategies, or ecommerce search architecture? Join the broader Elastic community conversation.

Why query interpretation is often a challenge

Storing policies as code (if/else blocks in the application layer) produces tens of thousands of lines of brittle logic that lacks any indexing for efficient policy retrieval at query time. Iteration is slow (a single query behavior change may require a six-week deployment cycle), accountability is unclear (why did results change?), and business users cannot modify search behavior without engineering involvement. This is shown on the left side in the following image:

Storing policies as data in an Elasticsearch index is shown on the right side of the above image. This approach solves all of the issues associated with hard-coded query resolution logic. However, for this to work, you need a way to quickly determine which policies match the user query and how conflicts should be resolved. This is where the governed control plane comes in.

The control plane pattern

A governed control plane sits between the raw user query and an Elasticsearch retrieval. It receives user text as its input, and its output is an execution plan that includes filters, boosts, and retrieval routing decisions.

A control plane pipeline consists of:

1. User query: A user enters a string of what they’re looking for, such as “oranges” or “gift for grandpa”.

2. Policy lookup: Match the user query against the policy index.

3. Return matching policies: Policies that match the user query are returned from the policy index.

4. Policy application: The control plane analyzes these returned policies and composes matched policies into a single coherent execution plan that includes filters, boosts, overrides, and guardrails and that applies the appropriate retrieval method (for example, lexical versus semantic versus hybrid).

5. Execute: The modified intent-aware Elasticsearch query is passed to the application to be executed against a product catalog index.

6. Explain (optional): In addition to creating a query that provides business and intent-aligned results, the control plane provides an optional explainability payload to show which policies were triggered and how they were combined.

Finding which policies should be applied for a user’s search string requires a fast reverse-matching primitive, which we solve with the percolator query. After retrieving relevant policies, combining multiple matched policies into a unified execution plan requires a judgment framework: priorities, conflict strategies, consumed phrase tracking, and cascading transformations that apply policies in sequence rather than independently. Additionally, the most appropriate retrieval technology needs to be selected (for example, BM25 for “oranges” versus semantic search for “gift for grandpa”).

Policy lookup: Checking the query before searching for products

When a shopper types a query, a search system with a governed control plane doesn’t send that query directly to be executed against the product catalog. First, the query is checked against a set of stored policies and modified to reflect the intent of the query and business priorities.

Policy structure

Each policy is a simple document that defines two things:

• Match criteria: What query text should cause this policy to fire. This could be an exact phrase, a single word, a pattern, or a combination.

• Action: What to do when the policy fires. This could be applying a category filter, excluding products, extracting a price constraint, or changing the retrieval strategy.

The system finds all matching policies, composes them into an execution plan, and only then runs the product search. Taken together, policies act like a knowledgeable store associate who understands what you’re looking for and walks you to the right aisle.

The policy pattern

The first articles in this series introduced examples of policies in action: constraining "oranges" to the produce category, treating "without peanuts" as an exclusion, and routing "gift for grandpa" to semantic retrieval. The key architectural point is that in each case, the query is checked against stored policies before the product search begins. The policies determine what constraints to apply, which text to modify, and which retrieval strategy to use. The query against the product catalog comes after the policies have been applied and a new rewritten query has been created.

Why this is fast

An enterprise ecommerce system might have millions of products but only hundreds or thousands of policies. The policy lookup step is searching against a small curated index, not the full product catalog, and is therefore fast. And because policies are stored as data in their own index, a merchandiser adding a new policy doesn't touch the application code, and an engineer optimizing the product search doesn't touch the policy index. The two concerns evolve independently.

The examples above describe what happens conceptually. Under the hood, the policy lookup is implemented using the Elasticsearch percolator query type, which is purpose-built for this kind of pattern: matching incoming text against a set of stored queries. Part 4 in this series provides a hands-on deep dive into the percolator implementation, including index mappings, boundary markers, and highlight-driven phrase tracking. With the lookup mechanism covered in depth in Part 4, let's turn to what a policy document actually contains and how the control plane composes multiple policies into a single execution plan.

Example policies

Now that we've seen what policies do conceptually, let's look at what they actually contain. The two policies below have been designed to intentionally conflict, which will demonstrate the conflict resolution system described in subsequent sections.

Cheap chocolate

The policy shown below detects if a user has submitted a search containing the phrase “cheap chocolate”. If so, results are restricted to the “Chocolates” and “Milk chocolates” categories. This policy also applies a price filter of $2. Also, notice that this policy has a priority of 210; we’ll come back to this when we discuss conflict resolution in more detail.

The filter mode and conflict strategy settings shown here (hard_filter, soft_boost, restrict, override) are explained in detail in the conflict resolution section below.

When the above policy is activated, a search for “cheap chocolate” respects the price filter of $2 and restricts results to the “Chocolates” and “Milk chocolates” categories. Example results are shown below:

Christmas chocolate

The policy shown below is an example of a policy that one could imagine applying at Christmas. This example restricts results to “Christmas foods and drinks” and “Christmas sweets”, boosts any products that are also in the “Advent calendars” category, and applies a price filter of less than $7 to focus on affordable seasonal items. Additionally, notice that this policy has a priority of 300. We’ll come back to this when we discuss conflict resolution in more detail.

When the above policy is activated without any conflicting policies, a search for “chocolate” respects the price filter of $7, and restricts results to the “Christmas food and drinks” and “Christmas sweets” categories, and boosts any products tagged as “Advent calendars”. Example results are shown below:

Combining matched policies

The policy lookup described above is half the story. The other half is what happens when multiple policies match the same query.

In any nontrivial deployment, a single query will routinely trigger several policies at once. "Cheap chocolate" will match both of the policies that we demonstrated above. Each policy is correct in isolation. The challenge is composing them into a single, coherent execution plan without contradictions, without double-counting, and without one policy silently undoing the work of another.

This isn’t a lookup problem; it’s a judgment problem. The system must decide:

• Order of application: If a negation policy removes "without peanuts" from the query, does the price policy still see the original text or the modified text?

• Filter conflicts: If two policies set different price ceilings, which one wins? Is the loser silently dropped, or does it degrade gracefully into a soft boost?

• Phrase ownership: If two policies both matched on the same word and the first one already consumed it, should the second one still fire?

A naive implementation (apply all matched policies independently, merge the results) breaks as soon as policies interact. The architecture needs an explicit model for how policies compose. The next two sections describe that model: a priority and conflict resolution framework; and a cascading transformation model that makes policy interaction deterministic.

The key insight is that policy application isn’t a set of independent operations; it’s a cascading transformation. Each policy receives the rewrite state produced by all higher-priority policies and transforms it further:

initial state → [Policy A] → state' → [Policy B] → state'' → ... → execution plan

The state carries the rewritten query text, accumulated filters, current intent, and any synonym expansions. A high-priority policy can remove text from the query, and every subsequent policy sees the modified query, not the original. Context accumulates. Order matters.

Precedence and conflict resolution: Determinism matters

The specific conflict strategies are a design choice. Different organizations may resolve conflicts differently, depending on their business requirements. The following approach illustrates the kind of judgment framework a control plane needs. The important thing is not these specific strategies but that the system has explicit, deterministic strategies rather than letting conflicts resolve through unpredictable interactions.

Priority ordering

Policies are sorted by priority (highest first). When multiple policies match the same query, they’re applied in priority order. If two policies try to set the same filter field, the higher-priority policy's declared strategy for that field takes precedence. If there are multiple policies triggered that have the same priority, then the policy with the highest ID is given precedence (as if it were assigned a higher priority); this choice ensures deterministic behavior when conflicts arise.

Per-field resolution, not per policy

A critical design principle: Conflict resolution operates per field (for example, brand, category, or description), not per policy. When two policies produce filters that overlap on specific fields, only those specific fields are affected by the conflict resolution strategy, and the resolution strategy is defined by the highest-priority matching policy. Non-conflicting fields from both policies survive intact.

This matters because the alternative of a per-policy approach would force the system to either accept or reject an entire policy when only one of its fields conflicts.

Per-field resolution preserves the maximum amount of useful constraint information.

Three settings per filter field

Each filter field in a policy has three independent settings:

Filter mode: How the filter is applied when there’s no conflict.

• hard_filter (default): Applied as an Elasticsearch bool.filter clause. This is useful for excluding unrelated products entirely. For example, restricting a search for "oranges" to the produce category eliminates hits such as orange juice and orange marmalade. Non-matching documents are completely excluded from results.

• soft_boost: Applied as an Elasticsearch function_score weight with a configurable boost_weight. Documents that match get a ranking boost, but non-matching documents aren’t excluded. This is useful for something like boosting a brand, without excluding other brands.

Conflict strategy

What happens when a lower-priority policy sets the same field:

• override: This high-priority policy's value wins; the lower-priority value is dropped entirely. Valid for all field types.

• restrict: Take the more restrictive numeric value (for example, the lower ceiling for price__max, the higher floor for price__min). Valid for numeric range fields only.

• merge: Combine both values into a union. Valid for non-numeric fields only.

• soft_boost: Convert the conflicting filter to a function_score weight with a configurable boost_weight instead of a hard filter. For more details on function_score boosting, see Influencing BM25 ranking with multiplicative boosting in Elasticsearch. This is only valid for non-negation fields.

Value: The actual filter value (for example, a categories list, a price threshold).

Strategies by field type: Not all strategies make sense for all field types. For instance, an exclusion is inherently binary, so it cannot be soft-boosted. The following table shows which strategies are available for each field type:

Field type

Available strategies

Default

Negation fields (__not, __match__not)

override, merge

override

Numeric range fields (__max, __min, __gt, __lt)

restrict, override, soft_boost

restrict

All other fields (keyword, text)

soft_boost, override, merge

soft_boost

Negation fields cannot be soft-boosted because exclusions are binary. Converting "never show canned foods" to "slightly prefer not-canned-foods" fundamentally changes the semantics; a product from "canned foods" would still appear, just ranked slightly lower, which defeats the purpose of the exclusion.

A concrete example: Searching for "cheap chocolate" during a Christmas campaign

Suppose a merchandiser has created the two policies for chocolate that we previously demonstrated, a lower priority one for cheap chocolate and another higher-priority chocolate-related policy that will be enabled during Christmas. If both of these policies are enabled, then how these are combined depends on the filter mode and conflict strategy of the higher-precedence policy. If both of the previously discussed policies are enabled, they’ll be combined as follows:

This shows two conflicts, one on categories and one on price. It’s worth noting that the query that will be executed after this transformation has the following characteristics:

• Only products from the “Christmas foods and drinks” and “Christmas sweets” categories will be shown.

• Within those categories, if the products are also tagged as being in the “Advent calendars” category, they’ll be boosted up by 3x.

• A price filter for $2 is applied, which came from the lower-priority policy (because the higher-priority policy specified to “Restrict” on conflict).

• The word “cheap” is removed, only returning products matching “chocolate”.

With both of these policies enabled, “cheap chocolate” returns results similar to the image shown below:

Relaxing constraints

Perhaps the retailer doesn’t want to exclude products in the categories of “Chocolates” and “Milk chocolates” during Christmas. The settings on the Christmas policy might have overreached and inadvertently removed categories applied by the “cheap chocolate” policy. This is an example that shows why it might be more desirable to combine lower-priority policies with conflicting higher-priority policies. For example, we could modify the Christmas chocolates promotion so that instead of “Override” on conflict, we do a soft boost. The change to that policy would be as follows:

After this modification, the query rewriter transformation pipeline execution for “cheap chocolate” looks as follows:

With the soft boost on conflict, the conflicting filters are converted into soft boosts rather than being dropped. The query that will be executed on the product catalog after this transformation has the following characteristics:

• Because “On conflict” is specified as “Soft boost” on the higher-priority policy, the conflicts will be converted to boosts as follows:

  • Products from the “Christmas foods and drinks” and “Christmas sweets” categories will have a boost of 1x applied to them.

  • Products from the “Chocolates” and “Milk chocolates” categories will have a boost of 3x applied to them.

• As in the previous example, if the products are also tagged as being in the “Advent calendars” category, they’ll be boosted up by 3x.

• As in the previous example, a price filter for $2 is applied.

• The word “cheap” is removed, only returning products matching “chocolate”.

With relaxed filtering, results look as follows:

Overriding price from a high-priority policy

Or perhaps the retailer wants to allow slightly more expensive chocolates to be shown during Christmas by increasing the price max to $7. To ensure that the max price from the Christmas chocolates policy is not overridden if someone searches for “cheap chocolates”, we can set the conflict mode on the price to “override” rather than “restrict”, as follows:

With this override, the query for “cheap chocolate” ignores maximum price that is defined in the “cheap chocolate policy” and only applies the price specified in the “Christmas chocolates” policy, as follows:

This is similar to the previous example, with the difference being that the max price is set to the $7 value from the higher-priority policy because that policy specified “Override” on conflict. With the Christmas price filter taking precedence, the results look as follows:

These three variations (override, soft_boost, and override on price) demonstrate a key property of the system: A merchandiser can change how two policies interact by modifying a setting on a single field within a single policy, without deploying any code. The conflict strategy is the lever that controls business behavior.

Consumed phrase tracking

There’s a subtler form of conflict: two policies that match on the same phrase. If a higher-priority policy removes "without peanuts" from the query, a lower-priority policy that also matched on "without" has nothing left to act on. The system detects if the matched phrase is no longer present in the rewritten query and skips the lower-priority policy.

Intent policies are exempt from consumed phrase tracking: They set the retrieval strategy based on the original query match, regardless of what text has been removed by higher-priority policies.

Priority ordering, per-field conflict resolution, and consumed phrase tracking together give the control plane a deterministic composition model. With that foundation in place, the system can make a routing decision that would be risky without it.

Governance makes retrieval strategy safe

An important insight about routing to the correct retrieval method (text, semantic, or hybrid) is that it executes after governance. If your policies have already enforced "produce category”, then semantic retrieval becomes far less risky because the candidate set is constrained. A semantic search over 500 product items is a very different proposition from a semantic search over 500,000 SKUs. Governance narrows the blast radius before retrieval begins.

For example, without governance, a semantic query for “Fruit high in vitamin C under $4”, in addition to fruits, might return vitamin bottles, carrots, and green pepper. The control plane ensures that these undesired results aren’t even considered as part of the semantic expansion.

With that constraint in place, the control plane applies pragmatic routing logic:

• Lexical for navigational and head queries where deterministic precision matters.

• Semantic for descriptive discovery queries where concept matching helps.

• Hybrid selectively, when constraints have already been enforced and the business accepts broader recall.

From architecture to implementation

The governed control plane translates business intent into deterministic, composable execution plans, without embedding that logic in application code. Policies are data: matched at query time, resolved through explicit per-field conflict strategies, and applied as cascading transformations that produce explainable results. Elastic Services Engineering has built and deployed this architecture for enterprise ecommerce teams, using repeatable patterns and accelerators that compress the path from concept to production. You can see a demo of our implementation of a control plane on YouTube at: Fixing Search Relevance in Seconds: Introducing PRISM.

What's next in this series

The next post goes hands-on with the implementation: how the Elasticsearch percolator powers the policy lookup, including index mappings, boundary markers, highlight-driven phrase tracking, and concrete query examples.

Put governed ecommerce search into practice

The control plane architecture described in this post (per-field conflict resolution, cascading policy transformations, and governance-constrained retrieval routing) was designed and built by Elastic Services Engineering. Every pattern, screenshot, and transformation pipeline shown in this series comes from a working system built by Elastic Services Engineering and validated against enterprise-scale product catalogs.

If you want to implement a governed, policy-driven control plane on Elasticsearch, Elastic Services can get you there faster.

Join the discussion

Have questions about search governance, retrieval strategies, or ecommerce search architecture? Join the broader Elastic community conversation.

This post answers those questions. A governed control plane doesn't just improve search relevance; it changes the operating model. It moves search behavior changes from engineering deployment cycles to business-driven workflows, without sacrificing safety or accountability.

The scenario that exposes the operating model

Imagine that you’re in the weeks leading up to Christmas, and your merchandising team has identified three urgent changes that must immediately be made to search behavior:

• Campaign launch. Due to an ordering error, there’s an oversupply of in-house branded turkeys. Therefore any query for "turkey" must boost the in-house brand.

• Product recall. A supplier has recalled a product line. Queries that would surface those products shouldn’t be shown.

• Seasonal reinterpretation. Queries for "stocking" are returning women's hosiery and tights. During the holiday season, "stocking" should resolve to Christmas stockings and stocking stuffers. Once the season ends, the policy can be reverted in minutes.

Under the traditional operating model, where search logic is embedded in application code, each of these changes requires an engineering ticket, a code change, a review cycle, a staging deployment, and a production release. In organizations with conservative release processes, that's a timeline measured in weeks, not hours or minutes. The Christmas shopping window closes before engineering can ship the necessary modifications.

The bottleneck isn’t the retrieval engine; it’s the operating model. The core challenge is that business intent cannot be translated into search behavior without engineering acting as a constant intermediary, turning every strategic pivot into a technical ticket.

The anti-pattern: Search logic in application code

Part 1 described how search logic embedded in application code can turn into a "spaghetti" implementation, which creates operational friction. Here’s what that friction looks like at scale. What starts as a few targeted overrides, a filter here, a boost there, grows over time into tens of thousands of lines of if/else branching, regex patterns, and conditional query modifications. This creates problems beyond just technical debt:

This model introduces four systemic frictions that hinder both organizational speed and system scalability:

Coupling. Business strategy changes daily. Application infrastructure should remain highly stable. When both live in the same codebase, a merchandiser's request to boost a seasonal product becomes a deployment risk, and a scoring function refactor can silently break a campaign.

Latency (organizational and computational). A single query behavior change can require a six-week deployment cycle: ticket, investigation, code change, review, staging, release. Furthermore, the application layer lacks any indexing mechanism to efficiently determine which policies apply to a given query, so policy evaluation often adds meaningful latency at query time as the system walks through sequential if/else checks.

Accountability gaps. When results change unexpectedly, nobody can quickly answer why. Was it a synonym update? A scoring change? A new filter added three releases ago? When business logic is distributed across thousands of lines of application code, shipped by different teams across different releases, tracing a relevance change back to its root cause becomes an archaeology project.

Misallocated engineering. This model turns skilled software engineers into full-time relevance mechanics. Instead of building platform capabilities, they spend their cycles translating merchandising requests into code changes and debugging interactions and conflicts between hard-coded business policies.

The paradigm shift: Policies as data

The solution is to decouple business policies from application code entirely. Instead of hard-coding query modifications in middleware, store governed policies as structured documents, each one expressing a discrete business intent, and evaluate them at query time in a dedicated governed control plane layer.

A policy is a first-class data object. It has match criteria (when should this policy fire?), an action (what should it do?), a priority (how does it interact with other policies?), and metadata (a title and a description). The control plane evaluates matching policies, resolves conflicts deterministically, and produces an execution plan including constraints, boosts, and routing decisions that Elasticsearch executes against a product catalog.

For each additional search requirement, the application code doesn't change. The retrieval engine doesn't change. What changes is that business decisions are no longer encoded in code. They live in a policy index as data that can be updated without a deployment.

This changes your org chart, not just your query.

Policies vs. triggers vs. rules

A note on terminology used in this series: a policy refers to this complete governed document, including a trigger (match criteria), rule (action), priority, enabled/disabled, and metadata. A trigger refers to the matching criteria that determines when this policy fires, and a rule refers specifically to the action inside the policy, such as applying a filter or changing the retrieval strategy.

The workflow: Author → Test → Promote

Moving policies out of code and into data opens the door for business-driven search management. But enabling non-technical teams to alter search behavior requires strict operational guardrails. The goal is fast and safe iteration with governance.

To empower non-technical teams to modify search behavior with confidence, we suggest a three-stage workflow: Author, Test, and Promote. Let’s examine the components of this workflow in detail.

Author. A merchandiser creates a policy using structured fields: what the policy should match, what action it should take, and at what priority. The interface guides the business user through what’s expressible.

Elastic Services has built and deployed a governed framework for enterprise ecommerce customers, which has an admin UI that looks as follows:

Test. The policy is validated in a non-production environment where the merchandiser can run representative queries and verify that the policy produces the expected behavior, including how it interacts with other active policies. Because the control plane infrastructure is identical across environments, what works in the test environment will work in production.

Review. Before a policy is promoted to production, it passes through review. Depending on the organization's risk tolerance, this might be a peer review from another merchandiser, an approval from a search lead, or an automated validation that checks for conflicts with existing policies.

Promote. Once approved, the policy is promoted to the production policy index. It takes effect on the next query: no code deployment, no engineering release, no staging build. The entire promotion is a data operation: the same JSON document, moved to a different index.

Disable. If a production policy produces unexpected behavior, it can be disabled immediately without engineering involvement. Disabling removes the policy from query evaluation instantly, without affecting any other policy in the system.

This is the "zero-deploy" promise. It doesn't mean "no process." It means the process operates on policy data, not application code. This distinction compresses the change cycle from weeks to hours or minutes.

Why "zero-deploy" matters for revenue-critical queries

The economics of ecommerce search are asymmetric. A small number of high-volume queries ("milk," "bread," "oranges," "diapers") drive a disproportionate share of revenue. When one of these queries returns unexpected results, the cost is immediate and measurable: Conversion drops, customer complaints spike, and the merchandising team opens an urgent ticket.

Under the traditional model, the response cycle is:

1. The merchant notices the problem.

2. The merchandiser files a ticket with engineering.

3. Engineering investigates, identifies the cause, and writes a fix.

4. The fix goes through code review, staging, and release.

5. Production is updated.

Depending on the organization, steps 2 through 5 may take weeks. For a revenue-critical query during a peak sales period, that latency costs money.

Under a governed control plane, the response cycle compresses:

1. The merchant notices the problem.

2. The merchandiser drafts a policy fix (or modifies an existing policy).

3. The policy goes through review and is published.

4. The fix is live.

The difference isn't just speed. It's ownership. The person closest to the business context (the merchandiser who understands why "oranges" should resolve to produce, not beverages) is the person making the change. Engineering is freed from the daily merchandising loop to focus on the platform. This shift also unlocks something that's nearly impossible under the traditional model: attributing search performance changes to specific business decisions.

Measurability: Which policy moved conversion

When policies are discrete, versioned documents that are stored in an Elasticsearch index, each one becomes independently deployable and therefore its impact can be more easily measured. You can answer questions that are nearly impossible to answer when business logic is scattered across application code:

• Did the "cheap laptops" price threshold policy improve conversion for that query class, or did it suppress it?

• What was the click-through rate impact of the holiday campaign boost?

• When we rolled back the "oranges" category constraint last Thursday, what happened to add-to-cart rates?

This turns search governance into a data-driven discipline. Instead of vague "relevance tuning," where a release contains a dozen changes and nobody can attribute the outcome, you get measurable, attributable impact per policy. Merchandisers can iterate with evidence. Engineers can evaluate whether a policy schema change produced the expected downstream effect. Leadership can see which policies are driving revenue and which are inert.

What this means for each role

For merchandisers and business users

Search behavior becomes something you can directly influence through structured policies without understanding Elasticsearch syntax or scoring algorithms. You can see what policies are triggered for a given query to understand why it produces specific results, and make changes within hours instead of weeks. The same policy mechanism also supports sponsored product placement: A merchandiser can create a boost policy that elevates a product or brand and flags it for a 'Sponsored' indicator in the UI, without requiring engineering involvement or additional infrastructure.

For search engineers

The control plane separates two concerns that are currently entangled: retrieval optimization and business logic. Instead of maintaining tens of thousands of lines of application code that encodes business decisions, you maintain the retrieval engine and the control plane infrastructure. When a merchandiser needs a new campaign boost, they don't need engineering to write it.

This doesn't eliminate engineering involvement. Engineers design the policy schema, maintain the control plane, set guardrails on what policies can express, add new capabilities as required, and handle edge cases that fall outside the policy framework. But the day-to-day operational cadence of modifying query behavior shifts to the people who own the business context.

For site reliability engineers and platform teams

Because policies are structured documents rather than application code, they fit naturally into existing operational workflows. Policies can be stored in version control, reviewed through pull requests, and deployed through the same continuous integration and continuous deployment (CI/CD) pipelines the team already uses. Conflicts between policies are detected and resolved deterministically at query time through the control plane's priority system, not through unpredictable interactions between code branches shipped in different releases.

When something does go wrong, diagnosing the cause is straightforward: Policies are discrete, named, and individually toggleable. A problematic policy can be disabled or deleted immediately without affecting anything else in the system. Compare that to debugging a relevance regression caused by an interaction between a synonym update, a scoring function change, and a new analyzer, all shipped in the same release with no clear attribution.

Beyond manual authoring: Large language model–assisted (LLM-assisted) policy suggestions

The policies described so far are authored by humans (a merchandiser identifying a gap and drafting a fix). But the same governed workflow supports a second mode: LLM-assisted policy suggestion.

An LLM can run offline or in the background, analyzing query logs, identifying patterns where search results underperform, such as queries with high exit rates, low click-through, or frequent reformulations. An LLM can then suggest new policies that enter the same Author → Test → Promote pipeline, where a human evaluates each one before it reaches production.

Governance is the enabler, not the constraint

It might seem counterintuitive: Adding a governance layer makes the system faster to change, not slower. This is the same pattern that works in other domains. CI/CD pipelines don't slow down software delivery; they make it safe to ship frequently. Access control doesn't slow down collaboration; it makes it safe to share broadly.

A governed control plane works the same way. The reason a query behavior change takes six weeks isn't that the code change is complex; it's that nobody is confident enough to ship it faster, because the blast radius is unclear and the rollback path is uncertain.

Governance provides that confidence. When every policy is explicit, every conflict is resolved deterministically, and every change can be instantly disabled and then rolled back (because policies are structured JSON documents that can be version controlled using existing workflows), the cost of iteration drops dramatically. Business teams move at the speed of the market. Engineering focuses on the platform.

From operating model to architecture

The shift from business logic in code to business policies as data is more than a technical refactoring; it's an organizational change that puts relevance ownership with the teams closest to the business context. But it raises an architectural question: How do you evaluate policies at query time without adding latency or turning the control plane itself into a new form of spaghetti?

The next post will dig into exactly that: the design pattern that enables fast, deterministic policy evaluation at query time.

Put governed ecommerce search into practice

The workflow described here, merchandisers authoring, testing, and promoting search policies without engineering deployments, is already available. Elastic Services Engineering designed and built it, and Elastic Services has the skills to deploy it for enterprise ecommerce teams.

If your organization is ready to move from deployment-gated relevance tuning to business-editable search with governance and auditability, we can accelerate your implementation. Contact Elastic Professional Services.

Join the discussion

Have questions about search governance, retrieval strategies, or ecommerce search architecture? Join the broader Elastic community conversation.

This blog's content was developed and verified using Elastic versions 9.2.8 and 8.19.14, along with Filestream Integration versions 2.3.0 and 1.2.0.

Important note: Depending on your environment, some steps may require specific modifications. Furthermore, be aware that dynamic templates were removed from the @package component template starting with Filestream Integration version 2.3.3.

Before starting the reindexing process, it’s important to consider the current storage allocation in your environment. The steps outlined below involve creating a copy of the existing backing index, which will temporarily reside in the hot tier.

Elasticsearch data tiers

• Hot: The hot tier is the Elasticsearch entry point for time series data, storing the most recent, frequently searched data. Hot tier nodes require fast reads and writes, necessitating more resources and faster storage (SSDs). This tier is mandatory, and new data stream indices are automatically allocated here.

• Warm: Time series data can move to the warm tier once it’s being queried less frequently than the recently indexed data in the hot tier. The warm tier typically holds data from recent weeks. Updates are still allowed but are likely infrequent. Nodes in the warm tier generally don’t need to be as fast as those in the hot tier. For resiliency, indices in the warm tier should be configured to use one or more replicas.

• Cold: Data that’s infrequently searched can move from the warm to the cold tier. The cold tier, while still searchable, prioritizes lower storage costs over search speed. Alternatively, the cold tier can store regular indices with replicas instead of searchable snapshots, allowing use of less expensive hardware for older data without reducing disk space requirements compared to the warm tier.

• Frozen: Data that’s queried infrequently or no longer queried moves from the cold to the frozen tier for its remaining lifecycle. This tier uses a snapshot repository and partially mounted indices to store and load data, reducing local storage and costs while still allowing search. Searches on the frozen tier are generally slower than on the cold tier because Elasticsearch may need to fetch frozen data from the snapshot repository. We recommend dedicated frozen tier nodes.

Prerequisites: Determine which fields have conflicts

To determine which fields have mapping conflicts, navigate to Stack Management -> Data Views -> logs-* (using the logs-* data view is the highest hierarchy of data present with the logs- prefix.) If there are any conflicts, there will be a yellow box stating that. You may either click View conflicts or, under the Field type box next to the Search box, select conflict.

Clicking the yellow Conflict button will reveal which indices are associated with which mapping types.

This situation (where the field is mapped as both a keyword and a long) typically occurs because data was ingested before a specific mapping type was defined in the component template for the relevant data stream. In such cases, Elasticsearch attempts to set the mapping based on its dynamic templates.

In order to determine which mapping is appropriate for the field, and if the field is an ECS field, verification with ECS field reference is needed. If the field in question is not an ECS field, its value must be reviewed to determine the correct mapping.

If a field, such as log.offset in this example, isn’t documented in the ECS, the next steps are to investigate the field's value, determine which conflicting mapping type has the most backing indices, and examine the component templates of the other indices.

Typically, the mapping type associated with the highest number of indices is the correct one, but we recommend you verify the value of the field in question to validate this. To confirm the validity of a mapping type (for example, long), you must also verify that the field's value is appropriate for that type. This verification can be done by using Discover to search for the field in question. Reviewing other data streams that contain the same field can provide additional confirmation also.

To review the values present for the field with the mapping issue, navigate back to the yellow Conflict button stated earlier, click the Conflict button, highlight one of the backing indices, and paste into a Discover session. Your Kibana Query Language (KQL) statement should look like the following screenshot, to include the _index: field delimiter.

Prepare the new backing index custom component template

To address the mapping conflict in the data stream, first examine the relevant @package component template. You can find this under Stack Management -> Index Management -> Component Template. Search for the data stream and select the corresponding @package link. This template contains mappings for the fields out of the box and, while it isn’t common to have a mapping mismatch, it’s possible for the more appropriate type to be overlooked.

Review the template to confirm it contains the necessary field nesting and mapping for the field in question. For example, if the template incorrectly lists log.offset as a keyword, this is the source of the issue.

Important: Because modifying @package/managed templates isn’t recommended, you must use or create an @custom component template to correct the mapping type (for example, for log.offset) for all future data.

• We don’t recommend modifying the @package/managed templates, since when you update the integration to a more recent version, any changes you make to the @package template will be overwritten. This is why we recommend using the @custom templates.

• If a data stream is experiencing mapping conflicts, you need to add any missing field (ECS and non-ECS) nestings or mappings to the data stream's @custom component template. Create this template if it doesn't exist yet, and make sure to specify the correct mapping type for the field.

• If you have multiple conflicts in your data view, apply all the necessary missing mappings for the data stream simultaneously so that the reindex is performed once versus multiple times. Having entries for proper data typing in the @custom component template will ensure any future data ingestion will follow the same mapping guideline.

To create the @custom component template (or verify it’s in use and populated), navigate to Index Templates, type in the name of the data stream in question, and click the appropriate @custom template being used by the data stream. If the template is not yet created, a yellow box will appear, allowing you to create the template through the UI.

The screenshot below shows the next page once Create component template is selected. Leave the defaults as is on the first page and click Mappings or Next until you reach the Mappings page.

To explicitly set the mapping for a new field coming in or to update a field that has a mapping conflict, when the data stream rolls over due to configuration set in the index lifecycle policy, an entry is needed for the field that the conflict exists in.

The below will set the mapping for the log.offset field in the @custom component template for the filestream data stream. Repeat the steps to add any custom fields or update necessary fields from the @package with the appropriate mappings, if needed, for this dataset. In this example, when setting offset to Long, the field type will be Numeric and the Numeric type will be Long. Click Add field and then outside of the area to continue.

Once all needed fields have been added, click through to review, and select Create component template when ready. All new data being ingested from this step forward will have log.offset set to long.

Creating the new backing index structure

The new backing index needs to have the existing mappings from the data stream’s component template, as well as the ECS ecs@mappings component template. The ecs@mappings component template is applied after the data stream’s component as a catchall for additional mappings that potentially weren’t captured in the previous component templates.

Navigate to the browser tab for the data stream's @package mappings. (Go to Stack Management -> Index Management -> Component Template -> logs-filestream.generic@package -> Manage -> Edit.) Once there, click on the Review section, then Request, and finally the Copy button on the right. The JSON contents of the component template copied will ensure the remaining field mappings and settings are retained while we update the log.offset field mapping. The JSON will form the backing structure for the newly reindexed backing index.

Important: If the template’s JSON was not copied and work was continued on with the reindex, the log.offset conflict would be resolved but there would be new conflicts with the integration, as the integrity of the current mappings were not upheld, creating double work to resolve the original issue.

Open a second browser tab, navigate to Dev Tools, and paste the copied content. Now, to clean up what was pasted:

Modifications to the request

1. Index name: Replace _component_template/logs-filestream.generic@package with the name of the backing index you intend to reindex, appending -1 to the end. For example, use PUT <backing index to reindex>-1.

• The appended -1 signifies a reindex and won’t conflict with the default ILM rollover settings, which are based on the index's creation date.

2. Settings: Remove the line "template" (line 3), as well as the very last closing brace for the entire JSON payload; Line 3 should start with "settings": {.

• Replace the inner contents of the settings section with "index.codec": "best_compression". This action will apply Elastic's best compression to the index upon creation.

• Add in "index.lifecycle.name": "logs", as well as a line for "index.lifecycle.rollover_alias": "".

  1. The "index.lifecycle.name": "logs" entry will apply the logs ILM policy to the new backing index. Modify the ILM policy name if you aren’t using logs.

  2. The "index.lifecycle.rollover_alias": "" is blank, since this backing index won’t be rolled over, yet the setting is required to avoid ILM rollover errors into the next ILM phase after hot.

3. Structure: The request should now include both a Settings section and a Mappings section. Inside "mappings": {, you should find "dynamic_templates" and a "properties" section containing hard-coded fields and their mappings.

4. Dynamic templates modification: The current dynamic templates section contains entries for fields that may be overwritten when the ecs@mappings dynamic templates are added next, causing redundancy and extra lines that aren’t needed.

• Remove all sections in "dynamic_templates" except for the second section titled "_embedded_ecs-data_stream_to_constant": {.

• Repeat the same process as described above, gathering the dynamic mappings for the @package component template, but this time the dynamic mappings for ecs@mappings component template.

  • It may be easier to copy the entire contents of the mappings from the UI for the ecs@mappings component template, paste into the working Dev Tools dynamic_templates section, and remove duplicate and unnecessary lines where appropriate. Include these dynamic template setting contents after the"_embedded_ecs-data_stream_to_constant": { entry. The dynamic_templates section should look very similar to the below sample contents in Dev Tools.

• If dynamic_templates are not included/removed altogether, other fields (review the screenshot below) will have double mappings: text and keyword versus the appropriate mappings, if the dynamic_templates section was left included. What’s left should be the "properties" section under "mappings". This will also create issues in the data view by having the fields be double mapped (if not already mapped this way) and will cause additional mapping conflicts.

5. Metadata removal: Delete the last section labeled "_meta", as well as the section labeled "version", if present.

6. Formatting: Auto-indent the remaining sections, and adjust or remove any unnecessary curly braces that would prevent a successful execution.

7. Mapping change: Navigate to the "properties" section, find "log", and then locate "offset" nested underneath. Change the type from keyword to long, and remove the line entry (comma included) labeled "ignore_above": 1024,. If more than one entry was added to the @custom component template created earlier, include them here.

Your Dev Tools console view should now be similar to the example provided below.

After your console resembles the example (with any additional custom fields included and custom values specific to your environment), execute the command to create the shell of the new backing index, pausing to resolve any errors that arise.

Begin reindex process

With the shell of the new backing index successfully created, the next step is to reindex and resolve the mapping conflicts.

Important: If the backing index that has the mapping conflict is the most recent index and is the current write index (for example, the ending number for the backing index is -000001), the data stream needs to be rolled over. Rolling over the data stream is needed since the current write index, which is having documents fed into it, is a live backing index and cannot be modified.

With the correct field mapping now applied to the newer write index via the previously created @custom component template, all new documents will reflect this change.

This is performed by executing the following:

For example:

Reindexing involves copying the data from an existing backing index to a new one within the same naming convention, typically to apply necessary changes. These modifications could include updates to a component template or the addition of a new ingest pipeline for the data to be processed through.

Next, the data will be copied from the backing index that has the incorrect mappings into a new backing index. The original backing index has been rolled over, meaning no new documents can be added to it. The new backing index will follow the same naming convention, which preserves data visibility and integrity while applying the correct ILM policy, but will include a -1 suffix to indicate that it has been reindexed.

Adjust the index names as needed and paste the following code into the console. By including wait_for_completion=false, you can track the progress of document copying, which helps estimate the remaining reindexing time. Without this setting, you cannot track the status using the GET _tasks command below and will only be able to check the document count in the newer backing index using GET <backing index name>-1/_count.

Important: If issues arise during the reindex process, don’t rerun the reindex command; doing so will restart the process and create duplicate records in the index ending with -1. If a restart is necessary, first delete the index with the trailing -1, and then execute the preceding PUT command to recreate the new backing index shell.

Upon execution, the response will include a task ID. You can monitor the reindex progress using this ID with the command: GET _tasks/<task ID>.

The duration of the reindex depends on the volume of data in the original index. The completion can be tracked by looking for "completed": true when executing the GET command, which should yield a similar output.

GET _tasks/<task ID>

With the reindexing process now finished for the document count, the next step is to verify that the mappings for the new backing index and the specific field in question are correct.

For example:

You can verify that the mapping for log.offset is as shown below. To confirm that other fields have only a single mapping entry (not both text and keyword), compare them to a field that was not part of the dynamic template section in the preceding PUT command.

If the backing index that’s being reindexed has a large number of documents, it’s helpful to check the status of those documents being copied to the new backing index; this can be done by the following two Dev Tools commands to compare the counts.

GET .ds-logs-filestream.generic-default-2026.04.14-000001/_count

GET .ds-logs-filestream.generic-default-2026.04.14-000001-1/_count

Once the counts are verified to match and the correct mappings are present, update the data stream to include the new backing index, preventing an orphaned backing index in index management, where the ILM policy will never occur on the backing index.

• The return should be an acknowledgment of true, if successful.

Verify the new backing index is added with the following command, making sure the ilm_policy is correct:

Check the ILM status of the backing index next with the following command:

• It’s normal to see that the index is in hot, as it was created very recently (review line 8 or 10).

Execute the following to transition the backing index from the hot tier to the next appropriate tier that’s after the hot phase for the ILM policy for this data stream. The specific values for phase, action, and name in the current_step below can be referenced from lines 11, 13, and 15, respectively, in the provided screenshot above.

The next_step value indicates the subsequent ILM phase or data tier to which the index will transition to.

For example:

• It isn’t necessary, but as a safety measure, you may execute the _ilm/explain command again to ensure the backing index has moved to the next phase and is no longer in hot.

Once the following conditions are met, you can safely delete the original backing index that had mapping conflicts:

1. A new backing index has been successfully created.

2. Documents have been moved to the new index, and the document counts match.

3. Mappings have been corrected (both data stream specific and ECS).

4. The data stream incorporates the new backing index.

5. The ILM policy has been applied and has moved the index out of the hot phase.

Important: Alternatively, before deleting the original index, you can check the Data Views page. Select logs-* and verify that the reindexed backing index (which ends in -1) now appears in the long section. The original backing index should still be present under keyword. If the reindexed backing index is not in the long section, go back and review the preceding steps and make any necessary corrections.

For example:

After resolving the conflicts, return to the Data Views page and select logs-*. If the conflict was solely related to log.offset, you should no longer see any conflicts listed. If there were other conflicts, the original backing index should no longer appear in the conflict list; instead, the new backing index should now be listed in the long section.

You can also verify in Discover that the log.offset field now displays the appropriate icons.

Continue this process, repeating the above steps for every backing index that has a mapping conflict until all are successfully resolved.

References:

• ECS field reference

• Reindex documents

Final thoughts

By following the steps in this blog, you will resolve mapping conflicts and ensure that all new data is correctly mapped. This is achieved by linking the necessary component templates to your data source. This workflow not only fixes the immediate issues but also establishes a secure and repeatable process for managing schema changes as your data and requirements evolve.

Lexical retrieval (text matching), semantic retrieval (matching concepts), and hybrid retrieval (combining lexical and semantic signals) don’t solve these issues on their own. Lexical retrieval may return anything containing the word “oranges”, while pure semantic retrieval on a high-intent query like “oranges” may broaden toward related items, such as lemons or grapefruits. Hybrid retrieval blends these lexical and semantic signals, but it still doesn’t determine if this query should be treated as navigational, which constraints should be enforced, or which business policies should apply. The gap isn’t the retrieval technology itself; it’s the absence of a governance layer that understands what kind of query this is and what constraints should be enforced before retrieval begins.

In this blog, we explore ecommerce search governance, why it matters, and how a control layer ensures predictable, accurate retrieval.

What governance means in ecommerce search

Governance, in this context, means introducing a decision layer between the user's query and the retrieval engine. This layer performs the following functions:

• Classifies query intent: Is this navigation ("oranges") or discovery ("gift for grandpa")?

• Applies business constraints: What category boundaries, eligibility rules, availability constraints, or merchandising policies apply?

• Routes to the appropriate strategy: Should this use lexical retrieval, semantic retrieval, or hybrid?

A governance layer determines which retrieval approach should be used for each query, which constraints must be enforced, and which business policies should apply before retrieval begins. It’s important not to conflate governance with hybrid retrieval: hybrid is one retrieval strategy that combines lexical and semantic signals, while governance is the upstream decision layer that determines whether lexical, semantic, or hybrid should be used.

The status quo: The application layer "spaghetti" implementation

Currently, many retailers attempt to solve this by adding logic directly into the application layer. This often results in spaghetti code, that is, thousands of lines of hard-coded if-then statements, regex, and complex search templates.

This approach can provide desired search results as shown above; however, it creates significant operational friction:

• Engineering dependency: Business users and merchandisers cannot modify search behavior without engineering tickets and long deployment cycles that often span several weeks.

• Fragmentation: Search logic becomes scattered between application code and search templates, and is difficult to explain or audit, making it risky to evolve.

Even when teams recognize the need for routing, the debate often focuses on the wrong question: which retrieval method to pick.

The false choice: Lexical vs. semantic vs. hybrid

Search teams often frame the challenge as a retrieval strategy choice: lexical/BM25 versus semantic/vectors versus hybrid. That framing is understandable (retrieval methods matter), but it misses the most common failure mode in real deployments, which is that using a single retrieval approach for all queries will give suboptimal results.

Commerce search is a mix of fundamentally different intents:

• Deterministic, high-intent navigation ( "oranges", “milk”, “chocolate without peanuts”, “cheap olive oil”).

• Exploratory discovery ("jacket for hiking in the mountains", "gift for a 12-year-old who likes robotics").

• Operational constraints (availability, size, price, color).

• Merchandising and campaigns (boost, bury, seasonal campaigns).

When the system routes all of these through the same retrieval strategy, the results are often systematically wrong in predictable ways because the operating model lacks governance. When teams don't recognize this as a governance gap, they respond with the only lever they have: more tuning.

Why "relevance tuning" can become cyclical

Without a routing layer, “relevance” often turns into a never-ending backlog:

• Why is this query showing accessories above the core product?

• Why did this head query suddenly start surfacing related items?

• Why did results change after we added synonyms, adjusted analyzers, or enabled hybrid?

• Why does the business team need an engineering release to fix a single query?

Teams respond with more tuning: more synonyms, more boosts, more reranking experiments, more exceptions in application code. This can work for a while, but it often produces brittle behavior because the system still lacks an explicit decision layer for determining query type and enforcing the right constraints before retrieval.

The anatomy of ecommerce intent: Head and tail

In this section, we use “head” and “tail” as practical shorthand for common navigational and exploratory query patterns in ecommerce. In the real world, many queries contain aspects of both:

Head queries (deterministic intent)

These are direct, navigational queries where the user knows exactly what they want:

• Single-item intent ("oranges", "milk", "bread").

• Exact brands or product families ("iPhone 15 Pro", "Diet Coke").

• SKUs, model numbers, sizes ("ABC123", "air max 270").

For these queries, lexical retrieval can handle token correspondence (matching words), but the business also expects to respect constraints, return predictable rankings, and have controllable outcomes. A merchandiser needs to ensure that a query resolves within the correct category boundaries, respects eligibility, and surfaces specific business priorities.

Governance is required to enforce the intended resolution. For example, “oranges” should map to the produce category, not to orange juice, orange marmalade, or orange soda.

Tail queries (exploratory discovery)

These are descriptive, intent-rich queries where shoppers are exploring:

• "Gift for grandpa who has a sweet tooth"

• "Jacket for hiking in the mountains"

• "Shoes for standing all day"

Lexical retrieval often struggles here. Semantic retrieval excels because it can connect the query concept to the product, even when wording does not match. But semantic retrieval alone is rarely sufficient either. Real queries often require constraints to be enforced, regardless of which retrieval method is used.

Constraints are orthogonal to retrieval method

Applying constraints to semantic retrieval doesn’t mean hybrid search. These are orthogonal concepts. Constraints, such as filters and boosts in Elasticsearch, can be applied to any lexical, semantic, or hybrid retrieval. The challenge is deciding how the query should be interpreted, which constraints must be enforced, and which retrieval strategy should be used.

Below are some examples of queries that combine retrieval with hard constraints:

• Oranges: Lexical retrieval for “oranges” plus a category constraint, such as “Fruits” or “Produce”, eliminating orange marmalade, orange juice, and orange soda.

• Fruits high in vitamin C under $4: Semantic retrieval for nutritional intent plus constraints limiting results to the fruit category and products under $4.

• Comfortable shoes for work: Semantic retrieval for contextual intent plus a category constraint limiting results to shoes.

These queries can't be handled by a single approach:

• Pure lexical retrieval is often insufficient here because phrases like “high in vitamin C” or “comfortable” may not exist as clean, structured attributes. They may need to be inferred from product descriptions, reviews, or specifications.

• Pure semantic retrieval is also not always sufficient because, without explicit constraints, a query like “fruits high in vitamin C” might broaden toward vitamin supplements, fruit-flavored drinks, or high-vitamin vegetables outside the intended category and price range.

A governance layer determines whether a query needs lexical retrieval, semantic understanding, constraint enforcement, or some combination of these. Without this layer, ecommerce teams may end up:

• Over-constraining: Using lexical retrieval for semantic requests (for example, "gift for grandpa").

• Under-constraining: Using semantic queries for high-intent head queries (for example, “oranges”).

The governance challenge is to build a system that can make the right judgment call for each class of query.

What happens without governance

The most common failure mode is straightforward: Teams take the raw user query and pass it directly into a single retrieval strategy (lexical, semantic, or hybrid), without an intermediate governance layer.

Lexical retrieval misses intended resolution

When a user searches for “oranges”, a lexical retrieval strategy may return anything containing that token: orange juice, orange marmalade, or orange soda. The system matched the term correctly, but without governance it may not resolve the intended shopping context (the fruit).

Semantic retrieval broadens beyond intended constraints

When a user searches for “oranges”, a semantic system may retrieve conceptually related items across nearby product concepts. The system may correctly understand the broader domain (fruit or produce), but without explicit governance it can still over-broaden beyond the user’s intended constraint (specifically oranges).

The gap is governance

What’s required is an upstream decision layer that determines query intent and enforces the right constraints before retrieval begins. This fixes issues such as the following:

• Similar or related items appearing alongside what the user actually wanted.

• Blurred category boundaries ("beverages" versus. "produce").

• Inability to implement seasonal boosts or campaigns.

• Unpredictable and unexplainable results.

Intent understanding and routing: The necessary control plane

A governed search system introduces a lightweight control plane in front of retrieval (prior to executing a query in Elasticsearch). The control will be discussed in detail in parts 3 and 4 of this blog series; for now, we just discuss what it can do but not how it works:

A control plane can detect intent, apply business policies, and ensure the appropriate retrieval strategy as follows:

1. Detect intent signals

• Is this query likely navigation versus discovery?

• Is it a known head query (milk, bread, bananas)?

• Is there a known product, brand, or category interpretation (for example, “oranges” should resolve to produce).

• Is the query an SKU-like pattern?

• Does the query fall under an active campaign or seasonal policy (for example, during Christmas, boost turkey-related results)?

• Does the query imply constraints (category, attributes, exclusions, price/size/color)?

2. Apply governance and business policies

• Enforce deterministic constraints first (category/attribute/negation/availability).

• Apply active merchandising policies (boost/bury/pin/override).

• Resolve conflicts with precedence rules (for example, campaign overrides versus global policies).

3. Route to the appropriate retrieval strategy

• Lexical (fast, deterministic) for navigational/high-intent head queries.

• Semantic retrieval for true discovery queries.

• Hybrid where combined lexical and semantic signals add value under explicit business constraints.

In practice, the output of the control plane is not simply “use hybrid” or “use semantic.” It’s a governed retrieval plan: an interpretation of the shopper’s intent, the constraints and policies that should apply, and the retrieval strategy that should be executed. A few simple examples make this concrete:

Shopper query

Governed interpretation

Example retrieval plan

“chocolate without peanuts”

Product-oriented query with a hard exclusion constraint

Lexical retrieval for chocolate plus an exclusion filter for products containing peanuts

“cheap olive oil”

Product/category query with a price constraint

Lexical retrieval for olive oil plus a price filter capped at the retailer’s threshold for cheap

“fruit high in vitamin C under $4”

Discovery query requiring semantic understanding plus hard constraints

Semantic retrieval for nutritional intent, constrained to the fruit category and filtered to products priced under $4

A control plane selects the right policy and retrieval strategy for each query consistently, predictably, and at scale. This makes advanced retrieval methods more predictable in production because intent-aligned constraints are enforced first and routing decisions are explicit rather than implicit.

How this relates to other approaches

Some teams use improved embedding models to better capture product semantics, which can materially improve semantic retrieval quality. Others use reranking approaches, such as Learning To Rank (LTR), to optimize result ordering based on engagement or business signals after retrieval. Both are valuable and often complementary. Better embeddings improve similarity matching. Reranking improves ordering among retrieved candidates.

Governance addresses a different layer of the problem: It sits upstream of retrieval. It decides which retrieval strategy to use (for example, lexical, semantic, or hybrid), what deterministic constraints are required, and which queries should combine multiple business policies.

What a governed control plane enables

Once a governance layer is in place, the operating model changes fundamentally. Revenue-critical queries become predictable. Business teams can update search behavior without waiting on engineering release cycles. And advanced retrieval methods, like semantic and hybrid, can be adopted incrementally, behind routing and guardrails, instead of as a global on/off switch.

The next post in this series explores what that operating model looks like in practice and why it may matter as much as the retrieval technology underneath it.

If a merchandiser has to open a Jira ticket and wait for a deploy to fix a revenue-critical query, the bottleneck isn't the engine; it's the operating model. Modern ecommerce search needs a way to translate business intent into controlled, auditable search behavior quickly and safely, while still using advanced retrieval where it adds measurable value.

What’s next in this series

The patterns explored in this series operate upstream of retrieval: translating business intent into the right query strategy before query generation begins. In the next post, we shift from the technical problem to the operational one: what happens when business teams can change search behavior without an engineering deployment, and why governance makes that safe.

Put governed ecommerce search into practice

Engineering bottlenecks, brittle application-layer logic, and unpredictable search results are problems that Elastic Services can help you solve in enterprise ecommerce services engagements. The governed control plane architecture described in this series was built by Elastic Services Engineering.

If your team is spending engineering cycles translating merchandising requests into code changes, or if your search relevance backlog never seems to shrink, we can help you assess your current architecture and build a path to governed, business-editable search. Contact Elastic Services.

Join the discussion

Have questions about search governance, retrieval strategies, or ecommerce search architecture? Join the broader Elastic community conversation.