Three SLOs every search team needs: monitoring search latency, availability and quality with OpenTelemetry

Curious about taking Elastic Cloud for a spin? Subscribe to Elastic Cloud on AWS Marketplace or Microsoft Azure Marketplace to receive up to $1,000 in billing credits.

Every search request your API handles already emits an OpenTelemetry span with latency, error, and result count data. You built that instrumentation for product analytics. Turns out it also gives you search monitoring for free. This post takes those spans and turns them into three SLOs (99% of queries under 250ms, 99.9% availability, zero-results rate below 15%), then layers on alerting, anomaly detection and an incident response workflow, all with Elastic Observability's built-in tooling. If you instrumented your search API following Blogs 2-4, you can set this up in an afternoon.

What you'll discover

In this post, you'll learn how to:

• Use Elastic APM's built-in views to explore search latency and throughput and to explore errors.

• Define Service Level Objective (SLOs) for search, including latency targets and availability, along with search quality.

• Create SLOs in Kibana that track your search health over time, with burn rate alerting.

• Build operational dashboards with Elasticsearch Query Language (ES|QL) that show latency percentiles and time breakdowns and that include trends.

• Set up alerts for latency regressions and error spikes, along with zero-results rate increases.

• Establish an incident response pattern for search degradation.

What you'll need

• Search instrumentation from Blogs 2–4 (search spans with search.* attributes in Elastic).

• Kibana access with permissions to create SLOs and alert rules.

• Basic understanding of SLOs. (We'll explain the search-specific parts.)

• An Elastic cluster with an Enterprise subscription, an Elastic Cloud trial, or a local deployment with the trial activated.

Why search monitoring matters beyond cluster health

Search is the primary navigation path for a significant share of visitors, and search-initiated sessions tend to show stronger purchase intent than browse sessions. A search outage is a revenue event, rather than a minor feature degradation. A latency regression from 100ms to 500ms changes user behavior before anyone files a ticket.

Most platform teams monitor search at the infrastructure level, checking whether the Elasticsearch cluster is healthy and whether nodes are responding. They also determine whether the disk is full. These are all necessary but not sufficient. A cluster can be green while search quality silently degrades; for example, queries returning stale data after a bad index deployment or latency creeping up as the index grows. This could also include zero-results rates climbing because a synonym list wasn't updated.

The gap is between "search is up" and "search is working well."

A note on examples: As we have throughout this series, we use ecommerce search for concrete examples, but these reliability patterns apply equally to any search application, including content platforms, internal knowledge bases, job boards, and support portals.

OpenTelemetry search spans as monitoring signals

If your team followed Blogs 2–4 in this series, every search request already emits an OTel span with search.* attributes. These include the query text, result count, Elasticsearch execution time, and error status. Those spans land in traces-generic.otel-default in Elastic.

Following along with code? The reference project has all the instrumentation from Blogs 2–4. Generate traffic, and then follow along with the SLO and dashboard setup below. See queries/blog6_reliability.esql for ready-to-run queries.

The search team built that instrumentation for product analytics; that is, understanding what users search for and measuring click-through rates (CTRs) and conversion rates. They’ve prioritized relevance work, but the same spans contain everything you need for operational monitoring. Span duration gives you a latency signal, and search.result_count == 0 value reflects quality. Span errors point to availability signals.

This post shows how to put this operational value to work, beginning with what you can see right now in Kibana and then building SLOs and alerting on top of it, along with incident response.

Search monitoring out of the box with Elastic APM

Before building anything new, let's look at what Elastic APM already gives you out of the box.

If your search API is instrumented with Elastic Distribution of OpenTelemetry (EDOT) (as in Blog 2), it appears automatically as a service in the Elastic APM UI. Open Observability > APM > Services in Kibana, and select your search service (named search-analytics-demo if you're using the reference project). You'll immediately see the following:

Elastic APM service overview for search

The service overview page shows latency distribution and throughput over time, along with error rate, and doesn’t require configuration. You can see at a glance whether search is healthy, and the time-series charts make regressions obvious. If latency crept up after yesterday's deployment, you'll see it here.

Trace waterfall: breaking down search request latency

Click into any transaction, and you'll see the trace waterfall, which is a visual breakdown of every span in the request. For a search API call, this typically shows:

The waterfall makes the invisible visible. You can see that the 503ms API response breaks down into HTTP handling, a 241ms query rules lookup, and a 260ms Elasticsearch query,  plus the custom search span (36ms) carrying all of our search.* attributes. Click any span, and the metadata flyout shows exactly what was captured: search.query: "usb hub", search.result_count: 33, search.took_ms: 43, the index name, hit IDs, and more.

Blog 2 discussed the gap between search.took_ms and span duration. The waterfall shows you exactly where that gap lives, without writing any queries.

Automatic search error capture with OpenTelemetry

One of the most valuable things OTel auto-instrumentation gives you is automatic error capture. When an Elasticsearch query fails because of issues like a tripped circuit breaker or a timeout, or if an index isn’t found,  the span records the exception type and message, along with the stack trace. Blog 4 mentioned this as a side benefit of span-based conversion tracking; here it becomes an operational lifeline.

The errors tab on your service page automatically aggregates these, grouped by error type and frequency. The instrumentation captures the details for you, so you don't need custom error handling or logging. During an incident, this is often the fastest way to understand what's actually failing.

Service map: search API and Elasticsearch dependencies

The service map shows dependencies between your search API and Elasticsearch, making it easy to see whether a latency problem is in your service or in the cluster it depends on.

All of this is available the moment you deploy the instrumentation from Blog 2, without building any dashboards or writing any queries. This is the foundation everything else in this post builds on.

Defining service level objectives for search

An SLO defines good enough in measurable terms. You define what working means, measure it continuously, and alert when you're burning through your error budget too fast, instead of reacting when something breaks.

Elastic Observability has a built-in SLO framework that handles Service Level Indicator (SLI) calculation and budget tracking. It also takes care of burn rate alerting. You create SLOs directly in Kibana. No ES|QL or custom pipelines are required for the core indicators.

Three SLOs every search service needs

Navigate to Observability > SLOs in Kibana, and click Create SLO. The SLO creation workflow walks you through three steps: Define the SLI (what to measure), set the objective (the target), and describe the SLO.

1. Search latency SLO: 99% of queries under 250ms

Indicator type: Elastic APM latency target: 99% of searches complete in under 250ms.

The Elastic APM latency indicator is purpose-built for this. Select your search service (search-analytics-demo), and set the threshold to 250ms.  Elastic handles the rest, including calculating the percentage of transactions below the threshold and tracking your error budget over time.

Note: The Elastic APM latency indicator measures all HTTP transactions for the search-analytics-demo service, including health checks and click/cart/checkout endpoints, along with static asset requests, not just the POST /api/search endpoint. For a search-only latency SLO, use a custom Kibana Query Language (KQL) indicator with name: "search" AND attributes.search.query: * on the traces-generic.otel-default index. The Elastic APM indicator is still valuable for whole-service health. For full coverage, combine both.

This measures end-to-end span duration; that is, what the user actually experiences. If your Elasticsearch query takes 50ms but the user waits 300ms because of network overhead or slow application logic, this SLO catches it. Use search.took_ms in the Elastic APM waterfall to diagnose where the latency lives when the SLO starts burning.

2. Search availability SLO: 99.9% success rate

Indicator type: Elastic APM availability target: 99.9% of searches succeed.

The Elastic APM availability indicator calculates the percentage of successful transactions for your service. When the Elasticsearch client throws an exception or the search endpoint returns a 5xx, the span's status records an error and this SLO counts it.

Note: Like the latency SLO, the Elastic APM availability indicator covers all HTTP transactions on search-analytics-demo, not just POST /api/search. Click/cart/checkout errors will consume this budget. For a search-only availability SLO, use a custom KQL indicator with name: "search" AND attributes.search.query: * for good events and name: "search" as the total query.

An 0.1% error budget on 100,000 daily searches means that you can tolerate 100 errors per day. That's tight, but search errors are hard failures and the user gets nothing. Availability SLOs should be stricter than latency SLOs.

3. Search quality SLO: tracking zero-results rate

Indicator type: Custom KQL target: 85% of searches return at least one result (zero-results rate < 15%); index: traces-generic.otel-default; good query: name: "search" AND attributes.search.result_count > 0; total query: name: "search" AND attributes.search.query: *.

Note on KQL versus ES|QL: The SLO framework uses KQL for its indicator filters rather than ES|QL. KQL uses field: valuesyntax and is the same language you see in the Kibana search bar. The ES|QL queries throughout this series are for ad hoc analysis and dashboards; KQL here is the SLO indicator's document filter. Both query the same traces-generic.otel-default index.

This is the SLO that surprises most teams. A search that returns an empty result set isn't an error; HTTP status is 200 and the span status is OK. Plus, no exception was thrown. But from the user's perspective, it failed. They asked for something and got nothing.

The quality SLO uses the custom KQL indicator type because it relies on our custom search.result_count attribute, which the built-in Elastic APM indicators don't know about. But the SLO framework handles everything else, including budget tracking and burn rate calculation, along with alerting.

A sudden spike in zero-results rate, such as from 12% to 40% over an hour, is almost always an infrastructure event, like a failed index deployment or a mapping change that broke queries. It could also be a synonym list misconfiguration. That's an operational problem, not a relevance problem.

Reading your search health in the SLO overview

Once the latency, availability and quality SLOs are created, the SLO overview page shows your search health at a glance:

Each SLO shows the current value, the target, the remaining error budget, and the burn rate. Green means healthy:  Search availability is at 100%, and Search quality is just above its 85% target. Red means violated: Search latency is at 50% against a 99% objective, with the burn rate breached at 200x the sustainable rate. When a budget bar starts shrinking faster than expected, you know something changed, even before users complain.

Clicking into the SLO detail shows burn rate across multiple time windows (1h, 6h, 24h, 72h) and the historical SLI trend. It also shows remaining error budget. For the latency SLO, the Elastic APM latency indicator tracks the percentage of transactions below your 250ms threshold. For the quality SLO, the custom KQL indicator uses traces-generic.otel-default with the good query filtering for attributes.search.result_count > 0. This is where the custom search.* attributes from Blog 2 pay off, since they're the foundation of meaningful SLOs.

Burn rate alerting for search SLOs

When you create an SLO through the Kibana UI, a default burn rate alert rule is automatically created. This is where the real operational value lives.

Burn rate alerts improve on threshold alerts ("error rate > 1%"), which are noisy and miss slow degradation. : Burn rate alerts measure how fast you're consuming your error budget relative to the SLO window.

A burn rate of 1.0 means that you're spending budget at exactly the sustainable rate, but a burn rate of 10.0 means that you're burning 10x too fast and you'll exhaust the budget in 1/10th of the window.

The default burn rate rule uses a multi-window approach, with four severity levels:

The short window prevents alerting on brief spikes that self-resolve, and the long window catches sustained degradation. Together, they balance responsiveness with alert fatigue.

Routing search alerts to PagerDuty, Slack and Jira

Alerts are only useful if they reach the right people in the right tools. Elastic's alerting framework supports a wide range of connectors out of the box, including:

• Incident management: PagerDuty, Opsgenie, xMatters for on-call routing.

• Chat: Slack, Microsoft Teams for team notifications.

• Case management: Jira, ServiceNow for automatic ticket creation when SLOs breach.

• Custom: Webhooks for integrating with any system via HTTP.

You can also use Elastic's built-in cases to track incidents directly within Kibana, linking alerts and traces in one place, along with investigation notes, with push to Jira or ServiceNow when escalation is needed.

A typical routing setup:

Anomaly detection for search quality

Some search degradations are gradual shifts that slip past threshold-based alerts, rather than sudden spikes. A relevance regression after a model update might reduce CTR by 15% over a week, and latency might creep up by 5ms per day as the index grows. These are real problems, but they don't trigger burn rate alerts until it's too late.

Elastic's anomaly detection is built for exactly this. It learns normal patterns in your search metrics and flags deviations automatically, and you don’t have to configure any thresholds. 

Detecting search quality degradation with ML anomaly detection

• Latency anomalies:

 Elastic APM anomaly detection can be enabled directly from the Elastic APM UI for your search service. It learns the typical latency distribution, including daily and weekly patterns, and alerts when behavior deviates. A gradual 5ms/day creep will eventually register as anomalous before it hits your SLO threshold.

• CTR drops:

A relevance regression is invisible to traditional monitoring; latency is fine and errors are zero, plus the result counts are normal, but the ranking changed and users aren't clicking. Anomaly detection on click volume per query is a practical proxy: When a query that normally receives 20 first-click events per hour drops to 5, something likely changed.

To set this up: In Kibana → Machine Learning → Anomaly Detection, create a new job. Use the Multi-metric wizard, and select traces-generic.otel-default as the index. Configure a count detector on attributes.search.first_click split by attributes.search.query. This creates a per-query click-volume baseline and alerts when individual query engagement drops outside the expected range.

Note: This job detects click-volume anomalies per query, not CTR (which requires dividing clicks by searches). Click volume is a useful proxy (a CTR regression usually manifests as a drop in absolute click count), but be aware that a traffic surge with flat click volume would show as a CTR drop without triggering this alert. For true CTR anomaly detection, use a scheduled ES|QL transform to materialize hourly CTR values and run anomaly detection on the computed ratio.

Route the resulting ML alert rule to your Slack search channel.

• Throughput shifts:

A sudden drop or unexpected surge in search volume can indicate upstream problems (like load balancer changes or traffic shifts) or downstream issues (such as search becoming unresponsive or users retrying).

Configure ML anomaly alert rules to route these to your notification channels. These complement your SLO burn rate alerts; burn rates catch budget consumption, and anomaly detection catches pattern changes.

Building a search monitoring dashboard with ES|QL

SLOs tell you whether search is healthy. When they indicate a problem, you need a dashboard that tells you why.

The search team and the on-call team need different views of the same data. A search engineer wants query-level detail, such as which queries have low CTR and which ones return nothing. They’re also interested in where to invest in relevance. But an on-call SRE wants the operational picture, including whether search is fast and whether it’s up. It also wants to know whether search is degrading, and if so, since when.

Search monitoring panels for the on-call dashboard

Build it in Kibana dashboards using Kibana Lens panels. Lens supports ES|QL as a data source, so the queries from Blogs 2–4 can power dashboard panels directly. The key panels include:

• Search throughput over time:  A sudden drop is often the first sign of a problem.

• Latency percentiles (p50, p95, p99) over time: When they diverge (p50 flat, p99 spikes), you have a subset of slow queries.

• Error rate over time: Spikes here mean hard failures.

• Zero-results rate over time: A step change upward, especially correlated with a deployment, means something changed in the index or query pipeline.

The ES|QL for each panel follows the patterns from earlier blogs. For example, a latency percentile panel:

FROM traces-generic.otel-default
| WHERE name == "search"
AND attributes.search.query IS NOT NULL
| EVAL bucket = DATE_TRUNC(5 minutes, @timestamp)
| STATS
    p50 = PERCENTILE(attributes.search.took_ms, 50),
    p95 = PERCENTILE(attributes.search.took_ms, 95),
    p99 = PERCENTILE(attributes.search.took_ms, 99)
BY bucket
| SORT bucket

This includes three lines on one chart. When they diverge, such as when p50 stays flat but p99 spikes, you likely have a subset of queries that are slow while the majority are fine. That's a different diagnosis than all queries slowing down (cluster-level pressure).

Drill-down panels: slowest queries and top zero-result queries

For investigation, add a few detail panels, such as:

Slowest queries: A table showing the queries with the highest p95 latency and their search volume. During an incident, this narrows the problem from "search is slow" to "these specific queries are slow."

• Top zero-result queries: A table showing which queries most frequently return nothing. When zero-results rate spikes, this panel immediately shows which queries are responsible.

These drill-down panels use the same ES|QL patterns as Blogs 2 and 3, just surfaced on a persistent dashboard instead of run ad hoc.

Search incident response using OpenTelemetry traces

As an example, an alert fires, noting that search latency has spiked. What happens now?

The trace data from Blog 2's instrumentation gives you a structured path from symptom to root cause.

Step 1: Assess scope

Start at the on-call dashboard, and get answers to the basics:

• When did it start? Narrow the time range to the degradation window.

• How bad is it? Is p50 affected (all queries slow) or just p99 (a subset)?

• Is it just search? Check the Elastic APM service map to determine whether the Elasticsearch dependency is also degraded.

Step 2: Find the problem queries

If the problem is a subset of queries (p99 spike but p50 is fine), use the slowest queries panel or run:

FROM traces-generic.otel-default
| WHERE name == "search"
AND attributes.search.query IS NOT NULL
  AND attributes.search.took_ms > 100
| STATS
    count = COUNT(*),
    avg_ms = ROUND(AVG(attributes.search.took_ms), 0),
    max_ms = MAX(attributes.search.took_ms)
BY attributes.search.query
| SORT count DESC
| LIMIT 10

Adjust the 100 ms threshold to match your environment's normal range. It can be lower for a fast cluster or higher if your data volume makes 100ms typical. This narrows the problem from "search is slow" to "these specific queries are slow." That's the difference between restarting the cluster and investigating a specific query pattern.

Step 3: Drill into the trace waterfall

Pick a slow query, and open it in the Elastic APM trace view. The waterfall shows exactly where time was spent. (Refer back to the trace waterfall GIF above to see a real example of a POST /api/search trace broken down into its component spans.)

The overhead gap between search.took_ms (Elasticsearch time) and span duration (end-to-end time) is your diagnostic tool:

If the problem is in Elasticsearch, drill into the Search Profile API or cluster monitoring. If it's application overhead, look at the spans around the search span in the waterfall.

Step 4: Correlate with events

Check whether the degradation correlates with:

• Deployments: Did someone deploy a new version of the search service or push a new index?

• Cluster events: Is Elasticsearch under memory pressure, or are there long garbage collection pauses? Or maybe the disk is I/O saturated?

• Network: Is latency between the search service and Elasticsearch elevated?

Elastic Observability's unified platform makes this correlation straightforward because traces, logs, metrics, and infrastructure data all live in the same Kibana instance. You're adding filters in the same interface, rather than switching between tools.

Going further: Infrastructure metrics and cost attribution

This post focuses on what you get from trace data; that is, the spans your search API already emits. But OTel and Elastic Observability support a wider instrumentation picture that becomes valuable as your search infrastructure matures.

• Infrastructure metrics. Adding host and container metrics (like CPU, memory, disk I/O, and network) alongside your traces lets you correlate search performance with infrastructure use. When p99 latency spikes, you can immediately see whether the Elasticsearch nodes are under memory pressure and whether garbage collection  pauses are increasing. You can also check whether disk I/O is saturated, and you can do all this in the same Kibana interface. The Elastic Agent collects these automatically for your infrastructure, and the infrastructure monitoring UI surfaces them alongside your Elastic APM data.

• Total cost attribution (TCA). With infrastructure metrics flowing alongside traces, you can start attributing infrastructure costs to specific services and operations. How much compute does your search service consume? How does that correlate with query volume? If a new ranking model doubles CPU usage per query, you can see the cost impact directly. This is particularly valuable for teams running search on cloud infrastructure where costs scale with resource consumption; understanding the cost per search helps justify infrastructure investment and identify optimization opportunities.

• Logs correlation. OTel auto-instrumentation injects trace context (such as trace ID and span ID) into your application logs. This means that when you're investigating a slow search in the trace waterfall, you can click through to the exact log lines from that request, including Elasticsearch slow log entries and application debug output. It also includes error details that don't fit in span attributes. The logs correlation feature automatically ties them together.

These are natural next steps once you have traces working. Each one extends the same unified platform, without new tools or separate pipelines.

How search analytics and search monitoring share one data pipeline

Here's how it all fits together:

The data flows from the instrumentation you built in Blog 2. This one investment supports two audiences: The search team gets product analytics (Blogs 2–5), and the SRE team gets operational monitoring (this post). Neither team needs separate data pipelines.

Getting started with search monitoring in Elastic

This is the last post in the series, and it brings us full circle. Blog 1 describes the vision: Instrument search once with OTel, and send spans to Elastic. Then use ES|QL to answer any question about search behavior. Blogs 2–4 build the instrumentation and analytics, and Blog 5 shows how to feed that data back into relevance improvements. This post shows how the same data powers operational monitoring, including SLOs, alerting, anomaly detection, and incident response.

The key takeaway for search engineers is that the instrumentation you built for analytics already generates the signals. The SLOs and alerts are built-in capabilities of Elastic Observability, as are the dashboards. You're closer to production-grade search monitoring than you might think. Observability isn't a separate discipline you need to learn from scratch. 

If you've been following along and built the instrumentation from Blogs 2–4, start here:

1. Open Elastic APM: Look at your search service, and explore a trace waterfall. You can also check the errors tab.

2. Create three SLOs: Latency (Elastic APM latency), availability (Elastic APM availability), and quality (custom KQL for zero-results).

3. Enable anomaly detection: One click in the Elastic APM UI for latency anomalies.

4. Build the on-call dashboard: Four Lens panels with the queries from this post.

By the end of an afternoon of work, your search service can have the same observability coverage as any other critical production system.

Get started

Working code

• Reference project: Working code for the entire blog series; clone, configure, and run.

Elastic APM and traces

• Elastic APM Overview: Elastic APM concepts and trace analysis.

• Elastic APM UI: Service overview, transactions, dependencies, errors.

• Service Maps: Dependency visualization and health.

• OpenTelemetry Integration: OTel ingestion in Elastic.

SLOs and alerting

• SLOs in Elastic Observability: Creating and managing SLOs.

• Create an SLO: SLI types (like Elastic APM latency, Elastic APM availability, or custom KQL), time windows, budgeting.

• SLO Burn Rate Alerts: Multi-window burn rate alerting.

• Alert Connectors: Slack, PagerDuty, webhook, email integrations.

• Alerting Framework: Rule types and configuration.

• Jira Connector: Automatic ticket creation from alerts.

• ServiceNow Connector: ITSM integration.

• Elastic Cases: Built-in incident tracking with external push.

Anomaly detection

• Anomaly Detection Overview: Unsupervised time series anomaly detection.

• Elastic APM Anomaly Detection: Enable ML for latency, throughput, error rate.

• ML Anomaly Alert Rules: Alerting on detected anomalies.

Dashboards and ES|QL

• Kibana dashboards: Building operational dashboards.

• Lens: Visualization editor.

• ES|QL in Lens: ES|QL-powered dashboard panels.

• ES|QL Overview: Language reference.

Elasticsearch operations

• Slow Log Configuration: Threshold-based query slow logging.

• Search Profiling: Profile API for query execution analysis.

• Monitoring Elasticsearch: Cluster stats, search rate, latency.

From this series

• Modern search analytics with OpenTelemetry: The vision.

• Instrument your search API : Search spans and search.* attributes.

• Measuring search quality : CTR, MRR, click distribution.

• From clicks to conversions: Conversion tracking and revenue attribution.

• Personalizing search from behavior: Judgment lists, rank features, Learning To Rank (LTR).

This is the final post in a six-part series on search analytics with OpenTelemetry and Elastic. Start from the beginning: Modern search analytics with OpenTelemetry, or to start building, jump to Instrument your search API.