Roshan Gonsalkorale

Your SLO is on fire; here's how to find the arsonist in Elastic Observability

When SLO alerting flags a burn rate spike, follow the SLI from the alert detail page through bad event spans and trace waterfalls to find the exact dependency burning your SLO error budget, all without leaving the investigation.

5 min read

The redesigned SLO burn rate alert detail page in Elastic Observability connects the alert to the SLI, the events behind it, and the traces that show which dependency is burning your service-level objective (SLO) error budget. You see when burn rate rose, compare good and bad event spans side by side, then follow the bad ones into the trace waterfall to find the failing hop, all without switching tools.

This walkthrough shows the full path using the OpenTelemetry Demo (Astronomy Shop), where a failing shipping dependency drives every checkout SLI miss. You can reproduce it by running the demo against Elastic Observability, defining an APM availability SLO on checkout, and introducing a failing dependency.

Availability

SLO burn rate analysis is available in Elastic Observability serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.

Prerequisites for SLO burn rate alerting in Elastic Observability

You need an instrumented service and an SLO defined on its APM data in Elastic Observability.

  • Application instrumentation: one of the following:
  • SLO definition: an SLO based on APM latency or APM availability for the service you want to protect. Elastic creates a default burn rate alert rule when you save the SLO.
  • Backend: Elastic Observability serverless today, or Elastic Stack 9.5 on Elastic Cloud Hosted and self-managed when 9.5 releases.

SLO burn rate analysis walkthrough: alert to failing dependency

Step 1: Review SLO burn rate on the alert detail page

Open the SLO burn rate alert detail page from the notification.

The chart shows when burn rate rose and how much error budget remains for the rolling period. From there, open the SLI linked to the alert and decide whether you are looking at a short spike or sustained degradation before you dig into events.

In this example, the availability SLO for checkout is burning budget quickly, and the rise is recent, so the change likely landed in the last few hours rather than as long-running drift:

Step 2: Check the SLI error rate driving SLO error budget consumption

On the SLI view, check the error rate that feeds the SLO.

You can look at the SLI as RED metrics in the APM UI for a quick service-level picture, or open Traces in Discover when you want to filter, compare, or break down the spans that count toward the SLO.

Here, error rate on checkout has climbed, which matches the jump in availability SLO burn rate:

Step 3: Compare good and bad event spans for the SLI

Open the events for the SLI and switch between spans for good events and bad events.

Differences between those sets usually show up before you open individual traces. In this case, bad events share a pattern that good events do not, which narrows where to look next:

Step 4: Find the failing span in the trace waterfall

Open a few sample spans for bad events and view each trace in the waterfall.

That puts the failing span in the request path so you can see which hop is causing the SLI miss. In this example, the failing span is a downstream call, not application logic inside checkout itself:

Step 5: Verify which dependency is burning your SLO error budget

Open the service that owns the bad span and check its dependencies.

Here the waterfall points to shipping and a failing call to quote-old. Requests to that dependency fail on every attempt, so the SLO burn is driven by an external dependency rather than checkout code:

SLO burn rate investigation: from alert to root cause

From an SLO burn rate alert in Elastic Observability, you can see when burn rate rose, open the linked SLI, compare good and bad event spans, inspect bad spans in the trace waterfall, and verify failing dependencies to find what is burning error budget.

Share this article