Three clicks from alert to error log: breaking down RED metrics by any span attribute in Elastic Observability
See which pod, deployment or version is driving a RED metrics change by breaking down span attributes in Discover, then trace a failing span to the error log behind it.
Elastic Observability now lets you break down RED metrics in Discover by any span attribute on your traces. Split by pod, deployment, service version or any custom dimension to see which values moved the metric. From there, you can open a failing span's trace waterfall and follow it through to the linked error log in a few clicks, no query needed.
Availability
This is available in serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.
What you need for RED metrics breakdown in Elastic Observability
You need trace data from a service instrumented with any method Elastic APM supports.
- Application instrumentation: one of the following:
- Elastic APM agents for Java, .NET, Node.js, Python, PHP, Ruby, Go, and other supported languages.
- OpenTelemetry SDKs sending OTLP via Elastic Agent or an upstream OpenTelemetry Collector with the
elasticapmconnector under connectors (not processors).
- Useful attributes: breakdown works best when spans include the dimensions you want to compare (
k8s.pod.name,k8s.deployment.name,service.version, and others). You can also declare custom attributes on spans: add labels to transactions and spans with Elastic APM agents, or set OpenTelemetry attributes on spans and resources with OpenTelemetry SDKs. Those custom fields work as breakdown dimensions too. - Backend: Observability serverless today, or Elastic Stack 9.5 on Elastic Cloud Hosted and self-managed when 9.5 releases.
How to go from a RED metrics alert to the root-cause error log
Step 1: Review RED metrics on the alert detail page
When you receive a notification for a RED metric threshold breach, if you open the alert detail page, you can review the symptoms for the impacted service on one page.
In our example, failed transactions have clearly increased for the cart service, so we want to understand what is driving that RED metric change:
Step 2: Break down RED metrics by span attributes in Discover
To investigate why a RED metric changed, open Traces in Discover and use the new breakdown feature to split RED metrics by any attribute on your spans.
In our example, we're going to check Kubernetes attributes and service version, but you could break down by any span attribute you send (e.g. cloud.region, cloud.availability_zone, or container.id).
Each breakdown shows which attribute values moved the metric, so you can see whether the problem is isolated to one pod, deployment, version, or whatever dimension you split on.
In our example, error rate clusters on a single Kubernetes deployment, which points the investigation at a release. We will break down by service.version to validate our hypothesis:
Step 3: Open the trace waterfall and read the linked error log
Once trace breakdown has identified a specific service version as the likely cause, we can filter by that service.version and look at sample failing spans to see if they explain why the version is causing failures.
Open the trace waterfall for one failing span and follow through to the linked error log.
In our example, the error log points to bad configuration that could be causing the issue. Either way, we have narrowed the investigation to a solid hypothesis we can act on in just a few clicks:
From RED metrics alert to error log in Elastic Observability
From a RED metric alert, you can review the symptomatic service, break down Traces in Discover by any attribute on your spans, and open a failing span's trace waterfall to reach the error log in just a few clicks.
Frequently Asked Questions
What are RED metric drivers?
RED metric drivers are the span attributes that best explain why rate, errors, or duration changed for a service during an alert window. In Elastic Observability, you use the breakdown feature for Traces in Discover to split RED metrics by attributes such as k8s.pod.name, k8s.deployment.name, or service.version and see which values moved.
What setup does this workflow require?
You need trace data from a service instrumented with any method Elastic APM supports. That includes Elastic APM agents or OpenTelemetry SDKs sending OTLP to Elastic Observability. Breakdown works best when spans include dimensions such as k8s.pod.name, k8s.deployment.name, and service.version. You can also add custom labels with Elastic APM agents or OpenTelemetry attributes with OpenTelemetry SDKs, and use those as breakdown dimensions.
How is this different from latency or failure correlations?
Latency and failure correlations in the Elastic Observability Applications UI analyse attributes across transactions for a service you choose. This workflow starts from a RED anomaly alert, opens Traces in Elastic Observability's Discover with the alert context applied, and lets you break down by span attributes to test hypotheses before opening sample failing spans.
Which breakdown dimensions can I use?
In Elastic Observability, you can break down by any attribute present on your spans or span resources. This walkthrough uses k8s.pod.name, k8s.deployment.name, and service.version. Only attributes your instrumentation sends appear as breakdown options.
What are the limitations of this workflow?
In Elastic Observability, missing span attributes reduce the dimensions you can break down by. High-cardinality fields may be grouped or truncated. Sample failing spans are representative, not exhaustive, so confirm the pattern then widen the filter or time range if you need full coverage.
