Availability

This is available in serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.

What you need for RED metrics breakdown in Elastic Observability

You need trace data from a service instrumented with any method Elastic APM supports.

• Application instrumentation: one of the following:
  • Elastic APM agents for Java, .NET, Node.js, Python, PHP, Ruby, Go, and other supported languages.
  • OpenTelemetry SDKs sending OTLP via Elastic Agent or an upstream OpenTelemetry Collector with the elasticapm connector under connectors (not processors).
• Useful attributes: breakdown works best when spans include the dimensions you want to compare (k8s.pod.name, k8s.deployment.name, service.version, and others). You can also declare custom attributes on spans: add labels to transactions and spans with Elastic APM agents, or set OpenTelemetry attributes on spans and resources with OpenTelemetry SDKs. Those custom fields work as breakdown dimensions too.
• Backend: Observability serverless today, or Elastic Stack 9.5 on Elastic Cloud Hosted and self-managed when 9.5 releases.

How to go from a RED metrics alert to the root-cause error log

Step 1: Review RED metrics on the alert detail page

When you receive a notification for a RED metric threshold breach, if you open the alert detail page, you can review the symptoms for the impacted service on one page.

In our example, failed transactions have clearly increased for the cart service, so we want to understand what is driving that RED metric change:

Step 2: Break down RED metrics by span attributes in Discover

To investigate why a RED metric changed, open Traces in Discover and use the new breakdown feature to split RED metrics by any attribute on your spans. In our example, we're going to check Kubernetes attributes and service version, but you could break down by any span attribute you send (e.g. cloud.region, cloud.availability_zone, or container.id).

Each breakdown shows which attribute values moved the metric, so you can see whether the problem is isolated to one pod, deployment, version, or whatever dimension you split on.

In our example, error rate clusters on a single Kubernetes deployment, which points the investigation at a release. We will break down by service.version to validate our hypothesis:

Step 3: Open the trace waterfall and read the linked error log

Once trace breakdown has identified a specific service version as the likely cause, we can filter by that service.version and look at sample failing spans to see if they explain why the version is causing failures.

Open the trace waterfall for one failing span and follow through to the linked error log.

In our example, the error log points to bad configuration that could be causing the issue. Either way, we have narrowed the investigation to a solid hypothesis we can act on in just a few clicks:

From RED metrics alert to error log in Elastic Observability

From a RED metric alert, you can review the symptomatic service, break down Traces in Discover by any attribute on your spans, and open a failing span's trace waterfall to reach the error log in just a few clicks.