Availability

This is available in Elastic Observability serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.

Prerequisites for linking APM services to Kubernetes infrastructure

You need application traces and Kubernetes infrastructure metrics in the same Elastic Observability project.

• Application instrumentation: EDOT-instrumented services sending traces via the EDOT Collector or an upstream OpenTelemetry Collector with both the elasticapm connector and the elasticapm processor. The EDOT Collector includes both by default; for a custom upstream pipeline, see the upstream collector setup.
• Kubernetes observation: the cluster observed via OpenTelemetry with host and Kubernetes metrics from the EDOT Collector. See the Kubernetes quickstarts and Kubernetes observability with EDOT for setup.
• Backend: Observability serverless today, or Elastic Stack 9.5 on Elastic Cloud Hosted and self-managed when 9.5 releases.

APM alert triage: from notification to problematic pod

Step 1: Confirm the service degradation on the APM alert detail page

The redesigned alert detail page in Elastic Observability shows the impacted service, environment, endpoint and RED metrics in one view. Open it from the alert notification.

You can clearly see which service is impacted, which environment it runs in, what endpoint is being affected and easily look for correlations in their RED metrics. In this case, we can immediately rule out a spike in traffic as the throughput is clearly stable.

Step 2: Rule out service dependencies with the embedded service map

The newly embedded service map preview on the alert detail page shows the health and RED metrics of every dependent service, so you can rule out upstream causes without navigating away. In this case, we have been able to quickly rule out problems with other services causing the symptom with the symptomatic service:

Step 3: Review Kubernetes infrastructure metrics per pod, container and host

After ruling out service dependencies, open the service's updated Infrastructure tab in Elastic Observability to check for infrastructure-level patterns.

Step 4: Compare per-instance metric trends to find the root cause

The Infrastructure tab shows the average metric values over the specified time period. To really understand whether there is a problem with the infrastructure, we need to compare the pod, container and host metrics over time. This allows us to easily spot differences between different entities that may correlate with when the service started showing symptoms. In our example, we can clearly see a difference between some of the metrics between the pods that correlates with when the service symptoms began. So we know there is something going on with the infrastructure that needs investigating:

Summary: from APM alert to root cause in four clicks

In just a few clicks from an alert in Elastic Observability, you can rule out healthy dependent services without leaving the alert detail page, then compare per-pod infrastructure metrics to see which instance correlates with when the symptoms started.

Availability

This is available in serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.

What you need for RED metrics breakdown in Elastic Observability

You need trace data from a service instrumented with any method Elastic APM supports.

• Application instrumentation: one of the following:
  • Elastic APM agents for Java, .NET, Node.js, Python, PHP, Ruby, Go, and other supported languages.
  • OpenTelemetry SDKs sending OTLP via Elastic Agent or an upstream OpenTelemetry Collector with the elasticapm connector under connectors (not processors).
• Useful attributes: breakdown works best when spans include the dimensions you want to compare (k8s.pod.name, k8s.deployment.name, service.version, and others). You can also declare custom attributes on spans: add labels to transactions and spans with Elastic APM agents, or set OpenTelemetry attributes on spans and resources with OpenTelemetry SDKs. Those custom fields work as breakdown dimensions too.
• Backend: Observability serverless today, or Elastic Stack 9.5 on Elastic Cloud Hosted and self-managed when 9.5 releases.

How to go from a RED metrics alert to the root-cause error log

Step 1: Review RED metrics on the alert detail page

When you receive a notification for a RED metric threshold breach, if you open the alert detail page, you can review the symptoms for the impacted service on one page.

In our example, failed transactions have clearly increased for the cart service, so we want to understand what is driving that RED metric change:

Step 2: Break down RED metrics by span attributes in Discover

To investigate why a RED metric changed, open Traces in Discover and use the new breakdown feature to split RED metrics by any attribute on your spans. In our example, we're going to check Kubernetes attributes and service version, but you could break down by any span attribute you send (e.g. cloud.region, cloud.availability_zone, or container.id).

Each breakdown shows which attribute values moved the metric, so you can see whether the problem is isolated to one pod, deployment, version, or whatever dimension you split on.

In our example, error rate clusters on a single Kubernetes deployment, which points the investigation at a release. We will break down by service.version to validate our hypothesis:

Step 3: Open the trace waterfall and read the linked error log

Once trace breakdown has identified a specific service version as the likely cause, we can filter by that service.version and look at sample failing spans to see if they explain why the version is causing failures.

Open the trace waterfall for one failing span and follow through to the linked error log.

In our example, the error log points to bad configuration that could be causing the issue. Either way, we have narrowed the investigation to a solid hypothesis we can act on in just a few clicks:

From RED metrics alert to error log in Elastic Observability

From a RED metric alert, you can review the symptomatic service, break down Traces in Discover by any attribute on your spans, and open a failing span's trace waterfall to reach the error log in just a few clicks.

Where the APM service map is available

This is available in Elastic Observability serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.

Prerequisites for APM service map dependency analysis

You need trace data from services instrumented with any method Elastic APM supports.

• Application instrumentation: one of the following:
  • Elastic APM agents for Java, .NET, Node.js, Python, PHP, Ruby, Go, and other supported languages
  • Elastic Distributions of OpenTelemetry (EDOT) language SDKs
  • OpenTelemetry SDKs sending OTLP via the EDOT Collector, Elastic Agent, APM Server, or the Managed OTLP endpoint. If you run a custom upstream Collector pipeline, include both the elasticapm connector and the elasticapm processor. Those components ship with the EDOT Collector (or a custom EDOT-like build); they are not part of the standard OpenTelemetry Collector Contrib distribution. For wiring details, see the upstream collector setup.
• Service map data: distributed traces that link the services in your architecture. The map draws connections from span parent-child relationships across instrumented services.
• Backend: Elastic Observability serverless today, or Elastic Stack 9.5 on Elastic Cloud Hosted and self-managed when 9.5 releases.

Dependency analysis walkthrough: from APM alert to shipping dependency

Step 1: APM service map on the alert detail page

When you receive a notification for a RED metric threshold breach on the checkout service, open the alert detail page.

The upgraded service map is embedded on the page, so you can start analysing dependencies the moment you land on the alert. You see checkout in context with its upstream and downstream connections without navigating away.

In our example, failed transactions have increased on checkout. The map is already scoped to the alert time range:

Step 2: Spotting a failing dependency on the service map

When we scan through the downstream dependencies, we can see some indicators that there is a problem with the checkout and shipping services. When we open the shipping service fly-out to view more details, we can see the transactions for /get-quote have an elevated failure rate:

Step 3: Drilling into failing transactions to identify RED metric drivers

Click on the /get-quote transaction to drill into these transactions. We can see a release marker for version 2.3.0 that seems to correlate to increase in failures. To validate this is the main contributing factor, we want to analyse these transactions and check a few dimensions to isolate the behaviour to the release:

Step 4: Comparing RED metrics by dimension in Traces in Discover

Using Traces in Discover, we can use the breakdown feature to compare various attributes to confirm the RED metric change is due to the release. As we break down by service.version, k8s.deployment.name and k8s.pod.name, we can see the failure rate is highly elevated for just this Kubernetes deployment.

These field names match the OpenTelemetry semantic conventions used by the demo (EDOT / OTel instrumentation). If you instrument with classic Elastic APM agents, use the ECS-style kubernetes.* equivalents instead (see ECS ↔ OTel field alignment).

Given we have clear indicators that this problem is almost certainly due to a problem caused by this Kubernetes deployment, we would next investigate the changes this deployment made so we can identify a mitigation strategy to return the system to a healthy status, most likely a rollback of the Kubernetes deployment.

Dependency analysis with the service map on custom dashboards

The APM service map can also be added as a panel to custom Kibana dashboards. If you have a custom Dashboard you would like to surface for a debugging problem with your service, you can attach it to your alert rule so users can use this to complement the default alert detail view.

When you receive the notification and land on the alert detail page, you can click Related Dashboards and open your custom dashboard. From here, you can use the same Service Map panel to quickly analyse dependencies just like the alert detail page:

What's new in the APM service map

Below is a list of the upgrades to the service map:

Service map embedded on APM alert pages and dashboards

• All APM alert detail pages will have the service map embedded to facilitate faster dependency analysis.
• There is a new Service Map Dashboard panel that you can add to any Dashboard.

Full-screen service map for large architectures

Open the map in full screen from the APM UI or from a dashboard panel. Full screen gives you more room on large architectures and works the same whether you opened the map from an alert, a service, or a dashboard.

Service map controls: search, filters, and orientation

On the service map in the APM UI and on dashboard embeds, the map toolbar adds controls to move faster on busy environments:

• Search — find services by text string

• Orientation — switch layout direction when dense graphs are hard to read

• Filters — narrow the map by alert status, whether a service has dependencies, anomaly status, and SLO status

Service map legend: node shapes, connections, and anomaly scores

A built-in legend explains node shapes, connection styles, and health indicators on the map. You spend less time decoding colours and icons when you are triaging under pressure.

The legend covers node shapes (instrumented services, databases and messaging, grouped resources), connection styles (one-way and two-way requests), and anomaly score colour rings from low through critical:

Minimap for navigating large service maps

A minimap helps you orient yourself on large maps. It also highlights anomalous services at a glance, so you can spot outliers without panning across the full graph.

RED metrics on connections with drill-through to Traces in Discover

Select a connection between two services to view RED metrics for the requests between them: rate, errors, and duration for that specific dependency edge.

From there, open Traces in Discover in one click to analyse those requests with full query flexibility. That is how we confirmed shipping was failing the requests checkout sent to it in the walkthrough above.

Adding the service map to a dashboard from the APM UI

From the service map in the APM UI, add the current map view to a dashboard in one click. You do not need to rebuild filters or time range settings manually when you want the same map on a team dashboard.

Service fly-out: preview RED metrics, anomalies and SLOs from the map

Wherever you view the service map in Kibana, you can open a service fly-out to preview a service without leaving the map.

The fly-out shows:

• RED metrics for the service
• Anomaly status — whether machine learning has flagged unusual behaviour
• SLO status — whether the service is meeting its objectives
• Transaction breakdown — how rate, errors, and duration split across transaction types

Use it to sanity-check a node before you open the full service page or follow a connection into Traces in Discover.

Summary: From APM alert to root cause with the service map

From a RED metric alert on checkout, the embedded service map showed shipping as the failing downstream dependency. Map controls, connection RED metrics, dashboard embeds, and the service fly-out are available on every map view in Kibana, so you can start dependency analysis wherever you already work.