Four clicks from alert to root cause: how Elastic Observability links APM services to Kubernetes infrastructure

Check service dependencies and compare per-pod CPU, memory and network trends on the Infrastructure tab to find which instance is causing trouble, all without leaving the alert investigation.

Elastic Observability links your OTel-instrumented services to the Kubernetes hosts, containers, and pods they run on. The Infrastructure tab in APM puts per-instance CPU, memory and network trends a few clicks away, so when a service degrades you can spot which pod lines up with when the problem started, all from inside the investigation. This walkthrough follows a latency alert on a recommendation service from notification to the problematic pod in four steps.

Availability

This is available in Elastic Observability serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.

Prerequisites for linking APM services to Kubernetes infrastructure

You need application traces and Kubernetes infrastructure metrics in the same Elastic Observability project.

APM alert triage: from notification to problematic pod

Step 1: Confirm the service degradation on the APM alert detail page

The redesigned alert detail page in Elastic Observability shows the impacted service, environment, endpoint and RED metrics in one view. Open it from the alert notification.

You can clearly see which service is impacted, which environment it runs in, what endpoint is being affected and easily look for correlations in their RED metrics. In this case, we can immediately rule out a spike in traffic as the throughput is clearly stable.

Step 2: Rule out service dependencies with the embedded service map

The newly embedded service map preview on the alert detail page shows the health and RED metrics of every dependent service, so you can rule out upstream causes without navigating away. In this case, we have been able to quickly rule out problems with other services causing the symptom with the symptomatic service:

Step 3: Review Kubernetes infrastructure metrics per pod, container and host

After ruling out service dependencies, open the service's updated Infrastructure tab in Elastic Observability to check for infrastructure-level patterns.

Step 4: Compare per-instance metric trends to find the root cause

The Infrastructure tab shows the average metric values over the specified time period. To really understand whether there is a problem with the infrastructure, we need to compare the pod, container and host metrics over time. This allows us to easily spot differences between different entities that may correlate with when the service started showing symptoms. In our example, we can clearly see a difference between some of the metrics between the pods that correlates with when the service symptoms began. So we know there is something going on with the infrastructure that needs investigating:

Summary: from APM alert to root cause in four clicks

In just a few clicks from an alert in Elastic Observability, you can rule out healthy dependent services without leaving the alert detail page, then compare per-pod infrastructure metrics to see which instance correlates with when the symptoms started.

Share this article