Elastic Observability links your OTel-instrumented services to the Kubernetes hosts, containers, and pods they run on. The Infrastructure tab in APM puts per-instance CPU, memory and network trends a few clicks away, so when a service degrades you can spot which pod lines up with when the problem started, all from inside the investigation. This walkthrough follows a latency alert on a recommendation service from notification to the problematic pod in four steps.
Availability
This is available in Elastic Observability serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.
Prerequisites for linking APM services to Kubernetes infrastructure
You need application traces and Kubernetes infrastructure metrics in the same Elastic Observability project.
- Application instrumentation: EDOT-instrumented services sending traces via the EDOT Collector or an upstream OpenTelemetry Collector with both the
elasticapmconnector and theelasticapmprocessor. The EDOT Collector includes both by default; for a custom upstream pipeline, see the upstream collector setup. - Kubernetes observation: the cluster observed via OpenTelemetry with host and Kubernetes metrics from the EDOT Collector. See the Kubernetes quickstarts and Kubernetes observability with EDOT for setup.
- Backend: Observability serverless today, or Elastic Stack 9.5 on Elastic Cloud Hosted and self-managed when 9.5 releases.
APM alert triage: from notification to problematic pod
Step 1: Confirm the service degradation on the APM alert detail page
The redesigned alert detail page in Elastic Observability shows the impacted service, environment, endpoint and RED metrics in one view. Open it from the alert notification.
You can clearly see which service is impacted, which environment it runs in, what endpoint is being affected and easily look for correlations in their RED metrics. In this case, we can immediately rule out a spike in traffic as the throughput is clearly stable.
Step 2: Rule out service dependencies with the embedded service map
The newly embedded service map preview on the alert detail page shows the health and RED metrics of every dependent service, so you can rule out upstream causes without navigating away. In this case, we have been able to quickly rule out problems with other services causing the symptom with the symptomatic service:
Step 3: Review Kubernetes infrastructure metrics per pod, container and host
After ruling out service dependencies, open the service's updated Infrastructure tab in Elastic Observability to check for infrastructure-level patterns.
Step 4: Compare per-instance metric trends to find the root cause
The Infrastructure tab shows the average metric values over the specified time period. To really understand whether there is a problem with the infrastructure, we need to compare the pod, container and host metrics over time. This allows us to easily spot differences between different entities that may correlate with when the service started showing symptoms. In our example, we can clearly see a difference between some of the metrics between the pods that correlates with when the service symptoms began. So we know there is something going on with the infrastructure that needs investigating:
Summary: from APM alert to root cause in four clicks
In just a few clicks from an alert in Elastic Observability, you can rule out healthy dependent services without leaving the alert detail page, then compare per-pod infrastructure metrics to see which instance correlates with when the symptoms started.