It's the middle of your on-call rotation and your phone buzzes. CrashLoopBackOff. A pod is stuck in a restart cycle, and now the clock is running.
If you've been an SRE for any length of time, you know what usually comes next. You acknowledge the page, open your observability tool, and start the process: pull up the cluster dashboard, find the namespace, find the pod, check restart counts, pivot to logs, check whether an upstream dependency is degraded, compare against last week, and slowly assemble a picture from a dozen tabs. It works, but it's a process, and the process is where the minutes go.
Elastic's new Kubernetes Experience changes the starting point. When the CrashLoopBackOff alert fires, an Investigation Workflow runs automatically alongside it. By the time you open the alert, the evidence has already been gathered and a root-cause hypothesis is waiting for you. Instead of a blank dashboard, you open the alert to an answer. Or at minimum, a strong starting point that tells you exactly where to look next.
This post walks through a typical CrashLoopBackOff scenario end to end. The sections that follow break down what Elastic's UI shows at each step and why it saves you time.
The manual CrashLoopBackOff investigation: six steps every SRE runs
Here's the shape of a normal CrashLoopBackOff investigation, minus Elastic's workflow:
- You get paged. Something restarted too many times.
- You orient. Which pod? Which namespace? Which deployment owns it?
- You characterize. How many restarts? What was the last termination reason — OOMKilled, a failed liveness probe, a bad exit code?
- You classify. Is this a memory problem, a config problem, a dependency problem, a scheduling problem?
- You corroborate. Pull Kubernetes events, read the pod logs, check whether an upstream service started erroring first, compare current behavior against a healthy baseline.
- You conclude. Only now can you form a hypothesis and act.
Every one of those steps is a query, a click, or a context switch. None of them is hard on its own. Together, on a bad night, they're twenty minutes you don't have and they're twenty minutes of the same steps you ran during the last incident, and the one before that.
The insight behind Elastic's Kubernetes Experience is simple: that sequence is deterministic enough to automate. So Elastic automated it.
Alert to Root cause in minutes
Elastic's Kubernetes integration ships pre-built alert rule templates for states that are wrong by definition; no baseline or warmup required. A pod in CrashLoopBackOff is always a problem, so the rule fires the moment the restart count crosses your configured threshold within a rolling window.
Here's the alert firing in this scenario, both the [Kubernetes OTel] Pod CrashLoopBackOff and OOMKilled containers rules light up for the recommendation group in the otel-demo namespace:
The alert itself is defined by an ES|QL query, so it's transparent and tunable; you can read exactly what triggers it and adjust the threshold to match your environment. And its action is what makes the rest of this post possible: the rule runs the K8s CrashLoopBackOff Investigation (OTel) workflow per alert, the instant a new alert fires. (For a refresher on the alert library and how these templates work, see Part 1.)
In this scenario the rule fires on the recommendation pod (recommendation-788dc88c6c-56w74) in the otel-demo namespace. When you open the alert, Elastic's AI Agent is already summarizing what happened; no rule-details spelunking required:
But the alert firing is only half the story. Attached to it is a Kubernetes Investigation Workflow (technical preview), a directed graph of steps that triggers the instant the alert does. While your phone is still buzzing, the workflow is already querying your cluster, branching on what it finds, and synthesizing an answer.
Inside the investigation: what the workflow actually does
The workflow mirrors the exact sequence an experienced SRE would run by hand, except it runs in seconds and writes nothing down that it can't back up with evidence. For our CrashLoopBackOff scenario, here's the path it takes.
Step 1: What does the workflow check first on the crashing pod?
The workflow queries Kubernetes metrics for the restart count, the last termination reason, and utilization against the pod's declared limits.
Result: last termination reason OOMKilled, restart count 7. Memory utilization data happened to be unavailable at query time, but the OOMKilled reason is definitive: the container is being killed by the kernel for exceeding its memory limit on each startup, then immediately restarting.
The OOMKilled termination reason determines everything that follows. The workflow branches down the memory-investigation path rather than the log-investigation path it would take for a non-memory crash.
Step 2: How does the workflow tell a memory leak from a load spike?
The ML anomaly check is the step that separates a good investigation from a fast-but-wrong one. OOMKilled does not automatically mean "memory leak." Rather than recompute memory trends from scratch, the workflow queries the ML anomaly index for an active k8s_pod_memory_growth anomaly on this pod.
Result: no anomaly. The memory spike is flagged as load-driven, not a suspected leak. The ML baseline — established over the preceding days — didn't see the slow, creeping growth trajectory that characterizes a leak. It saw a jump consistent with real traffic.
Distinguishing a load-driven memory spike from a genuine leak would take a human several minutes and a good deal of judgment. The workflow reaches it because Part 1's anomaly detection jobs were already learning the workload's baseline in the background.
Step 3: Is the failure spreading to other Kubernetes services?
A crashing pod is often a symptom, not a cause, and it can also cause problems downstream. So the workflow enumerates the pod's dependencies from APM service_destination aggregates and compares the current error rate and latency against baseline. An AI classification step decides whether the failure is spreading.
Result: the sole direct caller is the frontend service, which is absorbing the impact with just a 0.14% error rate against the recommendation service and no other service exceeds its degradation threshold relative to baseline. The blast radius is isolated to the recommendation service; there's no significant downstream cascade. The problem is local to this pod.
Step 4: Did a recent change in the namespace cause the crash loop?
Finally, the workflow scans the namespace event log. It finds a continuous Pulled → Created → Started → Killing → BackOff cycle running from roughly 18:51 to 18:54 UTC, the textbook signature of an active crash loop at the time the alert fired. Nothing changed operationally; this is a steady-state resource problem.
What the CrashLoopBackOff root-cause hypothesis looks like
When you open the alert, this is what greets you:
ROOT CAUSE HYPOTHESIS (confidence: high)
The recommendation service pod (recommendation-788dc88c6c-56w74) is in a
crash-loop caused by repeated OOMKilled terminations. The pod has restarted
7 times and Kubernetes events confirm a continuous BackOff/restart cycle
since at least 18:51 UTC. Memory utilization data was unavailable at query
time, but the OOMKilled termination reason is definitive: the container is
exceeding its configured memory limit on each startup, being killed by the
kernel, and immediately restarting. No memory leak was detected by ML
anomaly analysis, indicating the memory pressure is load-driven — the
container's memory limit is simply insufficient for the current request
volume. The frontend service (the sole direct caller) is absorbing the
impact with a 0.14% error rate on the recommendation service itself, but
no significant downstream cascade is observed.
EVIDENCE
- Pod restarted 7 times; last termination reason: OOMKilled — container is
consistently exceeding its memory limit
- ML memory anomaly check: no anomaly found; memory spike assessed as
load-driven, not a leak
- Blast radius is isolated to the recommendation service; no other service
exceeds degradation thresholds relative to baseline
- Continuous BackOff events from 18:51–18:54 UTC confirm active crash-loop
at alert time
PROBABLE CAUSE: The recommendation container's memory limit is too low for
current traffic load, causing repeated OOMKilled terminations and a
crash-loop backoff.
RECOMMENDED NEXT STEPS
1. Immediately increase the memory limit (and request) for the
recommendation container in its Deployment spec to provide headroom
above the observed peak usage, then redeploy to break the crash-loop.
2. Profile the recommendation service under representative load to
determine the actual memory working set and set a right-sized limit
with a safe buffer (e.g., 20–30% above peak observed).
3. Add a Kubernetes HorizontalPodAutoscaler or VPA policy for the
recommendation service so memory resources scale with traffic rather
than requiring manual intervention.
Read that again from the perspective of the on-call engineer. You were paged. You opened the alert. And the alert didn't hand you a pile of logs and a dashboard; it handed you a calibrated root-cause hypothesis with the evidence attached and the next actions spelled out. The six manual steps from the "old way" are done. Your job is now to decide, not to dig.
That's the time savings: not shaving a few seconds off each query, but removing the entire investigative scavenger hunt from the critical path.
How does Elastic Observability avoid misdiagnosing OOMKilled as a memory leak?
Speed is worthless if the answer is wrong, so it's worth noting how the workflow avoids the classic misdiagnoses. It encodes the reasoning an experienced SRE applies instinctively:
OOMKilledis not automatically a leak. It compares against a 7-day baseline before ever claiming one. Here, that check is what turned "the app is leaking memory" into the correct "the limit is undersized for real load."- Co-symptoms are not causes. It explicitly checks whether the upstream degraded first before blaming or clearing it.
- Absence of evidence is not evidence. If a query returns zero rows, it reports "no data available" rather than inventing a failure mode.
- It's honest about confidence. The hypothesis is labeled
high,medium, orlow. When two failure modes fit the evidence, the workflow names both and says which it believes is causal and why. Manufacturing false confidence is treated as a failure of the investigation itself.
This is the same diagnostic protocol encoded in Elastic's observability-k8s-investigation Skill, a failure-mode taxonomy covering 16 distinct Kubernetes failure patterns, from OOMKilled and CPU throttling through scheduling and networking issues. (More on that in Part 2.)
Still want to double-check? Dashboards, Discover, and APM
The workflow gives you the answer. But a good root-cause tool should also make it trivial to verify that answer because sometimes you want to see it with your own eyes, and sometimes the workflow returns medium confidence and you need to close the gap yourself. Everything the workflow reasoned over is available to you directly.
Dashboards — confirm the restart cascade visually. The Kubernetes dashboards are built for drill-down. Start at the cluster Overview, where "top namespaces by container restarts" surfaces the problem at a glance. Click into the flagged namespace, then the pod driving the restarts. The Pods view flags container restarts on the recommendation pod and plots memory against requests and limits over time — you'll see the working set pressing against the limit exactly as the workflow described. It's roughly four clicks from cluster to container.
Discover — read the raw evidence. The pod detail dashboard links directly to correlated pod logs and events in Discover. Here you can confirm the OOMKilled events and the restart cadence in the raw log and event stream, and run your own ES|QL queries if you want to slice the data differently.
APM — verify the blast radius is really contained. The workflow found the impact isolated to the recommendation service, with frontend (the sole caller) absorbing it at a 0.14% error rate. You can confirm that independently in the APM UI: open the service map, check the caller's latency and error rate over the incident window, and compare against the weekly baseline yourself.
The point isn't that you have to do any of this; it's that the workflow's conclusion is fully auditable. Fast when you trust it, transparent when you want to check.
The same investigation from your IDE: the MCP App
Not every investigation starts from a Kibana alert. Sometimes a developer just asks, "why is this service crashing?" from their editor. Elastic's Observability MCP App (technical preview) exposes the same telemetry (and the same investigation workflow) as AI-callable tools that render interactive views inline in your chat or IDE, no context switch to Kibana required.
For our CrashLoopBackOff scenario, the flow looks like this from an MCP-compatible client such as Claude Desktop or VS Code:
"What's broken?" → the cluster health rollup returns an overall health badge, degraded services, top memory consumers, and a Kubernetes breakdown (CPU, memory, restarts, nodes) in one inline view. Here it flags a critical cluster with payment, cart, frontend, and frontend-proxy degraded.
"Is anything anomalous in the recommendation pod?" → the memory analysis view confirms the spike is load-driven, not a leak — the crashing 788dc88c6c pods report null memory (they die too fast to emit a sample) while the healthy pods sit flat at 45MB, exactly the ML result the workflow used.
"Why is the recommendation service crashing?" → the agent returns the same structured root-cause reasoning you'd see on the alert, rendered inline: memory limit set below what the container needs to boot, the ReplicaSet has been intermittently OOMing for weeks, and a concrete mitigation (roll back, then fix the limit), all without leaving the editor.
Same evidence, same root cause, delivered wherever you happen to be working. (For the full set of MCP App views and architecture, see Part 2.)
From paged to decided: skipping the Kubernetes investigation entirely
The change here is not "a better dashboard." It's a shift in what you do when you get paged.
Before, the alert was the start of the investigation. You were notified, and then you went and found the answer. Now, the alert arrives with the investigation already run: evidence gathered, dead ends eliminated, a calibrated hypothesis, and next steps in hand. You go straight from "notified" to "deciding," and you keep the full trail of dashboards, Discover, and APM for whenever you want to verify or dig deeper.
For a single incident that's a few minutes saved. Across a quarter of on-call rotations, across every engineer who no longer re-runs the same six steps at 3 a.m., it's real time back and a lot less alert fatigue.
Try it yourself
You don't need a production incident to see this. The OpenTelemetry Astronomy Shop demo environment ships with a feature-flag service that lets you trigger failure scenarios on demand. Enable a cart/checkout failure, watch the restart cascade unfold, and the CrashLoopBackOff alert rule fires with the investigation workflow running right behind it.
To get set up:
- Install the Kubernetes integration — dashboards are available immediately. (Part 1: Getting started)
- Deploy data collection via the EDOT Collector (OpenTelemetry) or standalone Elastic Agent, both Helm-based.
- Enable the alert rule templates in Observability > Alerts, including CrashLoopBackOff, and connect your notification channel.
- Let the ML modules warm up for 24–48 hours so anomaly baselines are ready when you need them.
- Enable the Investigation Workflow (technical preview) — import the Kubernetes Crashloop Investigation Workflow from the Workflows page and configure it to trigger on the alert. (Part 2: Getting started)
- Install the MCP App (technical preview) on your favorite agentic client to bring investigations into your IDE.
Running Kubernetes on Elastic today? Tell us which investigation steps you still repeat by hand on every incident, and which remediations you'd trust a workflow to propose. Join the Elastic Community Discussion.
The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.