A CrashLoopBackOff alert on checkout-api is paging you.
With PromQL in Elasticsearch and Kibana, you can move from that alert to OOMKilled, prove the container is hitting its memory limit (not the node), raise the limit, and watch the alert recover.
If you are new to PromQL in Elastic, start with Elasticsearch supports PromQL, PromQL queries in Kibana, and Investigate Kubernetes infrastructure with PromQL.
What you need
- An Observability serverless project, or an Elastic Cloud Hosted or self-managed stack at version 9.4 or later. PromQL is generally available in Elastic Cloud Serverless and Elastic Stack 9.5, and available as a technical preview in Elastic Stack 9.4.
- Kubernetes state and container memory metrics in Elasticsearch.
What is the alert telling us?
This is the alert that opened the investigation:
It comes from this waiting-reason query:
PROMQL
max by (namespace, pod, container) (
max_over_time(
kube_pod_container_status_waiting_reason{
namespace="checkout",
pod=~"checkout-api-.*",
container="api",
reason="CrashLoopBackOff"
}[2m]
)
) == 1
A result of 1 means Kubernetes is delaying another start because the container has failed repeatedly.
That is the correct paging signal here because the checkout path has a single replica: when that replica restarts, requests fail.
The max_over_time(...[2m]) range keeps the alert tied to recent samples.
Without it, the last observed value of 1 can outlive the pod, and the rule keeps matching after that pod is gone.
That PromQL query ran every minute over a two-minute window and created an alert after one matching run.
Why did the last container stop?
The alert shows what Kubernetes is doing now. It does not show how the previous container ended:
PROMQL
max by (namespace, pod, container) (
kube_pod_container_status_last_terminated_reason{
namespace="checkout",
pod=~"checkout-api-.*",
container="api",
reason="OOMKilled"
}
) == 1
A result of 1 for the same namespace, pod, and container means the last recorded exit was out of memory.
Kube-state-metrics keeps that last reason as a gauge, so the value can stay visible after recovery.
It points the investigation at memory; it does not prove that every restart in the window was an OOM kill.
Is the failure repeating?
A single restart can still be transient. The restart counter shows whether the failure keeps happening:
PROMQL
max by (namespace, pod, container) (
increase(
kube_pod_container_status_restarts_total{
namespace="checkout",
pod=~"checkout-api-.*",
container="api"
}[10m]
)
)
increase() shows how much the restart counter rose over the selected range.
Repeated increases during the incident window explain why Kubernetes entered backoff.
How close is memory to the limit?
We need to know how close the container is to its memory limit, and whether that gap collapses right before each restart. This deployment allows only 128MiB, so the next query divides working-set memory by that configured limit:
PROMQL
max by (namespace, pod, container) (
container_memory_working_set_bytes{
namespace="checkout",
pod=~"checkout-api-.*",
container="api",
image!=""
}
)
/
max by (namespace, pod, container) (
container_spec_memory_limit_bytes{
namespace="checkout",
pod=~"checkout-api-.*",
container="api",
image!=""
}
)
The chart shows a repeating sawtooth: memory approaches 90% of the limit, drops when the process stops, and climbs again after each restart.
Working set is the better signal here than total usage. Total usage includes reclaimable file cache, so it can sit near the limit without a kill. Working set is closer to the memory that triggers OOMKilled for this workload.
Is the node under memory pressure?
OOMKilled can mean the container hit its own limit, or the node ran low on memory and Kubernetes started reclaiming.
To separate those cases, first find which node runs the pod, then check whether that node (or any peer) reported MemoryPressure.
PROMQL
max by (namespace, pod, node) (
kube_pod_info{
namespace="checkout",
pod=~"checkout-api-.*"
}
) == 1
The pod sits on ip-10-0-2-18.ec2.internal.
That is the node whose MemoryPressure result matters most for this incident:
PROMQL
max by (node) (
max_over_time(
kube_node_status_condition{
condition="MemoryPressure",
status="true"
}[30m]
)
)
Every node returns 0, including ip-10-0-2-18.ec2.internal.
So the host was not under node-wide memory pressure.
The kill came from the container limit itself.
Does raising the limit clear the alert?
checkout-api was healthy, then began building an in-memory cache that grows to 200MiB in 10MiB steps.
The container only allows 128MiB, so the process is killed with OOMKilled before that cache is fully allocated.
We will raise the memory limit to 512MiB so the 200MiB cache fits with room for the runtime, then check whether CrashLoopBackOff clears:
kubectl set resources deployment/checkout-api -n checkout --limits=memory=512Mi
The same waiting-reason query then stops matching.
CrashLoopBackOff drops off:
PROMQL
max by (namespace, pod, container) (
max_over_time(
kube_pod_container_status_waiting_reason{
namespace="checkout",
pod=~"checkout-api-.*",
container="api",
reason="CrashLoopBackOff"
}[2m]
)
) == 1
And the alert that started this investigation? Gone.
That is how you detect and investigate a Kubernetes CrashLoopBackOff with PromQL: from the firing alert, through OOMKilled and the limit mismatch, to a recovered alert.
Elasticsearch holds the metrics; Kibana runs the same PromQL queries you already know from Prometheus.
Try it
- Open an Observability project on Elastic Cloud Serverless, or use Elastic Stack 9.4 or later.
- In the ES|QL editor in Kibana, run the waiting-reason query against a workload you care about.
- Follow the same path from that alert to termination reason, restarts, memory versus the limit, and recovery.
For more PromQL in Elastic, see Elasticsearch supports PromQL, PromQL queries in Kibana, and Investigate Kubernetes infrastructure with PromQL.