Kubernetes node pressure: Catching the memory drop 7 minutes before the first pod eviction

Monitor clusters, pods, and nodes with full application context across EKS, GKE, AKS, and OpenShift. 

Join our on-demand webinar: Kubernetes management with Elastic & agentic AI to level up your skills. You can also start a free cloud trial, or run Elasticsearch locally.

In a staged load test, an Elastic anomaly detection job opened an alert 2 minutes 31 seconds before the kubelet set MemoryPressure and 7 minutes 29 seconds before the first failed HTTP request. By the time MemoryPressure turns true, the kubelet is already choosing which pods to evict, and it doesn’t choose the pod that used the memory. In this run, it evicted an NGINX pod using under 9 MiB and took the service down with it.

Kubernetes node pressure starts when available memory or disk on a node crosses the kubelet's eviction thresholds and an alert on that condition arrives with little time left to act. Anomaly detection learns what normal available memory looks like for each node and flags the drop while the node still has headroom, though not every drop. A second run, where the same total load arrived at once, scored 4.2 and opened no alert at all. So this walk-through runs the low_mean job alongside a headroom rule and the [Kubernetes OTel] pressure rule templates. Node metrics come from the Elastic Distribution of OpenTelemetry (EDOT) Collector, and the triage path follows whichever alert opens to the pod that changed.

Prerequisites: Elastic, Kubernetes, and EDOT versions

• An Elastic deployment with machine learning (ML) enabled and the privileges and subscription for anomaly detection. This walk-through uses Elastic Cloud Hosted 9.5.4.

• A Kubernetes cluster where you can install a Helm chart. The examples use EDOT Collector 9.5.4 with the kube-stack chart 0.16.0.

• The Kubernetes OpenTelemetry Assets integration (2.6.0 here), which supplies the dashboards and alerting rule templates used below.

You can find the companion notebook here. It reproduces the whole lab from this article, including the anomaly detection job and the eviction check, and the steps in between, and it cleans up at the end.

How Kubernetes node pressure triggers eviction

The kubelet compares eviction signals, such as memory.available and nodefs.available, against thresholds. On Linux, the default hard thresholds are memory.available<100Mi and nodefs.available<10%. A hard threshold evicts pods immediately, and a soft threshold starts a grace period first. When either one is crossed, the kubelet sets the MemoryPressure or DiskPressure node condition.

The kubelet ranks pods for eviction by whether their usage exceeds their requests, then by pod priority, and then by how far usage exceeds requests. The pod that gets evicted can be a different pod from the one that consumed the memory.

The following table shows the three signals that this walk-through uses, from earliest to latest, what each one tells you, and what to do when it fires:

Staged memory growth on one worker node

The test cluster is a two-node kind cluster on a Docker virtual machine (VM) with about 8 GiB of memory. The worker's kubelet uses a soft eviction threshold sized for that VM, set through a kubeadm configuration patch in the kind cluster config:

kubeletExtraArgs:
  eviction-soft: "memory.available<6Gi"
  eviction-soft-grace-period: "memory.available=5m"

Three workloads run on the worker:

1. A single-replica NGINX deployment with no memory request and a low pod priority. A k6 probe requests it about once per second and exports the results through OpenTelemetry (OTel).

2. A stress pod that allocates 256 MiB after two hours of normal baseline.

3. A second stress pod that allocates 1 GiB about four and a half minutes later.

These memory thresholds fit a laptop-sized lab. Set your own from node capacity and your kubelet configuration.

Step 1: Collect Kubernetes node metrics with the EDOT Collector

Follow the Kubernetes quickstart to install the EDOT Collector through the Managed OpenTelemetry Protocol (mOTLP) endpoint. Add this override to the documented Helm installation:

clusterName: node-pressure-v2
collectors:
  cluster:
    config:
      receivers:
        k8s_cluster:
          collection_interval: 10s
          node_conditions_to_report: [Ready, MemoryPressure, DiskPressure]
  daemon:
    config:
      receivers:
        kubeletstats:
          collection_interval: 10s

The override adds DiskPressure to the reported node conditions and samples every 10 seconds. Replace node-pressure-v2 with your cluster name here and in every later example.

In Discover, confirm two data streams:

1. metrics-kubeletstatsreceiver.otel-* has fresh k8s.node.memory.available values for each k8s.node.name.

2. metrics-k8sclusterreceiver.otel-* has k8s.node.condition_memory_pressure and k8s.node.condition_disk_pressure, where 0 is false and 1 is true.

Run this Elasticsearch Query Language (ES|QL) query in Discover to see the latest value of both conditions per node in the selected time range:

FROM metrics-k8sclusterreceiver.otel-*
| WHERE k8s.cluster.name == "node-pressure-v2"
| STATS memory_pressure = MAX(k8s.node.condition_memory_pressure),
    disk_pressure = MAX(k8s.node.condition_disk_pressure),
    latest_sample = MAX(@timestamp)
    BY k8s.cluster.name, k8s.node.name
| LIMIT 1000

k8s.node.memory.available tracks the value that the kubelet acts on. In the test, the metric read 6,093,116 KiB one second before the kubelet logged an eviction with available: 6092844Ki.

Step 2: Create a per-node anomaly detection job for node memory

Create the job and its datafeed in Dev Tools > Console:

PUT _ml/anomaly_detectors/node-pressure-memory-v2
{
  "analysis_config": {
    "bucket_span": "1m",
    "detectors": [{
      "function": "low_mean",
      "field_name": "k8s.node.memory.available",
      "partition_field_name": "k8s.node.name"
    }],
    "influencers": ["k8s.node.name"]
  },
  "analysis_limits": {"model_memory_limit": "32mb"},
  "data_description": {"time_field": "@timestamp"},
  "model_plot_config": {"enabled": true},
  "datafeed_config": {
    "datafeed_id": "datafeed-node-pressure-memory-v2",
    "indices": ["metrics-kubeletstatsreceiver.otel-*"],
    "query_delay": "20s",
    "frequency": "10s",
    "query": {"bool": {"filter": [
      {"term": {"k8s.cluster.name": "node-pressure-v2"}},
      {"exists": {"field": "k8s.node.memory.available"}}
    ]}}
  }
}

The following settings matter most:

• low_mean scores only buckets where the average is unusually low, so a node that frees memory never raises an anomaly.

• partition_field_name gives every node its own baseline. A 64 GiB node and an 8 GiB node never share a model.

• bucket_span of 1m suits 10-second sampling in a small cluster. A span of 5m or 15m costs less and smooths noisy nodes, and the alert arrives later.

• query_delay of 20s leaves time for metrics to arrive before a bucket is analyzed.

• model_plot_config stores the model bounds, which Single Metric Viewer draws as the expected range.

Open the job, and start the datafeed from the beginning of your node history. Replace the start value with the timestamp of your first kubeletstats document:

POST _ml/anomaly_detectors/node-pressure-memory-v2/_open
POST _ml/datafeeds/datafeed-node-pressure-memory-v2/_start
{"start":"2026-09-19T18:53:00Z"}

In Machine Learning > Manage jobs, confirm that the processed record count increases. If a job created through the API is missing from the list, select Synchronize saved objects.

A mean detector needs at least eight non-empty buckets or two hours of data, whichever is greater. For production, let the model see full workload cycles, including the busy periods that you expect.

Step 3: Create an alerting rule on the anomaly score

Open Stack Management > Rules > Create rule, select Anomaly detection, name the rule Node pressure v2 as "unusual memory headroom", and choose the job. Use the following settings:

The anomaly score is a 0–100 measure of how unusual a bucket is for that node. Tune the threshold against routine changes in your cluster.

Add a connector and an action so the alert reaches Slack, email, or PagerDuty. Include {{context.anomalyExplorerUrl}} and {{context.topRecords}} from the anomaly detection action variables in the message, so the notification links straight to the affected node.

Step 4: Add Kubernetes alerting rules for headroom and pressure conditions

A workload that grows at its usual pace can exhaust a node without producing a high anomaly score. Cover that case with an Elasticsearch query rule that uses this ES|QL query:

FROM metrics-kubeletstatsreceiver.otel-*
| WHERE k8s.cluster.name == "node-pressure-v2"
    AND k8s.node.memory.available IS NOT NULL
| STATS available_gib = MIN(k8s.node.memory.available) / 1073741824.0
    BY k8s.cluster.name, k8s.node.name
| WHERE available_gib < 6.5

Name the rule Node pressure v2: memory headroom warning, set the time field to @timestamp, select Create an alert for each row, and use a two-minute window with one-minute checks. The 6.5 GiB level sits 0.5 GiB above the scenario's soft eviction threshold. Size your own margin to how fast memory can fall between two rule runs.

Then create the condition rules from Create rule > Templates: [Kubernetes OTel] Node memory pressure and [Kubernetes OTel] Node disk pressure. Add this line after each template's TS source line, so the rule only evaluates your cluster:

| WHERE k8s.cluster.name == "node-pressure-v2"

The templates wait for three consecutive matches by default. This walk-through sets the alert delay to one match, because a soft eviction grace period can be shorter than three rule runs. Alerting rule templates in Elastic integrations covers the window, schedule, and alert delay settings in more detail.

Step 5: Triage a Kubernetes node pressure warning before the eviction

When the anomaly alert or the headroom alert opens, work through this loop:

1. Read the anomaly to identify the node and the size of the drop.

2. Find the pod whose memory changed.

3. Check which pods the kubelet would evict first.

4. Mitigate with the workload owner.

5. Verify recovery from the node and from the service.

Read the anomaly record to find the affected node

Open Machine Learning > Anomaly Explorer, and select the job and the alert's time range. Then find the k8s.node.name in the anomaly record. Compare the Actual and Typical columns. Set the table interval to Show all to see one row per bucket.

In the scenario, the 20:56 bucket averaged 6.76 GiB against a typical 7.02 GiB, and the rule evaluated a score of 30 for it. MemoryPressure was still false, and the node was above the 6.5 GiB headroom level. Every HTTP request succeeded.

The Score column in this screenshot, taken after the run, reads 2 for the same bucket. Anomaly scores are renormalized as the model sees more data, which the results section covers.

Which pod caused the Kubernetes node pressure?

Open the [Kubernetes OTel] Node Detail dashboard, and filter to the alert's cluster and node. Then sort the pod table by Avg memory working set.

The largest pod and the one that changed can be different pods. This query compares each pod's last five minutes with its earlier baseline:

FROM metrics-kubeletstatsreceiver.otel-*
| WHERE k8s.cluster.name == "node-pressure-v2"
    AND k8s.node.name == "node-pressure-v2-worker"
    AND k8s.pod.memory.working_set IS NOT NULL
    AND @timestamp >= NOW() - 1 hour
| STATS
    recent = AVG(k8s.pod.memory.working_set) WHERE @timestamp >= NOW() - 5 minutes,
    baseline = AVG(k8s.pod.memory.working_set) WHERE @timestamp < NOW() - 15 minutes
    BY k8s.namespace.name, k8s.pod.name
| EVAL recent_mib = ROUND(recent / 1048576, 1),
       baseline_mib = ROUND(COALESCE(baseline, 0) / 1048576, 1)
| EVAL change_mib = ROUND(recent_mib - baseline_mib, 1)
| WHERE recent_mib IS NOT NULL
| SORT change_mib DESC
| LIMIT 5

The 10-minute gap between the two windows keeps a new pod's first minutes out of its baseline. Two minutes after the scenario's alert, the query returns memory-change with a change_mib of 246.8 and every other pod on the node within 1 MiB of its baseline.

Next, list the pods on the node with their Quality of Service (QoS) class and priority:

kubectl get pods -A --field-selector spec.nodeName=node-pressure-v2-worker \
  -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,QOS:.status.qosClass,PRIORITY:.spec.priority

BestEffort pods and low-priority pods that exceed their requests are evicted first. If one of them is a single-replica service, you lose that service when the grace period ends.

Mitigate with the workload owner, and verify recovery

If headroom is stable and the change is expected, keep watching the node and review the alert threshold. If memory keeps falling toward the eviction threshold, agree on one of these with the workload owner:

• Stop or scale down an optional workload.

• Roll back the deployment change that increased memory use.

• Add node capacity and move the workload in a controlled way.

Then confirm recovery from both sides. Available memory returns to its typical range, and service requests succeed. Plus, no pods were evicted during the response. This query reads eviction events collected by the k8s_events receiver:

FROM logs-k8seventsreceiver.otel-*
| WHERE k8s.cluster.name == "node-pressure-v2"
    AND k8s.event.reason == "Evicted"
| KEEP @timestamp, k8s.object.name, k8s.event.reason, body.text
| SORT @timestamp
| LIMIT 1000

When the scenario runs without intervention, the kubelet evicts the NGINX pod first, with the message Container nginx was using 8884Ki, request is 0. The 1 GiB memory-escalation pod follows one second later. The evicted NGINX pod used under 9 MiB, and its eviction took the service down

The pressure condition clears later than the memory recovers. The kubelet holds MemoryPressure for its eviction-pressure-transition-period, five minutes by default, after the signal returns above the threshold.

How much warning does anomaly detection give before a pod eviction?

The anomaly alert opened 2 minutes 31 seconds before MemoryPressure and 7 minutes 29 seconds before the first failed request, but only when memory grew gradually. The scenario ran twice. In Run 1, the load continued until the kubelet evicted pods. In Run 2, both allocations started together and the load was removed after the headroom alert, because Run 2 raised no anomaly alert.

The same dashboard embeds the Single Metric Viewer of the anomaly detection job, with the model bounds and the anomaly markers for both runs. The companion notebook imports this dashboard, so you can read your own run the same way.

Three results stand out:

1. The anomaly alert led MemoryPressure by 2 minutes 31 seconds. It flagged the 256 MiB change, before the 1 GiB allocation that pushed the node over its threshold. Five of the 7 minutes 29 seconds before the first failed request are the configured soft grace period.

2. The headroom alert trailed MemoryPressure in both runs, by 41 and 36 seconds. A 1 GiB step crosses a 0.5 GiB margin and the eviction threshold in the same rule interval. In Run 2, the grace period provided the time to act.

3. The anomaly score depends on how memory changes. The same total load arriving at once scored 4.2 and opened no anomaly alert.

Anomaly scores are renormalized as the model sees more data. The Run 1 record dropped from 30 to 2.7 in Anomaly Explorer after the 1 GiB step, while its initial_record_score kept the value that the rule evaluated. A flat two-hour baseline makes a 3.7% drop stand out. Expect lower scores for the same change on nodes with days of varied history.

What the three signals cover together

This walk-through modeled available memory per node with a low_mean anomaly detection job and alerted on a record score of 25. A headroom rule and the [Kubernetes OTel] pressure templates covered the changes that the model scored low. The triage loop went from the anomaly record to the pod whose working set changed, the pods first in line for eviction, and a recovery check that includes the service.

In the staged test, the anomaly alert arrived first, and only for gradual change. Keep all three signals active, and size the soft eviction grace period as the time your team needs to respond.

Where to take the Kubernetes monitoring setup next

The following resources extend the lab and give the background for each piece:

• Add a second low_mean detector on k8s.node.filesystem.available to the same job to model disk headroom per node.

• Common ES|QL queries for Kubernetes monitoring, including container memory utilization before an OOM kill

• Alerting rule templates in Elastic integrations

• Node-pressure eviction in the Kubernetes documentation

• Generating alerts for anomaly detection jobs

• EDOT Kubernetes guide