In Part 1, we covered how Elastic's Kubernetes integration ships telemetry via the EDOT Collector into Elasticsearch. In this post, we go further with an MCP (Model Context Protocol) app server that exposes that telemetry as AI-callable tools, complete with interactive React UIs rendered inline. We'll also cover how to take it further with Elastic Workflows: automated runbooks that handle the full root cause analysis loop from alert to remediation proposal.

Observability MCP App that renders where you work

The Elastic Observability MCP App (tech preview) ships six views, one per tool. Each renders inline when the tool returns, and each surfaces opinionated next-step prompts as clickable buttons so you don't have to guess the right follow-up. MCP Apps take it further than standalone agent workflows — they render live, interactive views directly inside your chat or IDE, inline in the conversation, without a context switch to Kibana.

Cluster health rollup

Ask "what's broken?" or "give me a status report" and get a one-shot orientation: overall health badge, degraded services with reasons, top pod memory consumers, anomaly severity breakdown, and service throughput — all in one inline view.

The view adapts based on what your deployment supports. APM gives you service health. Kubernetes metrics add pod and node context. ML jobs layer in anomalies. If a signal isn't present, the view tells you what's missing rather than failing. We'll begin with a status report of the Kubernetes cluster:

Compound reports like the health summary have condensed data presentation with detail-expansion so that you get to choose the appropriate amount of information to view at once. Suggested investigation actions provide guidance for both specific information being returned, as well as orienting users to other tools to run.

Service dependency graph

Ask "what calls checkout?" or "show me the topology" and get a layered dependency graph — upstream callers, downstream dependencies, protocols, call volume, and latency per edge. Hover over an edge to highlight the full call path. Let's ask Claude to "Show me the service dependencies of the frontend":

Zoom, pan, and hover to get all the details you need to understand the complex service relationships:

Anomaly Details

Ask "what's anomalous?" or "is anything unusual in checkout?" and get one of two views, chosen automatically. If multiple entities are affected, the overview mode shows severity counts, affected entities, and a by-job breakdown. If a single entity is the focus, the detail mode shows score, actual vs. typical values with a comparison bar, deviation percentage, and a time-series when available. Let's check on the frontend service:

This isn't an ESQL query — it's an explanation of results of a previously-defined anomaly detection job. As discussed in Part 1 of this blog series, the Kubernetes integration ships with a few for you to enable. This tool will help you make the most of them.

Observe

Observe is the agent's primary access primitive for Elastic — one tool, with two modes for three different needs. Say "what is the network throughput of each of my kubernetes clusters" for a table or chart of results. Say "tell me when memory drops below 80MB" or "watch the frontend memory for anything unusual for the next 10 minutes" and it blocks until the condition fires or the window expires.

The view adapts to the mode: a results table for one-shot queries, a live trend chart with current/peak/baseline stats for sampling and threshold conditions, and a severity-scored trigger card for anomaly mode. We'll use it here to identify the busiest Kubernetes node:

Assess risk with a blast radius

Ask "what happens if this node goes down?" and get a radial impact diagram: the target node at center, full-outage deployments in red, degraded in amber, unaffected in gray. A floating summary card shows pods at risk and rescheduling feasibility. Single-replica deployments are flagged as single points of failure. What would happen if our busy node were to fail:

Alert Management

With the alert management tool, you can create, list, get info, and delete alerts. We'll create an alert next, but first use Observe once more to take a quick baseline so we know the alert will make sense:

Say "alert me if frontend memory goes above 75MB" and the agent creates a persistent Kibana alerting rule — a saved object that keeps running after the conversation ends. The view renders a live rule card: rule name, condition, window, check interval, KQL filter, and tags. Next-step buttons offer to verify the rule, watch the metric stabilize, or check current cluster health. The agent confirms what was created and where to find it in Kibana:

MCP App Architecture

The app is composed of a Node.js server, six model-facing tools wired to six single-file view resources, app-only tools for re-queries, and vite-plugin-singlefile bundling. Tools are grouped by deployment backend (Universal, APM-dependent, K8s-dependent, ML-dependent), so the agent and the user both know up front which tools apply to a given deployment instead of discovering capability gaps at call time. The repo includes six Skills as separate .zip artifacts that teach the agent when and how to call each tool.

The following diagram shows the three components that make up the app: the MCP host (Claude Desktop, VS Code, or similar), which holds the LLM and the Claude skills that teach it how to use the tools; the MCP app server, a single Node.js process that exposes the tool registry, bundles the React UI views, and handles all communication with Elastic; and the Elastic Stack itself, where Elasticsearch and Kibana serve as the live data and alerting backends.

The diagram below traces the flow of a user request: Claude reads the relevant skill file to understand which tool to call and how to fill its parameters, calls the tool which triggers server-side queries against Elasticsearch and Kibana, and receives back a compact text summary alongside a React UI resource that renders inline as an interactive widget.

From alert to root cause: Investigation Workflows

Alert rules tell you something is wrong. ML modules tell you the pattern. Elastic Workflows run the diagnosis — automatically, the moment an alert fires.

We're shipping a Kubernetes Investigation Workflow (technical preview) that triggers on a Kubernetes alert and returns a structured root cause summary before you've opened a single dashboard. The SRE who gets paged opens the alert and finds the investigation already done.

The workflow is a directed graph of steps that queries multiple data sources — primarily via Elasticsearch Query Language (ES|QL), with an Elasticsearch search for the ML anomaly lookup. if steps branch on query results, choosing which corroboration to run (ML memory anomaly vs log classification) and whether to assess upstream health (only when APM dependencies exist). AI steps appear at three points: classifying log patterns on the non-OOM path, classifying upstream degraded-vs-healthy, and a final ai.summarize that synthesizes all structured evidence into a root-cause narrative.

What the investigation workflow looks like in practice

The example execution below is based on the OpenTelemetry Astronomy Shop running against Elastic — 16 services, Kafka, PostgreSQL, all pre-instrumented via OTLP. Alongside the Shop's real telemetry, we injected a synthetic OOMKill cascade, which writes synthetic K8s and APM signals into the same namespace via the EDOT data streams. The workflow can't tell our signals from real ones — it just investigates the alert.

Alert fires: CrashLoopBackOff — app-deployment in oteldemo-esyox-default. Restart count: 6.

Workflow step 1 — Characterize pod and container context

The workflow queries K8s metrics for restart count, last termination reason, and utilization against declared limits.

Result: Last termination reason OOMKilled, restart count 6. (Note: kubeletstats utilization was unavailable for this pod/window — the workflow continues gracefully.)

Workflow branches: Termination reason is OOMKilled, so the workflow takes the memory-investigation path, not the log-investigation path.

Workflow step 2a — Consult ML anomaly results

Rather than recomputing memory trends, the workflow queries the ML anomaly index for an active k8s_pod_memory_growth anomaly.

Result: No anomaly — the spike is flagged load-driven, not a suspected leak.

Workflow step 3 — Check upstream service health

The workflow enumerates upstream dependencies from APM service_destination.1m aggregates, then compares current error rate and mean latency against the same hour 7 days ago. An AI classification step decides whether upstream degradation preceded the alert. Result: One upstream — api-gateway. Current mean latency 15.13 ms, error rate 41.26%. Baseline (168h ago): identical. Classification: upstream_healthy — within 5× error / 3× latency thresholds. Upstream is ruled out.

Workflow step 4 — Correlate with recent K8s changes

Event log for the namespace shows a tight cycle of Pulled → Created → Started → Killing → BackOff repeating roughly every 60–90 seconds. No deployments or scaling events in the past two hours.

Workflow output:

ROOT CAUSE HYPOTHESIS (confidence: high)

app-deployment is OOMKilling under memory pressure. The pod has restarted
6 times with termination reason OOMKilled. ML flagged the memory spike as
load-driven (no leak). Upstream api-gateway is healthy at current vs 7-day
baseline. This is a resource-allocation issue — the container's memory
limit is too low for its real working set.

Evidence:
- 6 restarts, last termination reason OOMKilled
- No ML memory-growth anomaly → leak_suspected=false (load-driven)
- Upstream api-gateway unchanged vs 7d baseline (15.13 ms, 41.26%) → healthy
- K8s events show tight Pulled/Created/Started/Killing/BackOff cycles;
  no deployments in the last 2h

Likely cause: memory limit insufficient for actual working set under load.

Recommended next steps:
1. Raise the app-deployment memory limit based on observed usage
2. Review application code for memory-optimization opportunities
3. Consider graceful degradation on high-load paths

Downstream impact: none identified from APM destination metrics.

The output above is what the alert looks like when you open it — not a link to a bunch of logs or a dashboard, but an answer.

The same workflow is accessible as an MCP tool from Claude Desktop, VS Code, or any MCP-compatible client. When a developer asks "why is checkout erroring?" from their IDE, the agent calls the workflow and returns the same structured output inline — same evidence, same root cause, without leaving the editor.

Here's an animated walkthrough of the workflow execution:

Observability Skill for Kubernetes investigations

We're also shipping a single, comprehensive investigation Skill (observability-k8s-investigation) that encodes the full diagnostic protocol for Kubernetes workload, node, and control-plane issues. It is an opinionated investigation methodology that includes the reasoning an experienced SRE applies instinctively but rarely writes down. You'll get this by keeping Kibana up to date, as it's baked into our AI Agent skills. It starts with governing principles that prevent the most common misdiagnoses:

• Absence of evidence is not evidence. If log queries return zero rows, report no_logs_available — don't infer a failure mode from empty results.
• OOMKilled does not mean memory leak by default. Compare current usage against a 7-day baseline before claiming a leak. The limit may simply be undersized.
• Average CPU metrics hide throttling. A pod can look healthy at 40–60% average utilization while being severely throttled at p99. Look at max and p95, not just average.
• Co-symptoms are not causes. Two services degrading simultaneously usually share an upstream cause. Only attribute causation when one service's degradation clearly precedes the other's and the delta is large.

From there, the Skill encodes a failure-mode taxonomy covering 16 distinct K8s failure patterns across workload, node, control-plane, autoscaling, and networking layers — from OOMKilled and CFS throttling through admission webhook blocks and StatefulSet split-brain. Each mode has a pivotal signal that identifies it and a corroboration checklist that confirms it.

The investigation flow follows a structured arc: orient (resolve the target pod, namespace, deployment), characterize (get restart count, termination reasons, utilization), classify (match against the taxonomy), corroborate (pull events, logs, APM, baseline comparisons), and synthesize (produce a root cause hypothesis at calibrated confidence — high, medium, or low — with explicit evidence and recommended next steps).

When two failure modes fit the evidence, the Skill names both and says which it believes is causal and why. When evidence is ambiguous, it says so. "Competing hypotheses are a valid output" is an explicit design principle — manufacturing false confidence is treated as a failure mode of the investigation itself.

Getting started

These capabilities build on the Kubernetes integration described in Part 1. Once you have dashboards and data collection running:

Step 1 — Enable investigation workflows (technical preview). Import the Kubernetes Crashloop Investigation Workflow from the Workflows page in Kibana, and optionally configure it to trigger on an alert rule.

Step 2 — Install the MCP App on an MCP-compatible client (technical preview). The MCP App for Observability repo can be found on GitHub (see the Releases page for downloads). When installing the app, don't forget to also install and enable the included skills. Access the Example MCP App's tools from your favorite agentic client — instructions are in the README at the GitHub link above.

Step 3 — Leverage the K8s Investigation Skill (technical preview). This one is a freebie if you're using Agent Builder, because it's baked into AI Agent Skills. The Skill teaches the agent when and how to call the underlying tools and workflows, ensuring consistent diagnostics in conversational contexts.

What's next

Investigation workflows diagnose what's broken in the services you're monitoring. The next question is harder: what about the services you're not monitoring?

We're thinking about topology-aware coverage intelligence — automatically discovering every workload deployed in your cluster via the Kubernetes API, cross-referencing against telemetry flowing into Elastic, and surfacing the gap. "You have 47 services. 11 have no distributed traces. Here's your riskiest blind spot." That capability is under consideration and will likely be the subject of a future post.

In parallel, we're extending workflows toward remediation — not just diagnosis but action: creating a case with the investigation summary attached, proposing a rollback for human approval, or scaling a workload to buy time while the root cause is addressed.

If you're running Kubernetes on Elastic today, tell us which investigation steps you repeat manually on every incident, which remediations you'd trust a workflow to propose, and which MCP tools we should build next. You can join the Elastic Community Discussion Here.

Kubernetes observability with Elastic is built for the operator who gets paged at 3 AM. That operator is often in a terminal, a chat tool, or an IDE. They need an answer that is grounded in what is happening in the cluster right now.

The new Elastic Kubernetes integration is built for that operator. It includes dashboards with drilldowns, alert rule templates, and ML anomaly detection jobs. Additionally Elastic also offers Agentic Investigations, that drives investigations automatically.

This blog will cover the foundational observability components (dashboards, drilldowns, alert templates, etc), while a part 2 covering the agentic investigations will cover workflows, agent skills, and MCP tools and views

The new Kubernetes integration content in this post is generally available across Elastic Cloud Hosted, Serverless, and self-managed deployments.

Dashboards designed for drill-down, not just display

The new Kubernetes dashboards are organized around a three-tier design: a cluster Overview that surfaces what needs attention at a glance, object summary pages for clusters, nodes, namespaces, workloads, and pods, and object detail pages that give you the full picture for any single entity.

Every layer connects to the next: click any entity in a summary table and choose: apply it as a filter on the current view, or open its dedicated detail page.

Here's what that looks like when something's actually wrong:

Following a restart cascade from overview to container

Overview: The Overview surfaces what needs attention across your cluster. You can see top pods by CPU, top namespaces by container restarts, and top nodes by memory utilization in one screen. When the "container restarts" panel starts climbing, you know where to look.

Namespaces Overview: Click into the flagged namespace with 1232 restarts and CPU limit utilization at 116%. The detail view plots CPU and memory against requests and limits over time. This shows both the size and duration of the overage.

Namespace Details: We can get more info on the various pods in this namespace here. Click the pod driving the restarts.

Pod Details: The pod detail dashboard is organized into capacity, metrics, and containers sections. Container restarts are flagged in red at the top of the page. Most panels are metric-driven, and the dashboard also links to correlated pod logs in Discover.

It takes four clicks to move from the Cluster Overview to container logs that explain the failure. These dashboards are starting points for your team. You can copy and customize them with ESQL visualizations.

Alert rules that fire on day one

The integration ships with pre-built alerting rule templates for states that are wrong by definition. No historical baseline or warmup period is required. Enable them during setup and they work immediately.

These rules do not ask, "Is this abnormal for this service?" They ask, "Is this a known bad state in Kubernetes?" A pod in CrashLoopBackOff is always a problem. A container killed by the kernel for exceeding its memory limit is always a problem.

Like the Kubernetes dashboards, these alerts are built on ES|QL queries. You can see that in the CrashLoopBackOff definition below. If you are new to ES|QL, you can start with the ES|QL docs.

The alert templates cover:

• CrashLoopBackOff detection - Fires when a pod's restart count exceeds a configurable threshold within a rolling window. The default catches a real restart cycle without triggering on routine restarts during a rolling deployment.
• Container OOMKilled - Surfaces kernel-level container terminations due to memory limits. These events are easy to miss in dashboards and often precede wider failures. This rule fires on any occurrence.
• Deployment below desired replicas - Fires when a deployment runs fewer replicas than declared for longer than a grace period. This catches scaling failures and partially failed rollouts.
• Pod stuck in Pending - Fires when a pod cannot be scheduled past a configurable time threshold. This surfaces node capacity problems, missing resources, and affinity failures before availability drops.
• Node disk pressure - Fires immediately when the Kubernetes DiskPressure node condition is True. A node condition is a direct state signal, not a statistical threshold.
• Persistent volume near capacity - Alerts when storage utilization crosses a configurable threshold before writes start failing.

Each template is parameterized. Adjust thresholds in the ES|QL query to match your environment. Connect notifications to PagerDuty, Slack, or another destination in your runbook.

Anomaly detection jobs with ML baselines

Alert rules catch what is definitively wrong. ML anomaly detection catches patterns that often precede failures. If you are new to this area, see the Elastic anomaly detection overview.

A pod that always runs at 85% memory utilization might be healthy. A pod that grew from 40% to 85% over twelve hours is usually not healthy. A static threshold often catches this only after an OOM kill. The ML module should catch the trajectory earlier.

The integration ships with ML module configurations that learn workload baselines and flag meaningful deviations. These jobs need 24 to 48 hours of data before results become useful. Results become more reliable as jobs continue to run.

The included modules

1. Pod memory growth anomalies

• What it learns: per-pod memory consumption pattern over time
• What it flags: Growth trajectories that are inconsistent with baseline behavior, such as a slow leak that never crosses the hard limit.
• Why ML (not alert rule): The alert rule catches the OOMKill after the fact. The ML job catches the trajectory that leads there.

2. Network I/O anomalies

• What it learns: per-pod network transmit/receive byte rate patterns
• What it flags: Unusual spikes or drops relative to the pod baseline. A spike can indicate a runaway process or unexpected load. A drop can indicate a network partition that causes the pod to go idle.
• Why ML (not alert rule): Normal network traffic varies by time of day and workload type. A batch job pod at high throughput during its normal window is expected. The same throughput outside that window can be anomalous.

3. Pod restart frequency

• What it learns: Per-workload restart rate patterns during deployments, scaling events, and routine operations.
• What it flags: Restart patterns that are anomalous relative to each workload's own history. This is distinct from the CrashLoopBackOff alert rule, which fires on a fixed threshold regardless of context.
• Why ML (not alert rule): A deployment that restarts twice during every rollout can be healthy. The same deployment restarting twice on a Tuesday afternoon may be unhealthy. The alert rule cannot distinguish these cases without workload history.

Here's our Single Metric Viewer showing anomalies triggered against a specific pod, for the memory growth job:

And here's the multi-series Anomaly Explorer view of the same job, showing detections firing across a variety of pods:

Try it yourself: the OTel Astronomy Shop

If you do not have a Kubernetes cluster ready, you can use the OpenTelemetry Astronomy Shop demo environment. It uses the same commands as Getting Started Step 2, Path A, but points to demo services. Create the namespace and secret, then run the Helm install. All 16 services, Kafka, and PostgreSQL start flowing into Elastic without instrumentation changes.

The demo ships with a built-in feature flag service, flagd, that lets you activate failure scenarios. Enable cartServiceFailure and watch the checkout-service restart cascade unfold in real time. The CrashLoopBackOff alert rule fires. The ML modules begin establishing baselines. If you have the investigation workflow enabled, it runs automatically when the alert fires.

Getting started

Step 1 - Install the Kubernetes integration. Dashboards are available immediately. No additional configuration is required.

Step 2 - Deploy data collection. There are two supported paths, both based on Helm. Choose the one that fits your deployment model.

Path A - OpenTelemetry (EDOT collector): This path uses the opentelemetry-kube-stack Helm chart with the Elastic Distribution of OpenTelemetry (EDOT) collector. Create a namespace and a secret with your endpoint and API key, then install:

kubectl create namespace opentelemetry-operator-system

kubectl create secret generic elastic-secret-otel \
  --namespace opentelemetry-operator-system \
  --from-literal=elastic_otlp_endpoint='https://<your-endpoint>.elastic.cloud:443' \
  --from-literal=elastic_api_key='<your-api-key>'

helm upgrade --install opentelemetry-kube-stack open-telemetry/opentelemetry-kube-stack \
  --namespace opentelemetry-operator-system \
  --values 'https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v9.3.2/deploy/helm/edot-collector/kube-stack/managed_otlp/values.yaml' \
  --version '0.12.4'

Path B - Elastic Agent (standalone): This path uses the elastic/elastic-agent Helm chart. The default manifest includes resource limits that may not be appropriate for production. Review the Scaling Elastic Agent on Kubernetes guide before deploying.

helm repo add elastic https://helm.elastic.co/ && \
helm install elastic-agent elastic/elastic-agent \
  --version 9.3.2 \
  -n kube-system \
  --set outputs.default.url=https://<your-endpoint>.es.elastic.cloud:443 \
  --set outputs.default.type=ESPlainAuthAPI \
  --set outputs.default.api_key=$(echo "<your-base64-api-key>" | base64 -d) \
  --set kubernetes.enabled=true

Step 3 - Enable the alert rule templates. Go to Observability > Alerts in Kibana. The Kubernetes templates are in the rule library. Enable the templates relevant to your environment, set thresholds, and connect your notification channel.

Step 4 - Let the ML modules warm up. After 24 to 48 hours, anomaly detection modules establish baselines and begin surfacing pattern-based deviations. Longer running jobs usually produce better baselines. Find results in the ML Anomaly Explorer, linked from the Kubernetes dashboards.

Steps 5, 6, and 7 - Agentic content will be covered in Part 2 (forthcoming), Kubernetes observability with Elastic: Agentic Investigations.

What's next

The next step is the layer that runs investigation workflows when an alert fires. That includes skills that encode investigation logic, tools that expose facts like ML state and topology, and MCP apps that render outputs in places like Claude Desktop or VS Code. These technical preview capabilities are available today and will be covered in Part 2 (forthcoming), Kubernetes observability with Elastic: Agentic Investigations.

If you are running Kubernetes on Elastic today, tell us which investigation steps you repeat manually on every incident. Tell us which remediations you would trust a workflow to propose. You can join the Elastic Community Discussion here.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.