Both features follow the OTel GenAI semantic conventions and work with any OTel-instrumented provider. If your framework already emits OTel GenAI span attributes, there is nothing to change.

How OTel GenAI spans are structured

A GenAI span stores everything as span attributes. A typical chat span includes:

• gen_ai.provider.name: the provider (openai, anthropic, aws.bedrock, etc.); gen_ai.system is supported as a fallback for older instrumentation.
• gen_ai.operation.name: the operation type (chat, embeddings, etc.).
• gen_ai.request.model: the model being called.
• gen_ai.usage.input_tokens: tokens consumed by the prompt.
• gen_ai.usage.output_tokens: tokens generated in the response.
• gen_ai.input.messages, gen_ai.output.messages: conversation messages.
• gen_ai.system_instructions: the system prompt.

Both features read from these attributes:

| Feature | What it shows | Where it appears | Attributes it reads | | --- | --- | --- | --- | | GenAI tab | Details (operation type, request model, provider, input and output token counts, response model, response ID) and Conversation (system prompt, user messages, model response) | Span flyout in the APM trace view, and the span flyout in Discover | Appears with any gen_ai.* attribute. Conversation needs gen_ai.system_instructions, gen_ai.input.messages, and gen_ai.output.messages | | Token count badges | Input and output token counts for each GenAI span | Every GenAI span row in the trace waterfall | gen_ai.usage.input_tokens, gen_ai.usage.output_tokens |

How to read LLM prompts and responses in the GenAI tab

When any gen_ai.* attribute is present on a span, the span flyout shows a dedicated GenAI tab next to Metadata.

The Details section shows model metadata from the span attributes: operation type, request model, provider, input and output token counts, response model, and response ID. The Conversation section shows the full exchange, populated from gen_ai.system_instructions (system prompt), gen_ai.input.messages (user messages), and gen_ai.output.messages (model response), each with a copy button so you can pull the exact prompt or response out of the trace without scraping text from a formatted table.

All raw span attributes remain accessible on the Metadata tab.

The GenAI tab is also available in the span flyout in Discover, so you can inspect LLM prompts and responses directly alongside your log and trace data without switching to the APM view.

What instrumentation does LLM tracing require?

No Kibana-side configuration is needed. The GenAI tab appears automatically when any gen_ai.* attribute is present on a span. Full Conversation support requires the OTel GenAI v1.37.0 span-attribute model: gen_ai.input.messages, gen_ai.output.messages, and gen_ai.system_instructions.

Frameworks that emit the older span-events model (gen_ai.user.message, gen_ai.assistant.message, gen_ai.choice) will show the Details metadata section but will not populate the Conversation section. For a current list of compatible instrumentations, see the OTel GenAI semantic conventions.

To verify, open the span in Discover, check that gen_ai.input.messages and gen_ai.output.messages are present, and confirm the Conversation section renders.

If your application already sends APM data to Elastic from a GenAI workload, open any GenAI span in the trace view and check for the GenAI tab.

LLM token usage in the trace waterfall

Token count badges now appear on each GenAI span row in the waterfall, so you can scan the full trace without drilling in. In agentic traces with ten or twenty LLM calls, this lets you identify which span is driving token consumption before opening any span.

Each row shows input and output token count badges sourced from gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. The row label is the span name, which instrumentation frameworks typically set to something like chat gpt-4o-mini.

Waterfall-level token counts are most useful in agentic traces where a chain of LLM calls uses different models or the same model with varying context sizes.

Which LLM providers does Elastic APM support for GenAI tracing?

Elastic APM's LLM tracing works with any OTel-instrumented provider: the GenAI tab and waterfall token counts use the same OTel attribute schema regardless of which provider your application uses. Provider is read from gen_ai.provider.name, falling back to gen_ai.system for older instrumentation.

The OTel GenAI semantic conventions that enable this provider detection are currently in a Development lifecycle. Check the release notes before upgrading instrumentation.

How to enable LLM tracing in Elastic APM

Availability: Both features are available as a Technical Preview on Elastic Serverless and will be available as a Technical Preview in Elastic Stack 9.6.

To try these features:

1. Instrument your GenAI application with an OTel SDK that follows the OTel GenAI semantic conventions (v1.37.0 or later for full Conversation support).
2. Send traces to Elastic Observability using OTLP, the Elastic APM agent, or an EDOT SDK.
3. Open the APM section in Kibana, navigate to a service that makes LLM calls, and open the trace waterfall for any transaction.

The GenAI tab appears on any span with at least one gen_ai.* attribute set; token count badges appear when gen_ai.usage.input_tokens or gen_ai.usage.output_tokens are present.

If you don't have a GenAI application to test with, the otel-genai-chat-app repository is a minimal OpenAI chat app pre-instrumented with EDOT. Set OPENAI_API_KEY and follow the EDOT commands in the README to send traces to Elastic and see both features in action.

What's next for LLM observability in Elastic APM

We're exploring cost estimation per span (estimated spend based on model pricing and token counts, surfaced in the waterfall) and tool call rendering (structured display of tool/function call inputs and outputs for agentic spans).

If you are building GenAI applications and want early access or to share feedback, reach out through the Elastic community forums or open an issue in the kibana repository.

In this article, we'll build that loop end to end. We'll use Elastic Agent Builder to feed logs, traces, metrics, alerts, and runbook context into an agent that investigates a problem, and Elastic Workflows to execute the known next steps: opening a case, sending a notification, running an enrichment query, or triggering a remediation path.

We'll work through a checkout latency regression as our example, but the same pattern applies to any incident class where your team already knows the manual steps.

Prerequisites

• Elastic Cloud or self-managed cluster running 9.4+

We'll use a checkout latency regression as the running example. If you want to follow along against your own telemetry, point the queries at your service instead. If you'd rather reproduce the exact incident, the supporting notebook simulates it and sets up the role, skill, tool, and workflow for you.

Why dashboards are not enough for incident response

Dashboards show the symptom but cannot choose the next query. A dashboard is still one of the best tools for shared situational awareness, and during an incident the hard work starts after the chart turns red and the engineer still needs to answer a sequence of operational questions.

• What changed? You need access to related deploys, alerts, logs, traces, and metrics from the same time window.
• What is affected? You need visibility into services, hosts, users, regions, SLOs, and dependency paths.
• What is the likely cause? You need evidence from telemetry combined with runbooks or previous incident cases.
• What is safe to do next? You need a bounded action that includes proper permissions, an audit trail, and a rollback path.

That last question is where a dashboard stops. It can show the symptom, but it cannot decide which query to run next, which runbook applies, or which workflow should run. Engineers provide that judgment today by moving between Kibana, chat, tickets, terminals, and internal docs.

An SRE control plane keeps the judgment with the engineer while moving more context and more action into the same operational surface.

How automated root cause analysis works: state, policy and action

Automated root cause analysis needs three things in one place: the telemetry, the permissions that bound it, and the actions it can trigger.

• State: For SRE work, that state is telemetry in Elasticsearch: logs, traces, metrics, alerts, SLOs, and related operational records.
• Policy: Policy defines who can query which data, which tools an agent can call, which workflows can run, and where a human decision is required.
• Action: Action is a set of known tools and workflows that run with explicit inputs, permissions, and outputs.

Agent Builder is useful where the system needs reasoning over messy context, and Workflows are useful where the system needs deterministic execution.

The two can work in both directions; a workflow can call an agent with an ai.agent step when it needs analysis before the next step, and an agent can call a workflow through a workflow tool when a conversation needs a repeatable action.

What Elastic Agent Builder adds to AI incident response

Agent Builder skills are reusable capability packs. A skill can include instructions, tools, and context that guide an agent through a specific task.

Reusable skill packs matter for SRE work because incident response is rarely a single query. A good investigation has a shape, and root cause analysis is a good example. The useful unit is not "ask the model what happened." It's a repeatable investigation path that starts from an alert, scopes the time window, checks the right telemetry, records uncertainty, and hands a case or workflow a structured result. The agent needs to decide which signal to start from, query the right index, compare the right time windows, inspect related services, and explain the evidence without hiding what it doesn't know.

Elastic includes a built-in Elastic AI Agent for this pattern. Built-in skills are scoped by solution, so the one that carries an SRE incident loop is observability.investigation, alongside platform skills such as dashboard-management that any solution can use. The list shows the short name, so look for investigation in the UI.

The skill ships as Markdown instructions, the same format we use for our own skill in the next section.

There are also out-of-the-box tools such as platform.core.search, platform.core.get_document_by_id, platform.core.get_index_mapping, platform.core.list_indices, platform.core.get_workflow_execution_status, and platform.core.resume_workflow_execution.

Skills guide the work, tools perform bounded operations, and the agent chooses what to use based on the task.

The scenario: a checkout latency regression

Deployment 2026.07.09.1 ships a connection pool misconfiguration to checkout-api. Within minutes, p95 latency goes from 180ms to over 2s and HTTP 500s appear for the first time. Nobody knows yet that the pool is the cause.

The evidence is spread across three signals, and no single one answers the question:

| Signal | What it shows | | :---- | :---- | | Logs | PoolExhaustedException and HTTP 500s, only on the new version | | Traces | The payment-gateway span goes from ~180ms to ~2500ms | | Metrics | Connection pool pinned at 20 of 20 right after the deploy |

Correlating those three is the work we want the agent to do. That gives us the contract for the rest of this article:

| Contract | Detail | | :---- | :---- | | Input | Service name and the alert summary | | Access | Read-only search over logs-*, traces-*, and metrics-* | | Output | Likely cause, supporting evidence, confidence, and the next safe action | | Side effect | One Observability case with the analysis attached |

Everything after this point builds one piece of that contract: the skill shapes the investigation, the tool and role bound the access, and the workflow turns the output into a case.

Build a read-only investigation skill in Elastic Agent Builder

Let's start with a read-only skill that improves investigation quality without touching production:

# Checkout latency investigation

Use this skill when an engineer asks why checkout latency, errors, or failed transactions increased.

Work through the investigation in this order:

1. Identify the affected service, environment, and time range.
2. Query traces for the slowest transactions in that window.
3. Query logs for errors from the same service and dependency path.
4. Compare current error and latency rates with the previous healthy window.
5. Return the likely cause, supporting evidence, confidence level, and the next safe action.

Do not recommend a production change unless there is a workflow tool assigned for that action.

If the evidence is incomplete, say what data is missing.

This kind of skill is a runbook execution guide, and it keeps the agent consistent across incidents. It also helps less experienced engineers ask better follow-up questions, because the agent can show the next query and explain why it matters.

Without step 4, the agent describes what's happening now and stops there. Comparing against the previous healthy window is what makes it an analysis. And the last line lets the agent say the data is missing instead of guessing.

Add AI agent observability tools with narrow permissions

Each tool should expose the smallest operation the agent needs, with the smallest data access that still supports the task.

For a read-only investigation agent, the required privileges usually start with searching observability data and inspecting index structure. The Agent Builder permissions documentation calls out that tools run against Elasticsearch data as the current user, and that read-oriented tools need index privileges such as read and view_index_metadata.

Run this in Dev Tools to create an investigation-scoped role:

POST /_security/role/agent-builder-observability-investigator
{
  "cluster": ["monitor_inference"],
  "indices": [
    {
      "names": ["logs-*", "metrics-*", "traces-*"],
      "privileges": ["read", "view_index_metadata"]
    }
  ],
  "applications": [
    {
      "application": "kibana-.kibana",
      "privileges": ["feature_agentBuilder.read", "feature_actions.read"],
      "resources": ["space:default"]
    }
  ]
}

This role gives the agent enough access to inspect telemetry while keeping production-changing actions out of scope. The monitor_inference cluster privilege is what lets the agent use the inference endpoints behind Agent Builder, and it grants no data access on its own.

When you add a custom tool, describe it in operational language, because the tool description is part of how the agent decides when to call it. Prefer descriptions like this:

Use this tool to search checkout service logs for errors in a bounded time range.

Required inputs:
- service_name
- environment
- start_time
- end_time

Return:
- matching log samples
- error counts by message
- affected host and pod names when present

A narrow tool description is much safer than a broad tool that says "search all logs for anything relevant." The agent gets a clear contract, and reviewers can reason about what the tool can and cannot do.

Use Elastic Workflows for incident response automation

Once the investigation path is useful, we can add Workflows for the actions that should be repeatable. With Workflows the control plane becomes operational because it can query more context, ask an agent to summarize evidence, open a case, notify a channel, or call a remediation endpoint. The key is that each step is explicit.

The Workflows editor gives you a validation loop before you save or run anything. Use it to catch syntax issues before the workflow writes to Cases or calls any action.

Go to Workflows > Create workflow and paste the following:

name: obs-labs-checkout-control-plane
description: Checkout regression investigation with Agent Builder and case creation.
tags: ["sre-control-plane", "agent-builder", "workflows"]

triggers:
  - type: manual

inputs:
  - name: service_name
    type: string
    default: "checkout-api"
  - name: alert_summary
    type: string
    default: "Checkout API p95 latency increased above 2s and HTTP 500s rose in the last 15 minutes after deployment 2026.07.09.1."

steps:
  - name: rca_analysis
    type: ai.agent
    agent-id: elastic-ai-agent
    create-conversation: true
    with:
      message: |
        Investigate this checkout incident as an SRE would.

        Service: {{ inputs.service_name }}
        Alert: {{ inputs.alert_summary }}

        Search the available logs, traces, and metrics for this service.
        Compare the window before and after the most recent deployment.

        Return a concise likely cause, supporting evidence, confidence, and next safe action.
        If the evidence is incomplete, say what data is missing.

  - name: case_title
    type: ai.agent
    agent-id: elastic-ai-agent
    with:
      conversation_id: "{{ steps.rca_analysis.output.conversation_id }}"
      message: "Produce a short case title for this incident. Output only the title."

  - name: case_description
    type: ai.agent
    agent-id: elastic-ai-agent
    with:
      conversation_id: "{{ steps.rca_analysis.output.conversation_id }}"
      message: "Produce a concise case description. Output only the description."

  - name: create_case
    type: cases.createCase
    with:
      title: "{{ steps.case_title.output.message }}"
      description: "{{ steps.case_description.output.message }}"
      owner: "observability"
      severity: "medium"
      tags: ["sre-control-plane", "agent-builder", "workflows"]

  - name: add_agent_analysis
    type: cases.addComment
    with:
      case_id: "{{ steps.create_case.output.case.id }}"
      comment: |
        ## Agent Builder RCA

        {{ steps.rca_analysis.output.message }}

        Agent conversation: {{ kibanaUrl }}/app/agent_builder/conversations/{{ steps.rca_analysis.output.conversation_id }}

Each step feeds the next one through its output. ai.agent steps emit a message with the model's text and a conversation_id, and cases.createCase emits the new case.id. Those three fields are the whole contract:

This workflow doesn't restart anything. It asks Agent Builder to investigate, reuses the same conversation to generate the case title and description, creates an Observability case, and writes the agent analysis back as a case comment.

Two details are worth calling out. The create-conversation: true flag on the first step is what makes the next two steps cheap: case_title and case_description pass the same conversation_id, so the agent already has the investigation in context and doesn't repeat the queries. And we use a manual trigger with a default alert_summary so you can run the sequence before attaching it to a live alert rule. In production, you'd switch the trigger to alert and attach the workflow to the rule that owns that incident class.

Run the workflow with the play button. Our run took 3 minutes and 22 seconds, with rca_analysis, case_title, case_description, create_case, and add_agent_analysis all marked as successful. Almost all of that is the investigation itself: rca_analysis alone took 3 minutes and 5 seconds, while the two case writes finished in about a second each.

The workflow then wrote an Observability case. The case list shows one open case with the generated checkout title, our tags, medium severity, and one comment.

The case detail is the audit artifact for the investigation. It records the evidence considered, the affected hosts and deployment version, the likely error type, the agent's confidence, and any missing signals.

A useful operational control plane surfaces the limits of its evidence instead of turning uncertainty into a confident claim. If your agent never reports missing data or a lower confidence, that's a signal to tighten the skill instructions, not a sign that every investigation went well.

The read-only automated root cause analysis pattern improves response quality without changing the affected service. Add remediation only when the action is well understood, narrowly scoped, and paired with verification and rollback: clearing one cache key, restarting one worker, shifting traffic away from one unhealthy instance, or running a pre-approved maintenance task.

Turn Elastic Workflows into agent tools

Workflow tools let an Agent Builder conversation trigger an Elastic Workflow and use its output. This is the bridge from "the agent recommended a next step" to "the agent can offer a known action."

A workflow tool should have a narrow description:

Use this tool only when checkout errors are caused by connection pool exhaustion on a single worker.

The workflow drains and recycles the connection pool for one worker, then verifies that the worker resumes successful requests.

Required input:
- host_name

Do not use this tool for database outages, deploy regressions affecting all hosts, or multi-host failures.

The description matters because it sets the agent's selection boundary. Note how the last line excludes the very scenario we just investigated: our incident hit both hosts and was caused by a deploy, so the agent should not offer this tool. That's the point. A workflow tool that matches every incident is a workflow tool with no boundary.

The workflow still owns execution. The agent doesn't need to know how to recycle the pool. It only needs to recognize when a known workflow may apply, collect the required input, and present the action to the engineer.

How do you stop an AI agent from changing production?

An SRE control plane should be built around blast-radius control, which means every action path needs a clear boundary. Use these checks before exposing a workflow as an agent tool:

| Check | Importance | | :---- | :---- | | Read-only first | Proves the investigation path before adding production action | | Narrow input schema | Prevents vague prompts from becoming vague actions | | Explicit permissions | Keeps the agent limited to the current user's allowed data and actions | | Dry-run or case-only mode | Lets teams review outputs before enabling remediation | | Human review for risky steps | Keeps judgment in the loop where impact is high | | Post-action verification | Confirms that the workflow improved the service instead of only executing a command |

For the review boundary itself, Workflows gives you wait steps, timeouts, and execution history, so a risky path can pause for an approval and still leave an audit trail.

An agent can help gather evidence and propose the next step, but production action should stay inside known workflow paths.

Validate against one incident class first

For a real rollout, validate the control plane against one recurring incident class. Track whether the agent finds the right evidence, whether the workflow output is complete enough for review, and whether engineers trust the recommended next step.

Use a simple validation plan:

1. Pick one alert type with a known runbook.
2. Build a read-only investigation skill for that alert.
3. Add one or two query tools with scoped index permissions.
4. Run the agent against historical incidents and compare its summary with the actual case notes.
5. Add a case-creation workflow and review the output with the owning SRE team.
6. Only then consider a workflow tool that performs a bounded remediation step.

The main failure mode is not that the model gives an imperfect summary. It's granting broad action before the investigation path is proven. Keep the first version boring, scoped, and reviewable.

Conclusion

What we covered:

• An SRE control plane combines state (telemetry in Elasticsearch), policy (permissions and review boundaries), and action (known tools and workflows).
• Agent Builder handles reasoning over messy context, while Workflows handles deterministic execution, and the two can call each other.
• A read-only investigation skill turns a runbook into a repeatable investigation path that records uncertainty instead of hiding it.
• Scoped roles with read and view_index_metadata on logs-*, metrics-*, and traces-* keep the agent useful without letting it change production.
• Reusing a conversation_id across ai.agent steps lets later steps build on the investigation instead of repeating it.
• A case-only workflow gives you the full audit artifact before you enable any remediation.
• Tool descriptions are a security boundary, not documentation, because they decide when the agent offers an action.

Resources

• Agent Builder for Observability
• Agent Builder skills
• Agent Builder tools
• Workflow tools
• AI-augmented workflows
• Root cause analysis workflow for observability alerts

Why manual incident response can't keep up at scale

Modern platforms run at a scale the human brain was never meant to triage. A single service can handle tens of thousands of transactions a second, and each one leaves a trail of metrics, logs, and traces. When something breaks, the answer is somewhere in that flood of data, but finding it by hand is slow, and slow is expensive. The AI era is making it worse, not better: data volumes are compounding, and the more tools a team adds, the more scattered the answer becomes.

Three things go wrong in the manual model:

• Exploding cost and volume. The sheer amount of telemetry is compounding observability spend, and teams often drop data to control the bill, which means the one signal that explains the outage may not even be there when they look.
• Lost and fragmented context. The real story usually lives across container logs, infrastructure events, and application traces at once. When those sit in different tools, no single platform correlates them at the moment an alert fires, and stitching them together under pressure is slow and easy to get wrong.
• Slow investigations and rushed calls. Every minute spent searching for the cause is a minute the outage continues, and a 3am restart made on a hunch can make the incident worse instead of better.

The result is long outages, stressed teams, and a Mean Time to Resolution (MTTR) that stays stubbornly high no matter how many dashboards you build. Dashboards show you the problem. They do not fix it.

What automated incident response does end to end

Autonomous SRE means the system handles the full incident loop on its own: it detects the problem, investigates the likely cause, resolves it with a corrective action, and verifies the action worked. The same loop a skilled on-call engineer runs in their head, running continuously and at machine speed.

It follows a simple cycle: detect, investigate, resolve, verify.

1. Detect. Elastic Observability continuously collects the metrics, logs, and traces from across the environment and maintains a live system model: an always-current map of your services, hosts, and the dependencies between them. It surfaces the events that actually matter instead of flooding the team with raw alerts.
2. Investigate. When something crosses a threshold, the platform pulls together the related evidence and asks an AI model, one that reads the evidence and explains it in plain language, what is happening, how confident it is, and which other services the problem will affect.
3. Resolve. Based on that diagnosis, a remediation step is carried out, for example, scaling a service, restarting it, or rolling back a recent change, either automatically or with a person's approval.
4. Verify. The system then checks whether the fix worked and the service returned to healthy, and records the whole sequence so a human can review exactly what happened and why.

The important word is loop. The system does not stop at an alert or a recommendation. It closes the gap between knowing and doing, which is precisely the gap where outages live.

How Elastic Observability correlates telemetry for AI root cause analysis

Autonomous action is only as good as the context behind it, and context is where Elastic Observability is strong. It brings the metrics, logs, and traces from across your environment into one place, then builds a live model of how those pieces connect, so the system reasons over the full story rather than a single noisy signal.

That unified view matters for three reasons:

• Better diagnosis. Grounding the AI model in real, correlated telemetry rather than a single metric means the diagnosis reflects what is actually happening, not a guess.
• Fewer false moves. When the evidence is complete, the system is far less likely to act on a symptom and miss the cause.
• A record you can trust. Every observation, decision, and action is captured, so the incident comes with a built-in audit trail instead of a gap in the story.

This is the same foundation Elastic already provides for search and security, applied to keeping services healthy.

Why automated incident response runs on Red Hat OpenShift

The reason this approach works for regulated, sovereign, and on-premises environments is that the entire loop runs inside your own Red Hat OpenShift cluster. Nothing about an incident, not the telemetry, not the diagnosis, not the action, has to leave your walls.

Four things make Red Hat OpenShift the right home for it:

• The full stack is in-cluster. Elastic Observability is deployed and managed natively on Red Hat OpenShift through the Elastic Cloud on Kubernetes (ECK) operator, so the data foundation lives next to the workloads it watches. That keeps analysis fast and keeps your data under your control. It runs on managed OpenShift (such as Red Hat OpenShift Service on AWS or Azure Red Hat OpenShift) or self-managed Red Hat OpenShift.
• The reasoning model runs locally too. Red Hat OpenShift AI serves the language model, for example, IBM Granite, inside the same cluster. The AI that diagnoses your incidents never sends your telemetry to an outside service, which is what makes the approach viable for air-gapped and sovereign deployments.
• Remediation speaks to Red Hat OpenShift natively. When the system acts, it is performing ordinary Red Hat Kubernetes operations: scaling a deployment, restarting a workload, rolling back to the last good version. These are the same actions your platform team already trusts, now triggered automatically and verified.
• It is a packaged, validated starting point. The whole pattern ships as a quickstart in the Red Hat catalog, built jointly so a team can stand it up on an existing Red Hat OpenShift environment and see the loop work without assembling the pieces from scratch.

The payoff is sovereignty without a tradeoff: you get machine-speed, AI-driven incident response and you keep every byte of telemetry and every decision inside infrastructure you already run.

From alert to verified fix

Here is the routine incident, told the way the team experiences it once the loop is in place.

A service starts to slow down. Response times climb past their healthy range and an alert fires, the same trigger that would normally start a human's night. Instead, the platform immediately gathers the relevant evidence: which service, what changed recently, the related errors and events. It hands that package to the model running on Red Hat OpenShift AI, which returns a plain-language diagnosis, a confidence level, and the blast radius, which is the set of other services this incident will affect if it is left alone.

The recommended action, if it is one you have allowed to run automatically, is then carried out as a native Red Hat Kubernetes operation; the service is scaled to absorb the load, and the platform watches the response times settle back to normal. The entire sequence, from the first alert to the confirmed recovery, is written up as a case: what happened, why, what was done, and the proof it worked. In the morning, the team reviews a finished incident report instead of reconstructing a fire drill.

The engineer's job shifts from finding and fixing to reviewing and improving. That is the real change. The work moves from reactive firefighting to oversight.

Which Kubernetes incidents can be remediated automatically

Autonomous response is most valuable on the common, well-understood incidents, the ones that are tedious rather than novel. Each maps to a native Red Hat Kubernetes action the system can take and then verify:

| When this happens | The Red Hat Kubernetes action | And confirms it by | | ---- | ---- | ---- | | A service slows down under load | Scales the deployment to add capacity | Watching response times return to normal | | A service runs out of memory and crashes | Restarts the workload cleanly | Checking it comes back healthy and stays up | | A recent change breaks something | Rolls back to the last good version | Confirming the service passes its health checks again |

These are the incidents that make up most of the pages a team gets, and they are exactly the ones a closed loop is best suited to take off their plate. Novel or high-stakes incidents still rise to a human, which is by design.

How you stay in control of automated remediation

Autonomous does not mean unaccountable. The principle is simple: the agent recommends, you decide. The system removes the toil, not the oversight, and a few rules keep humans firmly in charge.

• Every action is recorded. The full sequence, from diagnosis to action to verification, is captured as a reviewable case. Nothing happens off the record.
• Confidence is part of the decision. Remediation options are ranked by confidence, so high-confidence fixes can run automatically while low-confidence situations are routed to a person instead of acted on.
• Humans set the boundaries. You decide which actions the system is allowed to take on its own and which require a human to approve. You can keep a person in the loop wherever it matters.
• Your model, in your cluster. The reasoning runs on Red Hat OpenShift AI inside your own environment, so sensitive telemetry never has to leave your walls. For regulated and sovereign environments, that keeps the whole loop, data and decisions alike, under your control.

The goal is a system that earns trust the way a good junior engineer does: it shows its work, it knows when to ask, and it never hides what it did.

How automated incident response reduces MTTR

The headline outcome is a lower MTTR, because the slow part of an incident, the time before anyone understands it, is largely removed. Teams using Elastic Observability have put real numbers on that shift:

• WePay, a Chase company, cut the time to find customer impact during incidents by 90%, improving app performance and releasing product faster.
• DISH Media reached 100% visibility, a 10x increase in coverage, and reduced problem-resolution time for its developers.
• Accolade monitors about 400 services on Elastic Observability and doubled developer productivity.

Beyond the numbers, the change is structural:

• Consistency. The loop responds the same correct way at 3am as it does at 3pm. No fatigue, no improvised fixes.
• Capacity. Engineers stop spending nights on routine incidents and get that time back for the work only humans can do.
• Resilience. Faster, more consistent recovery means outages stay small, and small outages are the ones customers never notice.

In other words, autonomous SRE does not just make incident response faster. It changes what your best people spend their time on. That direction is also where the market is heading: Elastic was named a Leader in the 2025 Gartner Magic Quadrant for Observability Platforms.

How to get started on your existing OpenShift cluster

You do not need to rebuild your stack to begin. Because the pattern ships as a quickstart in the Red Hat catalog, the natural first step is to deploy it on an existing Red Hat OpenShift environment, with Elastic Observability as the unified view across your services, so the context for good decisions already exists. From there you can let the loop run in an advisory mode, where it diagnoses and recommends while a human approves each action, and widen its autonomy as it earns trust on the routine incidents.

Standing it up on Red Hat OpenShift means the observability stack, the reasoning model on Red Hat OpenShift AI, and the remediation agent all come up together inside one cluster, so a team can see the full detect-investigate-resolve-verify loop work end to end before committing to it broadly.

Start by letting the system watch and explain. Let it recommend. Then, incident by incident, let it act. The path to zero-touch operations is incremental, and every step along the way buys back time your team is spending on the 3am page today.

Frequently asked questions

What is autonomous SRE? Autonomous SRE is an approach where the observability platform handles the full incident loop on its own: detecting the problem, investigating the likely cause, resolving it with a corrective action, and verifying the service recovered. It automates the routine incident so engineers can focus on the novel and the high-stakes.

Why run autonomous SRE on Red Hat OpenShift? Running the loop on Red Hat OpenShift keeps the entire stack in-cluster: Elastic Observability (deployed through the ECK operator), the reasoning model on Red Hat OpenShift AI, and the remediation agent all run inside your own environment. No telemetry or decisions leave the cluster, which is what makes the approach a fit for regulated, sovereign, air-gapped, and on-premises deployments. It runs on Red Hat OpenShift Service on AWS, Azure Red Hat OpenShift, or self-managed Red Hat OpenShift.

Does autonomous SRE replace my engineers? No. It removes the repetitive toil, the routine 3am pages, and shifts engineers from finding-and-fixing to reviewing-and-improving. Humans set which actions are allowed to run automatically, approve anything sensitive, and review every action after the fact.

How does Elastic Observability decide what to do? It grounds an AI model in real, correlated telemetry, the metrics, logs, and traces from across your environment, plus a live model of how your services depend on each other. The model returns a cause, a confidence level, and the blast radius, and the platform acts only within the boundaries you set.

Is it safe to let a system take action automatically? Every action is recorded as a reviewable case, remediation options are ranked by confidence, low-confidence situations are routed to a human, and you choose which actions run automatically versus require approval. The remediation steps are ordinary Red Hat OpenShift operations, the same ones your platform team already trusts.

Can this run without sending my data to an outside service? Yes. With the reasoning model served by Red Hat OpenShift AI inside your cluster, sensitive telemetry never leaves your infrastructure. That makes the approach a fit for regulated, sovereign, and on-premises settings.

To go deeper on the architecture behind this, see the companion technical walkthrough of the in-cluster autonomous SRE stack on Red Hat OpenShift (coming soon). For the broader picture of how Elastic supports agentic and AI-driven workflows, see Elastic's Search AI Platform.

Gartner, Magic Quadrant for Observability Platforms, 7 July 2025. Gartner does not endorse any vendor, product, or service depicted in its research publications. GARTNER and Magic Quadrant are registered trademarks of Gartner, Inc. and/or its affiliates and are used herein with permission. All rights reserved.

That team still wants a model. So the request lands on an SRE's desk, and it sounds deceptively small: "Can you stand up an open-weight model for the claims team? Sixty people. It has to run on our hardware." They aren't even allowed to use a neocloud. There is a cost associated with this, but we won't explore that part. Just the part that covers running the model and observing the configuration.

Standing it up is the easy half. Four manifests and an afternoon, and you have a model answering questions. The hard part arrives a week later, when someone says "it feels slow" and you realize you have no idea whether the deployment is configured well, badly, or catastrophically — and no obvious way to find out.

This guide shows you how Elastic Observability can help you analyze the metrics from the configuration. It walks through tuning a real vLLM deployment using the metrics vLLM already emits. vLLM exposes these on a /metrics endpoint in Prometheus exposition format — no instrumentation, no sidecar, no code change — which is why every query in this guide starts from a Prometheus scrape. The goal: turn "it feels slow" into a specific, defensible decision.

Test environment: vLLM on a Kubernetes cluster using NVIDIA A10G with dcgm-exporter and Prometheus metrics

Every figure in this guide was measured on the following stack — one replica, one GPU, no autoscaling.

• Workload — Kubernetes-native load generator, scaled from 8 to 32 concurrent requests.
• Model — Qwen/Qwen2.5-3B-Instruct, bf16, --max-model-len 4096
• Engine — vLLM v0.23.0, OpenAI-compatible server, Prometheus /metrics on :8000
• GPU — NVIDIA A10G, 24 GB — an AWS g5.xlarge
• Cluster — Amazon EKS 1.30, tainted GPU node pool with minSize: 0
• Telemetry — Prometheus scraping every 15s, plus dcgm-exporter on :9400, shipped via remote_write
• Analysis — Elastic Observability, queried with ES|QL and PromQL
• Measured — 2026-07-27

Why is it hard for an SRE to self-host and tune an open-weight LLM?

The difficulty is not the deployment, it's the tuning which has no feedback loop. vLLM starts, serves, and reports success whether it's configured brilliantly or wastefully. Nothing tells you which.

When loading up the model, your manifest would have this configuration:

      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.23.0  # pin an exact release — metric names shift between versions
          args:
            - "--model=Qwen/Qwen2.5-3B-Instruct"
            - "--max-model-len=4096"         # cap context → predictable KV-cache size
            # A10G has native bf16 — do NOT add --dtype=half (T4-only).
          ports:
            - name: http
              containerPort: 8000            # OpenAI API + /metrics

But you can run into specific issues, such as: Hugging Face downloads take minutes. If your cluster expects a server to start in 30 seconds, it will assume the app is dead and kill it mid-download, putting you in an infinite crash loop.

Or you could have a hardware mismatch, and potentially degrade your model’s speed or precision because hardware architectures vary

or a bevy of other issues.

Once the model is finally running, optimizing performance is complete guesswork because default metrics don't tell you if you're being efficient.

You could use nvidia-smi, but this only understands raw hardware state, not application software logic.

Now that you have it running, a few hours to maybe even a day in, the team says "it feels slow." You are, functionally, tuning blind.

How do you tune a self-hosted vLLM deployment?

You're not an inference engineer, you own forty other services besides this one, and you don't have a forward-deployed engineer from a model vendor on call. But tuning an LLM server turns out to need exactly one skill you already have: reading telemetry and reasoning about saturation. The only missing piece is telemetry that exists and means something.

It does. vLLM emits a rich Prometheus endpoint out of the box — latency decomposed by inference phase, cache hit rates, batch occupancy, token accounting, completion outcomes. Almost nobody looks at it. The rest of this guide is how to read it.

Defining the workload: sixty users, short prompts, streaming responses

With the slowness detected and reported, you gather the usage profile of the users. Their usage pattern is as follows:

• ~60 users, but not concurrent. Realistic peak is 8–12 simultaneous in-flight requests; sustained is lower.
• Short prompts, long answers. The user pastes a paragraph and asks for a structured summary. Prompts run ~50 tokens; useful answers run 500–1,000.
• Interactive, streaming UI. Perceived speed is dominated by time to first token (TTFT), not total time — the same psychology as a chat interface.
• Heavy prompt reuse. Every request carries the same system prompt and the same policy-language boilerplate.

From that, you write down actual service objectives — the step most self-hosted LLM projects skip:

• TTFT p95 < 300 ms
• inter-token latency < 50 ms (≥ 20 tokens/sec, faster than reading speed)
• zero queueing at 12 concurrent
• error + abort rate < 0.5%

Those four numbers are the point of everything that follows. Without them, "it feels slow" has no answer. With them, every metric below passes or fails a stated bar.

How do you get Prometheus metrics out of vLLM and the GPU?

There are two sources, and you need both.

• vLLM reports on itself — latency by phase, cache hit rates, batch occupancy, token counts — on /metrics at its serving port, with no adapter and no instrumentation work.
• The GPU reports separately, through NVIDIA's dcgm-exporter on :9400 (NVIDIA Data Center GPU Manager (DCGM) is a suite of tools and libraries designed to comprehensively manage, monitor, and diagnose enterprise-grade NVIDIA GPUs in clusters and data centers). vLLM tells you what the engine thinks is happening; DCGM tells you what the card is actually doing. Step 6 is built entirely on the gap between those two answers.

On this EKS cluster that means three things running side by side:

• vLLM as a plain Deployment on the tainted g5.xlarge GPU node pool. For one model on one card, a Deployment and a Service is the whole architecture.
• dcgm-exporter as a DaemonSet, pinned to the same GPU nodes.
• A Prometheus server on a CPU node, scraping both endpoints every 15 seconds.

Nothing here is AWS-specific. The production version is the same manifests on an on-prem cluster with L40S or H100 nodes — which is the point of doing this on Kubernetes rather than on a vendor's platform.

What about KServe and llm-d?

Neither was run for this guide, and neither changes where the metrics come from. KServe and llm-d sit on top of vLLM rather than replacing it — vLLM is still the engine, so /metrics is still the source of every number here. Each adds its own layer on top (KServe: autoscaler and revision metrics; llm-d: router and cache-routing metrics), but the inference telemetry underneath is identical.

What they change is when you need them — and each promotion is triggered by a metric you're already collecting:

How do you ship the vLLM metrics and DCGM metrics to Observability

A Prometheus scraping inside the cluster only holds hours of data — the metrics have to reach a store you can still query next week. There are two paths for that, and they are not equivalent.

Path A — OpenTelemetry Collector. Puts inference metrics into the same pipeline as your traces and logs. One collector, one auth path, one mental model. The cost is that Prometheus metrics sent through OTLP get normalized: the stored schema ends up neither purely Prometheus nor purely OTel, and metric names shift.

Path B — native Prometheus remote_write. Stands up a small Prometheus that scrapes both endpoints and pushes to a backend speaking the remote-write protocol. Names and labels land untouched, _sum / _count / _bucket histogram parts stay intact, and existing queries keep working.

For a tuning exercise, choose Path B. That's what produced every number in this guide. The reason is narrow but decisive: tuning means comparing against the vLLM documentation and the vLLM community, and both speak in exact metric names. When your chart says vllm:kv_cache_usage_perc, you can search for it.

The deployment is two YAML files — a Prometheus Deployment with two scrape jobs and a remote_write block, plus a Secret holding the backend credential. In this build the destination was an Elastic Serverless project, which exposes a Prometheus remote-write endpoint and lands data in a time-series data stream, metrics-vllm.prometheus-inference.

Two things cost me real time. If your backend has a separate ingest host for OTLP versus its main API, remote-write usually lives on the main API host, not the ingest one — pointing at the wrong one returns a 404 that looks like a path error. And the credential needs index-write privileges, not just ingest authentication; a key that works fine for OTLP can authenticate successfully and then 403 on every sample. Check prometheus_remote_storage_samples_failed_total on the Prometheus itself before looking anywhere else.

How do you confirm vLLM metrics landed in Elastic?

Once the pipeline is up, look at the field list. Roughly 127 metric series arrive from a single vLLM pod plus DCGM:

This screen is more useful than it looks. Scanning the field list is how you confirm your vLLM version's exact metric names — they do shift between major vLLM releases, and a dashboard built against the wrong names fails silently by returning nothing rather than erroring.

Two gotchas when querying vLLM metrics

Both produce results that look like "the metrics aren't working" when the pipeline is perfectly healthy.

vLLM metric names contain a colon (vllm:num_requests_running), so they need escaping in most query languages. More insidiously, if you filter by metric name across several metrics and then aggregate only one of them, you get rows back — full of nulls, with no error. Each Prometheus metric lands in its own field, so naming the field is the filter; you don't need the name predicate at all.

Counters need rate functions, gauges don't. vllm:generation_tokens_total is cumulative and monotonic — taking a max of it gives the pod's lifetime total, not its throughput. Gauges like vllm:num_requests_running, vllm:num_requests_waiting and vllm:kv_cache_usage_perc are instantaneous and want max or average. Mixing these up produces charts that are wrong but plausible, which is considerably worse than charts that are empty.

Which vLLM Prometheus metrics actually matter?

A reference for the metrics used in this guide, what each tells you, and the condition worth watching. Names are as vLLM emits them; the DCGM_FI_* series come from dcgm-exporter.

| Metric | Type | What it tells you | Watch for | |---|---|---|---| | vllm:time_to_first_token_seconds | Histogram | TTFT — how long before the first token streams | p95 above your interactive bar (300 ms here) | | vllm:inter_token_latency_seconds | Histogram | Streaming speed after the first token | Above ~50 ms is slower than reading speed | | vllm:e2e_request_latency_seconds | Histogram | Total request time | Rising while TTFT is flat = decode or workload change | | vllm:request_queue_time_seconds | Histogram | Time waiting for admission | Earliest saturation signal — any sustained rise | | vllm:request_prefill_time_seconds | Histogram | Time processing the prompt | Dominant share = prefill-bound workload | | vllm:request_decode_time_seconds | Histogram | Time generating tokens | Dominant share = memory-bandwidth-bound | | vllm:num_requests_running | Gauge | Requests currently being decoded | Batch occupancy | | vllm:num_requests_waiting | Gauge | Requests queued for admission | Sustained non-zero = add a replica | | vllm:kv_cache_usage_perc | Gauge | Occupancy of the KV block pool — not VRAM | Autoscaling trigger (~60%) | | vllm:prompt_tokens_total + vllm:prompt_tokens_cached_total | Counters | Prefix-cache hit rate | A drop means routing scattered your prefixes | | vllm:generation_tokens_total | Counter | Output throughput in tokens/sec | Headline throughput number | | vllm:request_prompt_tokens + vllm:request_generation_tokens | Histograms | Per-request token sizes; their ratio is the workload's shape | A moving ratio means the workload changed character | | vllm:iteration_tokens_total | Histogram | Tokens advanced per forward pass | Near 1.0 with concurrency = batching broken | | vllm:request_success_total{finished_reason} | Counter | Completion outcomes | error/abort = SLO; length share = truncation | | http_requests_total{status} | Counter | Server-level requests | Catches 4xx and malformed requests vllm:* never sees | | DCGM_FI_DEV_FB_USED / _FB_FREE | Gauge | Physical VRAM | Capacity planning only — never alert on it | | DCGM_FI_DEV_GPU_UTIL | Gauge | "A kernel is resident" | Not a measure of useful work | | DCGM_FI_PROF_PIPE_TENSOR_ACTIVE | Gauge | Tensor-core activity | Low here + high DRAM = memory-bound | | DCGM_FI_PROF_DRAM_ACTIVE | Gauge | Memory-bandwidth activity | High = the bottleneck is bandwidth | | DCGM_FI_DEV_POWER_USAGE | Gauge | Watts drawn | Pairs with throughput for tokens-per-watt |

Step 1: Where does vLLM latency go? TTFT, prefill, and decode decomposed

Decompose total latency into queue, prefill, and decode before optimizing anything. vLLM reports all three separately, and they have completely different fixes. This is the single most valuable chart in the setup.

The query averages each phase's cumulative time by request count in the same window — in Prometheus terms, rate(vllm:request_prefill_time_seconds_sum) / rate(vllm:e2e_request_latency_seconds_count), and the same for queue and decode:

TS metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS reqs = SUM(RATE(`metrics.vllm:e2e_request_latency_seconds_count`)),
        q_s  = SUM(RATE(`metrics.vllm:request_queue_time_seconds_sum`)),
        pf_s = SUM(RATE(`metrics.vllm:request_prefill_time_seconds_sum`)),
        dc_s = SUM(RATE(`metrics.vllm:request_decode_time_seconds_sum`))
    BY minute = BUCKET(@timestamp, 1 minute)
| EVAL queue_ms   = ROUND(q_s  / reqs * 1000, 2),
       prefill_ms = ROUND(pf_s / reqs * 1000, 1),
       decode_ms  = ROUND(dc_s / reqs * 1000, 1)
| KEEP minute, queue_ms, prefill_ms, decode_ms
| SORT minute ASC

At 8 concurrent requests on the A10G:

minute    | queue_ms | prefill_ms | decode_ms | e2e_ms  | ttft_ms | inter_token_ms
22:55:00  | 0.01     | 37.98      | 1664.69   | 1715.18 | 50.81   | 16.23
22:56:00  | 0.01     | 40.99      | 1670.44   | 1723.78 | 53.50   | 16.20

Read it against the objectives:

• TTFT is 51 ms against a 300 ms target. Passing, with almost 6× headroom. Perceived responsiveness is not the problem, whatever was said in the meeting.
• Inter-token latency is 16 ms — about 62 tokens/sec against a 50 ms / 20 tok-s bar. Text arrives roughly three times faster than a person reads it.
• Queue time is 0.01 ms. Nothing is waiting for admission; the engine has capacity to spare at this concurrency.
• Decode is 1,665 ms against 38 ms of prefill — 97% of the time is decode.

That last line is the finding. Every optimization aimed at prefill is worthless for this workload. Chunked prefill, prompt compression, a faster attention kernel for long contexts — all real techniques, all irrelevant when prefill is 2% of the time. Decode is memory-bandwidth-bound, so the levers that would actually move it are quantization, tensor parallelism across two cards, or a smaller model. A single chart eliminated the wrong shopping list.

Watch vllm:request_queue_time_seconds specifically. It is the earliest saturation signal in the entire vLLM metric set — queue time climbs before vllm:num_requests_waiting becomes visibly non-zero, because a request can wait milliseconds for admission without ever registering as queued at scrape time. If you alert on one thing from this section, alert on queue time crossing a small absolute threshold.

Step 2: Is vLLM using the GPU efficiently? KV cache, prefix caching, and batch occupancy

Four metrics answer this: prefix-cache hit rate, tokens per iteration, KV-cache occupancy, and running-vs-waiting requests. Latency tells you the experience is good; these tell you whether you're overpaying for it.

TS metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS ptok    = SUM(RATE(`metrics.vllm:prompt_tokens_total`)),
        cached  = SUM(RATE(`metrics.vllm:prompt_tokens_cached_total`)),
        gen     = SUM(RATE(`metrics.vllm:generation_tokens_total`)),
        it_s    = SUM(RATE(`metrics.vllm:iteration_tokens_total_sum`)),
        it_c    = SUM(RATE(`metrics.vllm:iteration_tokens_total_count`)),
        running = MAX(`metrics.vllm:num_requests_running`),
        waiting = MAX(`metrics.vllm:num_requests_waiting`),
        kv      = MAX(`metrics.vllm:kv_cache_usage_perc`)
    BY minute = BUCKET(@timestamp, 1 minute)
| EVAL prefix_cache_hit_pct = ROUND(cached / ptok * 100, 1),
       tokens_per_iteration = ROUND(it_s / it_c, 2),
       gen_tokens_per_sec   = ROUND(gen, 1),
       kv_cache_pct         = ROUND(kv * 100, 3)
| KEEP minute, prefix_cache_hit_pct, tokens_per_iteration,
       gen_tokens_per_sec, running, waiting, kv_cache_pct
| SORT minute ASC

minute   | prefix_cache_hit_pct | tokens_per_iteration | gen_tok/s | running | waiting | kv_cache_pct
22:55:00 | 32.5                 | 10.36                | 479.1     | 8.0     | 0.0     | 0.255
22:59:00 | 32.3                 | 10.34                | 480.0     | 8.0     | 0.0     | 0.121

A 32% prefix-cache hit rate is a third of all prefill work simply not done. The claims team's requests share a system prompt and policy boilerplate, and vLLM's automatic prefix caching recognizes that. This is a direct argument for raising prompt standardization: the more the application puts shared context in a consistent leading position, the higher this climbs and the cheaper every request gets. It is also the number that will crater the day a naive round-robin load balancer sits in front of two replicas — precisely the condition that justifies llm-d's cache-aware routing.

tokens_per_iteration ≈ 10.3 with 8 concurrent requests is continuous batching working correctly. Each forward pass through the model advances about ten sequences at once. If this sat near 1.0 with multiple requests in flight, batching would be broken and you'd be paying full model-forward cost per token per user. This metric proves you're getting vLLM's core value.

What does vllm:kv_cache_usage_perc actually measure?

vllm:kv_cache_usage_perc reports occupancy of vLLM's pre-allocated KV block pool — not physical GPU memory. At startup, vLLM reserves a fraction of VRAM (governed by --gpu-memory-utilization, default 0.9) and carves a KV block pool out of that reservation. This gauge reports how full that pool is.

That's why it read 0.25% here. Eight concurrent requests holding ~150 tokens each barely touch an A10G's block budget. The pool is large, and correctly so. Push the same server to 32 concurrent requests with 512–1,024 token generations and it moves — to about 2.8%. Still small.

The instinct is to read a number that low as "the cache is broken" or "I've massively over-provisioned." Both are wrong. Treating this gauge as a VRAM proxy is the most common self-hosted vLLM configuration error I see. Step 6 shows exactly how far apart the two are.

Step 3: What shape is your vLLM inference workload?

Confirm the workload is what you think it is before tuning anything. This is the panel that explains a latency "regression" that isn't your fault.

minute   | requests_per_min | avg_prompt_tokens | avg_generated_tokens | avg_max_tokens | gen_to_prompt_ratio
22:55:00 | 282.2            | 49.6              | 103.6                | 103.6          | 2.09
23:01:00 | 276.0            | 50.6              | 103.0                | 103.0          | 2.04

The generation-to-prompt ratio is 2.04 — this workload writes twice as much as it reads. That single ratio is the explanation for Step 1's 97%-decode finding, and it holds for any summarize-and-draft use case. If the team later adds a long-document RAG feature, prompts jump to thousands of tokens, the ratio inverts, the workload becomes prefill-bound, and the correct tuning changes completely. Watching this ratio is how you learn your workload changed character before someone files a ticket.

Now the column that should bother you: avg_generated_tokens equals avg_max_tokens exactly. Every request is stopping because it hit its token ceiling, not because the model finished its thought. The screenshot shows the same pattern holding as generation lengths scale to ~685 tokens against a ~777 ceiling.

In a load test that's an artifact of the generator. In production, that number is users getting cut off mid-sentence — and it is invisible in every latency metric you have. Which brings us to the metric that catches it.

Step 4: Are vLLM requests actually succeeding? Checking finished_reason

Break vllm:request_success_total down by its finished_reason label. This is the closest thing self-hosted inference has to an application-level SLI, and it catches a failure mode no latency chart can.

TS metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS completions_per_min = ROUND(SUM(RATE(`metrics.vllm:request_success_total`)) * 60, 2)
    BY minute = BUCKET(@timestamp, 1 minute), finish_reason = labels.finished_reason
| SORT minute ASC, finish_reason

Five outcomes, each meaning something different operationally:

| finished_reason | What it means | What to do about it | |---|---|---| | stop | The model finished naturally | This is the number you want large | | length | Truncated at max_tokens | High share means users are cut off — raise the ceiling, or shorten the ask | | abort | The client disconnected first | Users giving up, or a proxy timeout shorter than your generations | | error | The engine failed | Your hard SLO signal. Should be flat zero | | repetition | Degenerate looping output | A sampling-parameter problem, not an infrastructure one |

Under the small load generator the split was 100% length — expected, since it requested a fixed ceiling. In the screenshot, at a mixed load, stop and length run side by side at roughly 51 and 32 completions/min. That mix is the healthy shape: most requests finishing on their own, a minority hitting the ceiling.

The lesson generalizes. error and abort are what you page on. But the stop-to-length ratio is what you review weekly, because drift toward length means answers are being truncated and no latency dashboard on earth will tell you.

One blind spot to close: vllm:* metrics only count requests the engine accepted. Malformed JSON, 4xx, auth failures and dropped connections never reach it. Those live in http_requests_total with status and handler labels — worth a panel beside this one, because "the model is broken" reports frequently turn out to be the gateway in front of it.

Step 5: What NVIDIA DCGM metrics say the GPU is actually doing

Use NVIDIA DCGM as an independent witness to vLLM's own account. Everything so far is the engine describing itself; DCGM describes the silicon.

FROM metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS gpu_util_pct    = MAX(`metrics.DCGM_FI_DEV_GPU_UTIL`),
        mem_bw_util_pct = MAX(`metrics.DCGM_FI_DEV_MEM_COPY_UTIL`),
        vram_used_mib   = MAX(`metrics.DCGM_FI_DEV_FB_USED`),
        vram_free_mib   = MIN(`metrics.DCGM_FI_DEV_FB_FREE`),
        power_w         = ROUND(MAX(`metrics.DCGM_FI_DEV_POWER_USAGE`), 1),
        temp_c          = MAX(`metrics.DCGM_FI_DEV_GPU_TEMP`)
    BY minute = BUCKET(@timestamp, 1 minute)
| SORT minute ASC

→ 100% GPU util · 21,483 MiB used / 1,352 MiB free · 240 W · 73 °C

100% utilization and 94% VRAM. Under the conventional reading, this card is maxed out and it's time to ask for more hardware. That reading is wrong.

Why is GPU utilization a misleading metric for LLM inference?

DCGM_FI_DEV_GPU_UTIL means "a kernel is resident on the device," not "the device is doing useful work." It reads 100% for a perfectly-tuned server and 100% for a badly-tuned one, so it cannot distinguish them. The DCGM profiling counters can:

gr_engine_active 99.8%  ·  tensor_active 16.6%  ·  dram_active 80.4%

Read those three together. The GPU's compute engine is busy essentially all the time — but its tensor cores, the units that do the actual matrix math, are active only 16.6%, while DRAM is active 80.4%. The card is not computing. It is waiting on memory.

This is independent, hardware-level confirmation of what Step 1 inferred purely from timings: decode is memory-bandwidth-bound. Two entirely different instruments, two different layers of the stack, one conclusion — the difference between a hypothesis and a finding.

It also permanently retires GPU utilization as a capacity metric for LLM inference. If your GPU capacity planning rests on DCGM_FI_DEV_GPU_UTIL — and most does — it rests on nothing.

Step 6: vLLM KV cache vs GPU VRAM, and why they disagree

Put vllm:kv_cache_usage_perc and physical VRAM usage on one 0–100% axis. They describe the same GPU memory, they sit at opposite ends of the chart, and both are correct.

minute   | running | gen_tok_s | kv_cache_pct | vram_used_pct | gpu_util | dram_active_pct | tensor_active_pct | tokens_per_watt
00:05:00 | 31      | 1627.5    | 2.82         | 94.1          | 100.0    | 80.4            | 16.6              | 6.80

KV cache at 2.8%. VRAM at 94.1%.

vLLM pre-allocates a large fraction of VRAM at startup — governed by --gpu-memory-utilization, default 0.9 — and carves its KV block pool out of that reservation. vllm:kv_cache_usage_perc reports occupancy of the pool. DCGM reports what the driver sees, which is the whole reservation, whether or not it's holding anything.

The operational consequences are precise, and they're the practical payoff of the entire exercise:

• Autoscale on vllm:kv_cache_usage_perc and vllm:num_requests_waiting. These describe admission capacity — whether the engine can take another request right now.
• Capacity-plan on VRAM. This describes physical space — whether a second model could ever fit on this card. (It can't. 1.3 GB free.)
• Never alert on VRAM. It will page you at 3 a.m. for a healthy, mostly-idle server, every single night, forever.

And tokens_per_watt — generated tokens divided by power draw, 6.8 here — is a genuine cost-efficiency metric. It's comparable across GPU models, batch settings and quantization levels in a way that neither latency nor utilization is. When you go back to Finance for card number two, this is the number that makes the argument: at 32 concurrent we sustain 1,627 tokens/sec at 240 watts, and here's what that becomes on an L40S.

vLLM tuning decisions: what the SRE does with these Prometheus metrics

Six steps, thirty minutes, one server. The verdict against the stated objectives:

| Objective | Measured | Verdict | |---|---|---| | TTFT p95 < 300 ms | 51 ms | Pass, 6× headroom | | Inter-token < 50 ms | 16 ms (≈62 tok/s) | Pass | | Zero queueing at 12 concurrent | queue_ms 0.01, waiting 0 at 8; still 0 at 32 | Pass, large margin | | Error + abort < 0.5% | 0% | Pass |

The configuration is correct for this department, and the department is over-provisioned rather than under-provisioned. That's a defensible, evidence-backed answer to "it feels slow" — and it redirects the investigation to the app, the gateway, or the prompt, which is where the problem actually is.

The concrete follow-ups, each tied to a metric rather than a hunch:

1. Stop optimizing prefill. — vllm:request_decode_time_seconds vs vllm:request_prefill_time_seconds. Decode is 97% of the time against prefill's 2%, confirmed twice. Chunked prefill and prompt compression are off the table for this workload.
2. If more throughput is needed, quantize before buying hardware. — DCGM_FI_PROF_PIPE_TENSOR_ACTIVE (16.6%) vs DCGM_FI_PROF_DRAM_ACTIVE (80.4%). The bottleneck is memory bandwidth, not compute, so an FP8 or AWQ build of the same model is the highest-leverage single change: it moves fewer bytes per token, which is exactly the constrained resource.
3. Raise the client-side max_tokens ceiling. — vllm:request_success_total{finished_reason}. Every request finishing on length rather than stop is a user getting cut off mid-answer. This is the one finding that's a live user-experience defect. You need to increase the prompt max_token limit.
4. Standardize the prompt prefix. — vllm:prompt_tokens_cached_total over vllm:prompt_tokens_total, 32% today. But it should be better (more like 70%) More shared boilerplate in a consistent leading position raises it, and it's free.
5. Set the autoscaling trigger now, before it's needed. — vllm:kv_cache_usage_perc and vllm:num_requests_waiting. Scale when the first crosses ~60% or the second stays above zero. Do not scale on DCGM_FI_DEV_GPU_UTIL — it's pinned at 100% regardless.
6. Alert on queue time, not on VRAM. — vllm:request_queue_time_seconds is the earliest true saturation signal; DCGM_FI_DEV_FB_USED is a constant that looks like an emergency.
7. Revisit when the workload changes shape. — vllm:request_generation_tokens over vllm:request_prompt_tokens, 2.04 today. When the RAG feature ships that ratio inverts, the workload becomes prefill-bound, and half of this analysis needs redoing. The chart tells you the day it happens.

Notice that most of the metrics are looking at the vLLM metrics not the GPU metrics in helping optimize. These are still within the limit of a single service, but when you get vllm:num_requests_waiting to persistently non-zero, then you need to run KServe or you can use KEDA and HPA autoscaling. But you get the metrics to help you determine or allow KServe to scale. So you can see that understanding these metrics are crucial to tuning the inference service.

Elastic Observability can provide this to you.

Why self-hosted LLM tuning is an SRE problem, not an ML problem

Self-hosting an open-weight model is not primarily an ML problem. It is a capacity and saturation problem — something SREs have been extremely good at for twenty years. The blocker was never skill. It was that the telemetry sat unexamined on a /metrics endpoint nobody scraped, in a schema nobody had mapped to the questions they actually had.

Once it's collected, the reasoning is familiar work in unfamiliar clothes:

• Decompose latency by phase before optimizing anything (queue / prefill / decode).
• Distinguish the logical resource from the physical one (KV block pool ≠ VRAM), and know which each metric describes.
• Never trust a single-source utilization number — corroborate the engine's account with the hardware's.
• Tie every knob to a metric and every metric to a stated objective, so tuning converges instead of wandering.

For a team that isn't allowed to send its data anywhere, that difference — between running a model and operating one — is the whole ballgame. The department gets a capability it's otherwise locked out of, and the SRE gets to answer questions about it with numbers.

Frequently asked questions

What does vllm:kv_cache_usage_perc measure? It measures occupancy of vLLM's pre-allocated KV block pool, not physical GPU memory. vLLM reserves a fraction of VRAM at startup (--gpu-memory-utilization, default 0.9) and carves the KV pool from that reservation. In this deployment it read 2.8% while DCGM reported 94.1% VRAM used on the same card at the same moment. Use it as an autoscaling signal; use VRAM for capacity planning.

Why is my vLLM deployment decode-bound? Because the workload generates more tokens than it reads. Compare vllm:request_decode_time_seconds against vllm:request_prefill_time_seconds, and check the generation-to-prompt token ratio. In this deployment the ratio was 2.04 — twice as many output tokens as input — which produced 1,665 ms of decode against 38 ms of prefill. Decode is memory-bandwidth-bound, so quantization, tensor parallelism, or a smaller model help; prefill optimizations do not.

Should I autoscale vLLM on GPU utilization? No. DCGM_FI_DEV_GPU_UTIL means a kernel is resident on the device, not that the device is doing useful work — it reads 100% for both a well-tuned and a badly-tuned server. Autoscale on vllm:kv_cache_usage_perc (around 60%) or on vllm:num_requests_waiting staying above zero, since those describe whether the engine can admit another request.

Why does my GPU show 100% utilization when it isn't fully used? Because GPU utilization only reports kernel residency. Check the DCGM profiling counters instead: in this deployment DCGM_FI_PROF_GR_ENGINE_ACTIVE was 99.8% while DCGM_FI_PROF_PIPE_TENSOR_ACTIVE was only 16.6% and DCGM_FI_PROF_DRAM_ACTIVE was 80.4%. That combination means the GPU is waiting on memory bandwidth rather than computing.

How do I get vLLM metrics into Prometheus? vLLM already exposes Prometheus exposition format on /metrics at its serving port — no adapter or instrumentation needed. Point a Prometheus scrape job at the vLLM Service, add a second job for dcgm-exporter on :9400, and use remote_write to ship to long-term storage. Sending through an OpenTelemetry Collector also works but normalizes the metric names, which makes them harder to match against vLLM documentation.

What TTFT should I target for an interactive LLM application? For a streaming chat-style interface, a p95 time-to-first-token under 300 ms feels immediate, and inter-token latency under 50 ms (about 20 tokens/sec) outpaces reading speed. This deployment measured 51 ms TTFT and 16 ms inter-token latency on a 3B model on a single NVIDIA A10G, leaving roughly 6× headroom.

Why are all my vLLM requests finishing with length? Because they're hitting the max_tokens ceiling instead of the model choosing to stop. Break vllm:request_success_total down by its finished_reason label: a high length share means answers are being truncated mid-sentence. This is invisible in every latency metric, so review the stop-to-length ratio regularly and raise the client-side ceiling if it drifts.

When should I move from a plain vLLM Deployment to KServe or llm-d? Move to KServe when vllm:num_requests_waiting is persistently non-zero at peak and you need replicas to appear without human intervention. Move to llm-d when your prefix-cache hit rate collapses across replicas — a sign that load balancing scattered conversations that shared a prefix — or when prefill time starts stealing measurably from decode.

Why multi-team log routing needs structure

Multi-team Elasticsearch deployments typically converge on a single shared index, which works until teams need different retention periods, sharding settings or processor pipelines.

Before Streams, you had to set up your ingestion scripts to send data to different indices, or data streams, or use the reroute processor to define the data destination based on some field name.

With AI Partitioning, you let the data arrive first. Then the AI analyzes what showed up, reviews the suggestions it proposes, refines them as needed, and applies them. The result is a set of wired child streams that inherit retention, processors, and schema from the parent, while still allowing you to override any of those per child.

What you need before using Streams AI Partitioning

Before running the example, three things need to be in place:

1. Wired Streams enabled. On Elastic Cloud Serverless and Elastic Cloud Hosted 9.4+, wired streams are on by default. If you upgraded from an earlier version, open the Streams app and confirm the toggle is on under Settings.
2. The Elastic Managed LLM connector. Go to Stack Management > Connectors > Create connector > Elastic Managed LLM. Keep in mind the following considerations:
3. This connector ships preconfigured and does not require an external account or API key.
4. Any generative AI connector works with the feature.
5. Note that Elastic Managed LLMs incur a cost per million tokens for input and output.
6. The account you use needs the manage_inference cluster privilege (the built-in inference_admin role grants it).
7. The Managed OTLP endpoint URL and an API key. Open Cloud Console > Manage > Application endpoints > Ingest. Copy the Managed OTLP endpoint URL and generate an API key from that same panel.

Once these are ready, open Observability > Streams and confirm a logs.otel wired stream is listed. That stream is the parent we will partition.

If Streams does not appear in the sidebar, your Kibana space may be using a solution view other than Observability. You can change it in Stack Management > Spaces > edit your space > set Solution view to Observability.

Generating multi-team log data

Our example uses three apps produced by teams from the same fictional company:

• payments-api: structured JSON with transaction_id and amount_cents. Sensitive data, long retention needs.
• checkout-web: JSON with cart_id and customer_id. Mostly INFO and ERROR.
• notifications-worker: less structured, with recipient and channel. High volume.

We use a Python script with the OpenTelemetry Python SDK to emit logs for all three teams over OTLP directly to the Managed OTLP endpoint. The full code, including setup and execution, is in the companion notebook.

Each team is defined with a service name, a set of message templates, and a function that generates team-specific attributes:

TEAMS = {
    "payments": {
        "service": "payments-api",
        "messages": [
            ("INFO", "charge captured tx={tx} amount_cents={amt}"),
            ("ERROR", "charge declined tx={tx} reason=insufficient_funds"),
            ("INFO", "refund issued tx={tx} amount_cents={amt}"),
        ],
        "extra": lambda: {
            "transaction_id": f"tx_{random.randint(10000, 99999)}",
            "amount_cents": random.randint(100, 50000),
        },
    },
    "checkout": {
        "service": "checkout-web",
        "messages": [
            ("INFO", "cart updated cart={cart} customer={cust}"),
            ("INFO", "checkout started cart={cart} customer={cust}"),
            ("ERROR", "checkout failed cart={cart} stage=address_validation"),
        ],
        "extra": lambda: {
            "cart_id": f"c_{random.randint(1000, 9999)}",
            "customer_id": f"u_{random.randint(100, 999)}",
        },
    },
    "notifications": {
        "service": "notifications-worker",
        "messages": [
            ("INFO", "email queued recipient={rcp} channel=email"),
            ("INFO", "sms queued recipient={rcp} channel=sms"),
            ("ERROR", "webhook failed recipient={rcp} channel=webhook status=503"),
        ],
        "extra": lambda: {
            "recipient": f"+1555{random.randint(1000000, 9999999)}",
            "channel": random.choice(["email", "sms", "webhook"]),
        },
    },
}

Setting elasticsearch.index to logs.otel as a resource attribute routes the data into the wired streams root instead of the default OTLP data stream.

def setup_provider():
    resource = Resource.create({"elasticsearch.index": "logs.otel"})
    provider = LoggerProvider(resource=resource)
    provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter()))
    set_logger_provider(provider)
    handler = LoggingHandler(level=logging.INFO, logger_provider=provider)
    root = logging.getLogger()
    root.setLevel(logging.INFO)
    root.addHandler(handler)
    return provider

Run the notebook to emit 115 records with an uneven split across teams.

Open Observability > Streams > logs.otel and switch to the Partitioning tab. You should see the ingested data in the preview panel, with attributes like team, service.name, and the team-specific fields visible in the columns.

How Streams AI Partitioning proposes child streams

Streams AI Partitioning analyzes up to 1,000 documents from the parent stream, identifies attribute clustering and cardinality distribution, then proposes child streams, keyed on whichever field best separates the data logically (the ML approach behind this analysis is detailed in automated log parsing in Streams).

For the data we just emitted, the AI proposed three child streams keyed on attributes.service.name:

Each suggestion shows a Streamlang condition and the percentage of sampled documents that would match. The AI picked service.name because it is a standard OpenTelemetry attribute and a natural identifier for any single workload.

This is a reasonable first proposal, but it is worth thinking about what happens as the deployment grows. Right now there are three services because there are three teams. Tomorrow, Payments might add a refunds-api and a fraud-detector. Each new service would mechanically create another child stream, and over time you would end up with dozens of partitions for what is really just three organizational boundaries.

Elastic's partitioning recommendations prefer logical groupings like team or technology type, and aim for tens of partitions rather than hundreds. A team-keyed partitioning is more stable because Payments stays one child stream no matter how many services that team operates.

Refining Streams AI Partitioning suggestions in natural language

After reviewing the AI's initial suggestions in Streams AI Partitioning, click Modify suggestions to open a free-text prompt.

After submitting, the AI regenerates the suggestions. Now the three cards are keyed on attributes.team instead of service.name:

Select all three and click Accept selected. A confirmation dialog shows the streams that will be created, each with its WHERE attributes.team equals <team> condition.

Click Create all streams. The Partitioning tab now shows the three child streams as part of the logs.otel parent:

Every new document arriving at the OTLP endpoint will be routed into the correct child according to these conditions. You can open any child stream to verify its data. For example, logs.otel.checkout shows only checkout logs:

How do you set per-team log retention in Elasticsearch Streams?

After Streams AI Partitioning creates child streams, each one can have its own lifecycle configuration independently of the parent. Because wired streams use a parent-child hierarchy, every child inherits retention, processors, and schema from the parent by default. You only need to override the partitions you need to change.

Open the child stream logs.otel.payments and go to the Retention tab. Click Edit retention method, select Custom period, and set it to 90 days.

Do the same for the other teams with the retention that fits their needs:

| Stream | Retention | Rationale | | :---- | :---- | :---- | | logs.otel.payments | 90 days | Sensitive financial data, compliance requirements | | logs.otel.checkout | 30 days | Useful for debugging, no long-term need | | logs.otel.notifications | 7 days | High volume, low value after delivery confirmation |

Conclusion: from shared index to per-team streams, without routing rules

A shared Elastic deployment with several teams shipping logs is the normal starting point. Organizing it used to mean writing routing rules upfront or maintaining separate index templates and ILM policies by hand.

With Streams AI Partitioning, the workflow is different: you let the data arrive, let the AI read what showed up, refine the suggestions in natural language when they need adjusting, and accept.

The result is a set of child streams that inherit everything from the parent while giving each team its own retention and processing, without any manual template management.

Next steps

• Try the companion notebook to generate your own multi-team data.
• Read How Streams in Elastic Observability Simplifies Retention Management for a deeper look at the retention model.
• Read Streams Processing: Stop Fighting with Grok to explore the parsing side of Streams when teams need different processors.
• Read Introducing Streams for Observability for the broader investigation story Streams is part of.

Prerequisites

• Elastic Cloud hosted (9.3+) or serverless

• Claude Desktop

• An MCP server instrumented with the Elastic Distribution of OpenTelemetry (EDOT). We cover how to instrument one below.

The observability gap in MCP servers

MCP servers ship with no built-in observability, which means tool-call latency, errors, and performance baselines are invisible to developers. MCP (Model Context Protocol) servers are increasingly used as infrastructure for AI-powered applications, giving AI models access to databases, APIs, internal tools, and business data. The MCP SDK does not instrument any of it.

The gap shows up in three concrete ways:

• When a tool call takes 3 seconds, you don't know if the bottleneck is in your business logic, a downstream API, or the data layer.
• When a tool call fails, you get the error message but no context about what the server was doing before it failed.
• When you add a new tool, you have no baseline to compare its performance against.

These are the same problems any backend service faces. The answer is the same one backend developers have used for years: distributed tracing with OpenTelemetry.

MCP servers are standard programmatic processes. There is nothing special about them from an instrumentation perspective. You add the OTel SDK, define spans around your tool handlers, and ship traces to your backend. The only new part is knowing which span names and attributes to use so your traces are meaningful and consistent.

What we built

For this article, we use the @modelcontextprotocol/server-everything package, the official reference MCP server published by Anthropic. It ships with a set of tools that cover the common patterns you will find in real-world MCP servers: simple request/response, parameterized calls, long-running operations, and calls that return structured data.

The server exposes several tools. In this article we use three of them:

• echo: receives a string and returns it unchanged. A minimal request/response tool, useful for verifying that the instrumentation pipeline works end to end.

• get-sum: receives two numbers and returns their sum. Represents a parameterized tool with simple business logic.

• trigger-long-running-operation: starts a multi-step operation that takes several seconds to complete. Simulates tools that call downstream APIs or run expensive computations.

We instrumented it with EDOT Node.js (@elastic/opentelemetry-node), which is Elastic's distribution of the OpenTelemetry SDK. EDOT replaces the five or six individual OTel packages you would otherwise install and adds the elasticapm connector that the Kibana APM UI needs to build its service maps, transaction groupings, and latency charts. Without that connector, raw OTLP data arrives in Elasticsearch but the APM views have nothing to build from.

The architecture looks like this:

This is the loop: Claude executes tools, generates telemetry, and then uses a second MCP to analyze that telemetry. The observability data becomes something the AI can reason about, not just something that sits in a dashboard waiting for a human to check it.

Both MCP servers are active simultaneously in Claude Desktop. The instrumented MCP generates telemetry. The Agent Builder MCP lets us query it.

To connect Claude Desktop to both servers, the claude_desktop_config.json looks like this:

{
  "mcpServers": {
    "everything": {
      "command": "node",
      "args": [
        "--import",
        "/path/to/node_modules/@elastic/opentelemetry-node/import.mjs",
        "/path/to/everything/dist/index.js",
        "stdio"
      ],
      "env": {
        "OTEL_SERVICE_NAME": "everything-mcp-server",
        "OTEL_EXPORTER_OTLP_ENDPOINT": "https://<your-otlp-endpoint>",
        "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=ApiKey <your-api-key>",
        "OTEL_LOG_LEVEL": "none"
      }
    },
    "elastic-agent-builder": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://<your-kibana-url>/api/agent_builder/mcp",
        "--header",
        "Authorization:ApiKey <your-api-key>"
      ]
    }
  }
}

The --import /path/to/@elastic/opentelemetry-node/import.mjs flag is all it takes for zero-code auto-instrumentation. But auto-instrumentation only captures HTTP calls, database queries, and other Node.js instrumented libraries. MCP tool calls are application logic, and application logic needs manual spans.

What traces MCP tool calls generate

The OpenTelemetry specification includes official semantic conventions for MCP. Following them means your traces are consistent, searchable by name across tools and teams, and compatible with any OTel-aware backend, including Elastic APM.

Span naming follows the pattern {mcp.method.name} {target}. For a tool call, this becomes tools/call echo or tools/call get-sum. This is what you will see as the transaction name in Kibana APM.

Key attributes on each span:

| Attribute | Value | Purpose | | ----------------------- | ---------------- | ------------------------- | | mcp.method.name | tools/call | The MCP protocol method | | gen_ai.tool.name | echo | The specific tool invoked | | gen_ai.operation.name | execute_tool | GenAI semantic convention | | error.type | error class name | Set only on failure |

The wrapper pattern that creates these spans looks like this:

const tracer = trace.getTracer('everything-mcp-server', '1.0.0');

function withToolSpan(toolName, fn) {
  return tracer.startActiveSpan(`tools/call ${toolName}`, (span) => {
    span.setAttribute('mcp.method.name', 'tools/call');
    span.setAttribute('gen_ai.tool.name', toolName);
    span.setAttribute('gen_ai.operation.name', 'execute_tool');

    try {
      const result = fn();
      span.end();
      return result;
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      span.setAttribute('error.type', err.constructor.name);
      span.end();
      throw err;
    }
  });
}

Each tool handler wraps its logic in withToolSpan. The result is a named span in Elastic APM for every tool invocation, with duration, status, and error details attached.

Security note: The OTel spec defines two optional attributes for tool calls: gen_ai.tool.call.arguments and gen_ai.tool.call.result_. Both are flagged as potentially containing sensitive data. The_ get-env tool in the everything server is a good example of why this matters: it returns all environment variables, which may include API keys and credentials. Capture these attributes only if you have confirmed the data is safe to store in your observability backend, and consider masking or filtering at the SDK level before export.

Kibana APM: exploring MCP server performance

After starting Claude Desktop with both MCPs configured and triggering a few tool calls, the everything-mcp-server service appears in Kibana under Observability > Applications > Services Inventory:

Transactions view

Kibana groups traces by transaction name. Because we follow the semantic conventions, each tool gets its own row: tools/call echo, tools/call get-sum, tools/call trigger-long-running-operation. You can immediately see latency, throughput, and error rate per tool without any configuration.

This is where the value of consistent span naming becomes concrete. If you have five different developers adding tools to an MCP server and everyone follows tools/call {toolName}, the APM UI stays organized automatically.

Trace waterfall

Clicking on a specific trace shows the waterfall view. For a single tool call, the waterfall is straightforward: one span covering the full execution. If your tool handler makes downstream HTTP requests or database queries that are auto-instrumented, those appear as child spans. You can see exactly how much time was spent in business logic versus waiting for external calls.

Latency distribution

The latency chart shows p50, p95, and p99 distribution across all executions of a given tool. This makes it easy to distinguish between tools that are consistently fast and those that have occasional outliers. The trigger-long-running-operation tool, for example, shows a wide distribution depending on how many steps were requested: a useful baseline for understanding expected execution time ranges before setting alerts.

Error tracking

Failed tool calls appear in the Errors panel with their full stack trace, the span attributes attached at the time of failure, and a count of how many times the error has occurred. If you record the exception with span.recordException(err), Kibana links the error directly to the trace that produced it.

Closing the loop with the Agent Builder MCP

The Elastic Agent Builder MCP lets Claude query its own trace data from Elasticsearch in the same chat session that produced the traces. The Agent Builder MCP can query any Elasticsearch index the API key has access to, and APM traces are stored under .ds-traces-apm.otel-default-*. Granting the Agent Builder API key read access to those indices is what closes the loop: the agent that executed the tool calls can now reason about how they performed.

Here is what this looks like in practice. To generate traces, let's ask in Claude Desktop: "Use the echo tool to say hello, then use get-sum to add 1337 and 42, then run a long-running operation with 3 steps."

Claude executes three tool calls on the instrumented MCP. Three spans land in Elastic APM.

Now, without leaving the chat, let's try querying the traces by asking: "Search the APM trace data from the last 10 minutes. What tool calls were made, and how long did each one take?"

We can confirm the information against the services data:

Claude uses the Agent Builder MCP to run a query against the traces index. It returns the tool names, durations, and status from the actual trace data, then synthesizes an answer in natural language.

You can go further by asking:

• "Which of those tool calls had the highest p95 latency?"

• "Did any tool calls fail? If so, what was the error message?"

• "Compare the latency of the echo tool vs get-sum across all calls in the last hour."

Each of these questions translates into an ES|QL query via the Agent Builder's platform.core.execute_esql tool, run against the APM trace indices.

Why Elastic for MCP observability

The Agent Builder closes the loop: this is the part that is specific to the Elastic ecosystem. Because APM data lives in Elasticsearch, and Elasticsearch is queryable via the Agent Builder MCP, you can bring your AI agent's own observability data back into the conversation. Your AI can reflect on its own performance and spot anomalies.

APM UI built for distributed tracing: Kibana's APM interface is designed for exactly this kind of data: named transactions, trace waterfalls, latency percentiles, error tracking with stack traces, and service maps.

Managed OTLP endpoint: Elastic APM accepts OTLP directly since Elastic 8.x. You point OTEL_EXPORTER_OTLP_ENDPOINT at your APM server and it works.

EDOT simplifies the setup: the elasticapm connector is included, which means the APM UI views work without any additional configuration.

Conclusion

MCP servers do not need special observability tooling. They are programmatic processes, and OpenTelemetry is the right instrument for processes. The OTel MCP semantic conventions are stable and give you a consistent naming scheme that scales across tools and teams.

What makes the Elastic setup interesting is not the instrumentation itself. It is the second MCP. When your observability data lives in Elasticsearch, you can query it from the same AI session that generated it. That feedback loop is new, and it opens up use cases that dashboards alone cannot cover: real-time anomaly questions, automated triage, and AI-assisted incident investigation from the chat interface your team is already using.

Next steps

• Set up EDOT Node.js

• Configure the Elastic Agent Builder MCP

• OTel semantic conventions for MCP

• Managed OTLP endpoint on Elastic Cloud (now GA)

• Instrumenting Node.js applications with EDOT

Frequently asked questions

How do I add tracing to an MCP server? Use the OpenTelemetry SDK and follow the official OTel MCP semantic conventions. With the Elastic Distribution of OpenTelemetry (EDOT) for Node.js, a single --import flag enables auto-instrumentation for HTTP and database calls. Tool-call spans need to be added manually using a wrapper that sets mcp.method.name, gen_ai.tool.name, and gen_ai.operation.name.

Why are my MCP tool calls slow and how do I find the bottleneck? Without tracing, an MCP tool call is a black box: you see the result but not where the time went. Instrument the server with OpenTelemetry, ship traces to Elastic APM, and use the trace waterfall view in Kibana to see exactly how much time was spent in business logic versus downstream HTTP or database calls.

Can I send MCP server traces to Elastic APM without a custom collector? Yes. Elastic APM accepts OTLP directly. Set OTEL_EXPORTER_OTLP_ENDPOINT to your APM endpoint and OTEL_EXPORTER_OTLP_HEADERS with an API key, and traces flow in. EDOT bundles the elasticapm connector that the Kibana APM UI needs for service maps and transaction grouping.

What span names and attributes should I use for MCP tool calls? Follow the OpenTelemetry MCP semantic conventions: name spans {mcp.method.name} {target} (for example, tools/call echo), and set mcp.method.name, gen_ai.tool.name, and gen_ai.operation.name=execute_tool. On failure, set error.type to the error class name. Consistent naming means the Elastic APM transactions view groups your tool calls automatically.

How is this different from sending MCP traces to Datadog or Grafana? Any OTel-compatible backend can receive the traces. The Elastic-specific part is the Agent Builder MCP, which lets the same AI agent that generated the traces query them back from Elasticsearch in natural language. That feedback loop, where the AI reasons about its own tool-call performance, is not available with backends that do not expose their data through an MCP server.

Should I capture MCP tool call arguments and results in my traces? The OTel spec defines gen_ai.tool.call.arguments and gen_ai.tool.call.result as optional and warns they may contain sensitive data. Tools like get-env, which returns environment variables, illustrate the risk: API keys and credentials can land in your observability backend. Capture these only when the data is safe to store, and consider masking at the SDK level before export.

Does this work for MCP servers written in languages other than Node.js? The OpenTelemetry MCP semantic conventions are language-agnostic. EDOT is available for Node.js, Java, Python, .NET, and other languages, and any of them can send OTLP to Elastic APM. The wrapper pattern shown in this post translates directly: open a span around the tool handler, set the standard attributes, record exceptions on failure.

Problem context

Outages often show up as correlated errors on multiple services like checkout, frontend, and recommendation in Elasticsearch. That pattern can mean a shared dependency, or it can mean Kubernetes is misleading callers: wrong targetPort, empty Endpoints, or pods that never become ready. Observability tools like Elasticsearch tell you that callers fail and which edges look wrong. They generally do not fetch the live Service spec or compare containerPort to targetPort for you.

The Elastic AI Agent in Agent Builder is built for APM, logs, metrics, dependencies, and service maps. It is not a full EKS operations console. You could attach all EKS MCP tools to the same agent, but long tool lists increase wrong-tool calls, slow planning, and widen blast radius if a prompt accidentally asks for mutating actions.

Solution overview

Use Elastic AI Agent as the only agent your SRE chats with. It reasons from Elasticsearch first. When evidence points to cluster config, it calls a workflow tool that invokes the K8s Troubleshooter agent over /api/agent_builder/converse with a structured user_prompt. The K8s Troubleshooter agent carries only the EKS MCP tools, and cluster access stays scoped to one specialist identity, IAM, and RBAC. You can audit like any other integration.

Elasticsearch reaches EKS through an in-cluster bridge, exposed to Kibana as an MCP connector with a shared secret.

Before you start

You need:

• An EKS cluster with kubectl configured.
• An Elasticsearch 9.3+ deployment, an OTLP endpoint, an Elasticsearch API key, Agent Builder, and rights to create agents, MCP tools, and Workflows.
• An AI Connector in Elasticsearch for your chosen LLM.
• Budget two to four hours the first time you run these steps.

Implementation walkthrough

Step 1: deploy the Elastic OpenTelemetry Demo and ship telemetry to Elastic Observability

Follow elastic/opentelemetry-demo for Kubernetes and deploy elastic-opentelemetry-demo application to your EKS cluster. Configure your Elasticsearch OTLP endpoint and API key, confirm workloads are running, and note the namespace. In Kibana (APM, Logs, or Service Map), confirm data for checkout, frontend, recommendation, and product-catalog.

If you see healthy traffic to product-catalog, you are ready for the failure drill.

Step 2: run the EKS MCP bridge, register the connector, and bulk import EKS MCP tools

Complete the steps in aws-eks-mcp-setup end to end. The flow you would be following is:

1. Build and push the EKS MCP Bridge image
2. Create IAM policies
3. Create IRSA Service Account
4. Map IRSA role in aws-auth and apply Kubernetes RBAC
5. Deploy the bridge with a strong API_ACCESS_TOKEN to the EKS cluster
6. Connect Elastic Agent Builder with EKS MCP

A green MCP connector proves Kibana can reach the bridge.

For production, restrict LoadBalancer security groups to known Elasticsearch egress, prefer TLS on real paths, store tokens in Kubernetes Secrets, and use read-only MCP modes when you only diagnose.

Step 3: create a K8s Troubleshooter agent with EKS tools only

In Agent Builder, create an agent with agent ID k8s_troubleshooter, display name K8s Troubleshooter, and custom instructions from k8s_troubleshooter_agent. Attach only EKS MCP tools to this agent.

Chat directly with K8s Troubleshooter agent once and confirm a harmless read (for example list pods in the demo namespace).

Step 4 (Elasticsearch 9.3 only): clone the Observability Agent without EKS tools

Clone the bundled Observability Agent (Agent Builder → Manage Agents → Observability Agent → Clone) and name it Elastic AI Agent so it keeps the stock Observability system instructions and tools. Do not attach EKS MCP tools to this copy.

The parent Elastic AI Agent stays an observability-first interface for whoever chats with it.

Step 5: create the workflow and make it a callable tool

Create a new Elastic Workflow by importing k8s_troubleshooter_workflow.yaml and enable it.

Create a new tool in Agent Builder of type Workflow. Select the k8s_troubleshooter workflow, set tool ID custom.k8s_troubleshooter, and set the description to Tool to triage and troubleshoot kubernetes related issues (or equivalent wording your team standardizes on).

On Elastic AI Agent, attach the custom.k8s_troubleshooter workflow tool that you just created.

The parent’s tool list should show the custom.k8s_troubleshooter workflow tool attached, and K8s Troubleshooter agent should still answer when invoked on its own.

Step 6: inject the product-catalog service misconfiguration

Save the original targetPort, then patch to a wrong value (for example 9999).

kubectl get svc -A | grep product-catalog
kubectl get svc product-catalog -n YOUR_NAMESPACE -o yaml

kubectl patch svc product-catalog -n YOUR_NAMESPACE --type='json' \
  -p='[{"op": "replace", "path": "/spec/ports/0/targetPort", "value": 9999}]'

kubectl rollout restart deployment/checkout deployment/recommendation deployment/frontend -n YOUR_NAMESPACE

Callers still resolve Endpoints, but traffic lands on a port the container does not listen on, so Elasticsearch shows downstream errors on checkout, frontend, and recommendation.

You now have symptoms in Elasticsearch and a clear kubernetes cluster-side fault.

Step 7: run two prompts on the parent agent

Use AI Agent chat on Elastic AI Agent, not on the specialist.

Note: If you are using Elasticsearch 9.3, make sure you use the Elastic AI Agent that you created, not the stock agent.

Prompt 1: Why are failure transactions increasing for services like checkout and frontend?

Expect Elastic AI Agent to narrow the issue to the product-catalog service and note possible configuration issues as one of the probable causes, without yet invoking the custom.k8s_troubleshooter tool.

Prompt 2: Why is product-catalog service not servicing any requests in (insert your k8s cluster name) cluster? Is there any misconfiguration in the service?

Expect Elastic AI Agent to call the custom.k8s_troubleshooter tool, which invokes the K8s Troubleshooter agent and reads Service, Endpoints, and pods, compares targetPort to containerPort, and explains the mismatch with evidence. Expect to also see the recommended remediation steps.

Note: depending on the LLM you are using, the response from the agents may vary.

You get agent-led triage in Elastic Observability and cluster-grounded confirmation in the same thread, along with recommended remediation steps.

Step 8: patch the product-catalog service

Prompt 3: Patch the product-catalog service in (your EKS cluster name) cluster to have 8080 as the targetPort

Expect Elastic AI Agent to call the custom.k8s_troubleshooter tool, which invokes the K8s Troubleshooter agent and patches the product-catalog service.

Prompt 4: Rollout restart upstream services of product-catalog service

Expect Elastic AI Agent to identify all upstream services of product-catalog and call the custom.k8s_troubleshooter tool, which invokes the K8s Troubleshooter agent to roll out restarts for upstream services such as checkout, frontend, and recommendation.

Confirm product-catalog and upstream services recover in Elasticsearch.

Validation and trade-offs

You validated that Elastic AI Agent stays the main surface, that ~20 EKS tools live on one specialist K8s Troubleshooter agent, and that the Workflow plus Agent Builder converse API keeps a clear boundary for audits and reviews.

Trade-offs: MCP bridges need ongoing token and network hygiene, and you should keep mutating tools off or tightly RBAC-scoped until you accept the risk.

Frequently asked questions

Why is Elastic AI Agent not triaging the issues as explained in this article?

There could be two primary reasons. (A) If you are on Elasticsearch 9.3, make sure you chat on the Elastic AI Agent that you created, and not on the stock agent. (B) Make sure to use one of the LLM models rated Excellent or Great in Large language model performance matrix for Observability

Why do my services look unhealthy in Elasticsearch when the app code did not change?

Kubernetes can mislead HTTP clients: a bad Service targetPort, empty Endpoints, or pods that never become ready can fan out as errors on multiple edges in traces and service maps. Elastic Observability shows which dependencies fail; confirming the live Service spec usually needs cluster access.

How do I give Kubernetes access to Elastic AI Agent without putting every EKS tool on it?

Run two Agent Builder agents: keep the stock tools on the parent (Elastic AI Agent), and attach only EKS MCP tools to a specialist agent(K8s Troubleshooter agent). Invoke the specialist through a workflow that calls the Agent Builder converse API so the boundary is explicit and auditable.

Why chain agents with Elastic Workflows instead of one long system prompt?

Workflows give a callable, reviewable step between observability reasoning and cluster actions, which helps with governance and keeps the parent agent’s tool list short. Long unified tool lists often increase mistaken tool use and broaden blast radius if a prompt requests a mutating operation.

How does this compare to kubectl or a cloud console for incident response?

Consoles and kubectl stay the source of truth for live object state. This pattern automates the handoff from Elastic Observability signals to those checks through MCP, while still relying on IAM and Kubernetes RBAC on the specialist identity.

What are the main limitations or risks of an EKS MCP bridge with Agent Builder?

MCP bridges need token rotation, network restrictions, and TLS discipline on real paths. Mutating EKS tools should stay off or tightly RBAC-scoped until you accept operational risk.

Why do we need an EKS MCP bridge?

The managed EKS MCP server authenticates via AWS SigV4 through a stdio-based proxy (mcp-proxy-for-aws). Elastic's MCP connector requires an HTTP/SSE endpoint. The bridge pod runs mcp-proxy to expose the stdio proxy as an SSE/HTTP endpoint.

Can I reuse the same layout on GKE, AKS, or self-managed Kubernetes?

Yes. The separation principle is the same: observability data in Elasticsearch plus a specialist agent with cluster-scoped tools and a workflow-mediated handoff. Swap the MCP server or bridge, adjust RBAC, and parameterize cluster name or region in workflow inputs where needed.

The new Elastic Kubernetes integration is built for that operator. It includes dashboards with drilldowns, alert rule templates, and ML anomaly detection jobs. Additionally Elastic also offers Agentic Investigations, that drives investigations automatically.

This blog will cover the foundational observability components (dashboards, drilldowns, alert templates, etc), while a part 2 covering the agentic investigations will cover workflows, agent skills, and MCP tools and views

The new Kubernetes integration content in this post is generally available across Elastic Cloud Hosted, Serverless, and self-managed deployments.

Dashboards designed for drill-down, not just display

The new Kubernetes dashboards are organized around a three-tier design: a cluster Overview that surfaces what needs attention at a glance, object summary pages for clusters, nodes, namespaces, workloads, and pods, and object detail pages that give you the full picture for any single entity.

Every layer connects to the next: click any entity in a summary table and choose: apply it as a filter on the current view, or open its dedicated detail page.

Here's what that looks like when something's actually wrong:

Following a restart cascade from overview to container

Overview: The Overview surfaces what needs attention across your cluster. You can see top pods by CPU, top namespaces by container restarts, and top nodes by memory utilization in one screen. When the "container restarts" panel starts climbing, you know where to look.

Namespaces Overview: Click into the flagged namespace with 1232 restarts and CPU limit utilization at 116%. The detail view plots CPU and memory against requests and limits over time. This shows both the size and duration of the overage.

Namespace Details: We can get more info on the various pods in this namespace here. Click the pod driving the restarts.

Pod Details: The pod detail dashboard is organized into capacity, metrics, and containers sections. Container restarts are flagged in red at the top of the page. Most panels are metric-driven, and the dashboard also links to correlated pod logs in Discover.

It takes four clicks to move from the Cluster Overview to container logs that explain the failure. These dashboards are starting points for your team. You can copy and customize them with ESQL visualizations.

Alert rules that fire on day one

The integration ships with pre-built alerting rule templates for states that are wrong by definition. No historical baseline or warmup period is required. Enable them during setup and they work immediately.

These rules do not ask, "Is this abnormal for this service?" They ask, "Is this a known bad state in Kubernetes?" A pod in CrashLoopBackOff is always a problem. A container killed by the kernel for exceeding its memory limit is always a problem.

Like the Kubernetes dashboards, these alerts are built on ES|QL queries. You can see that in the CrashLoopBackOff definition below. If you are new to ES|QL, you can start with the ES|QL docs.

The alert templates cover:

• CrashLoopBackOff detection - Fires when a pod's restart count exceeds a configurable threshold within a rolling window. The default catches a real restart cycle without triggering on routine restarts during a rolling deployment.
• Container OOMKilled - Surfaces kernel-level container terminations due to memory limits. These events are easy to miss in dashboards and often precede wider failures. This rule fires on any occurrence.
• Deployment below desired replicas - Fires when a deployment runs fewer replicas than declared for longer than a grace period. This catches scaling failures and partially failed rollouts.
• Pod stuck in Pending - Fires when a pod cannot be scheduled past a configurable time threshold. This surfaces node capacity problems, missing resources, and affinity failures before availability drops.
• Node disk pressure - Fires immediately when the Kubernetes DiskPressure node condition is True. A node condition is a direct state signal, not a statistical threshold.
• Persistent volume near capacity - Alerts when storage utilization crosses a configurable threshold before writes start failing.

Each template is parameterized. Adjust thresholds in the ES|QL query to match your environment. Connect notifications to PagerDuty, Slack, or another destination in your runbook.

Anomaly detection jobs with ML baselines

Alert rules catch what is definitively wrong. ML anomaly detection catches patterns that often precede failures. If you are new to this area, see the Elastic anomaly detection overview.

A pod that always runs at 85% memory utilization might be healthy. A pod that grew from 40% to 85% over twelve hours is usually not healthy. A static threshold often catches this only after an OOM kill. The ML module should catch the trajectory earlier.

The integration ships with ML module configurations that learn workload baselines and flag meaningful deviations. These jobs need 24 to 48 hours of data before results become useful. Results become more reliable as jobs continue to run.

The included modules

1. Pod memory growth anomalies

• What it learns: per-pod memory consumption pattern over time
• What it flags: Growth trajectories that are inconsistent with baseline behavior, such as a slow leak that never crosses the hard limit.
• Why ML (not alert rule): The alert rule catches the OOMKill after the fact. The ML job catches the trajectory that leads there.

2. Network I/O anomalies

• What it learns: per-pod network transmit/receive byte rate patterns
• What it flags: Unusual spikes or drops relative to the pod baseline. A spike can indicate a runaway process or unexpected load. A drop can indicate a network partition that causes the pod to go idle.
• Why ML (not alert rule): Normal network traffic varies by time of day and workload type. A batch job pod at high throughput during its normal window is expected. The same throughput outside that window can be anomalous.

3. Pod restart frequency

• What it learns: Per-workload restart rate patterns during deployments, scaling events, and routine operations.
• What it flags: Restart patterns that are anomalous relative to each workload's own history. This is distinct from the CrashLoopBackOff alert rule, which fires on a fixed threshold regardless of context.
• Why ML (not alert rule): A deployment that restarts twice during every rollout can be healthy. The same deployment restarting twice on a Tuesday afternoon may be unhealthy. The alert rule cannot distinguish these cases without workload history.

Here's our Single Metric Viewer showing anomalies triggered against a specific pod, for the memory growth job:

And here's the multi-series Anomaly Explorer view of the same job, showing detections firing across a variety of pods:

Try it yourself: the OTel Astronomy Shop

If you do not have a Kubernetes cluster ready, you can use the OpenTelemetry Astronomy Shop demo environment. It uses the same commands as Getting Started Step 2, Path A, but points to demo services. Create the namespace and secret, then run the Helm install. All 16 services, Kafka, and PostgreSQL start flowing into Elastic without instrumentation changes.

The demo ships with a built-in feature flag service, flagd, that lets you activate failure scenarios. Enable cartServiceFailure and watch the checkout-service restart cascade unfold in real time. The CrashLoopBackOff alert rule fires. The ML modules begin establishing baselines. If you have the investigation workflow enabled, it runs automatically when the alert fires.

Getting started

Step 1 - Install the Kubernetes integration. Dashboards are available immediately. No additional configuration is required.

Step 2 - Deploy data collection. There are two supported paths, both based on Helm. Choose the one that fits your deployment model.

Path A - OpenTelemetry (EDOT collector): This path uses the opentelemetry-kube-stack Helm chart with the Elastic Distribution of OpenTelemetry (EDOT) collector. Create a namespace and a secret with your endpoint and API key, then install:

kubectl create namespace opentelemetry-operator-system

kubectl create secret generic elastic-secret-otel \
  --namespace opentelemetry-operator-system \
  --from-literal=elastic_otlp_endpoint='https://<your-endpoint>.elastic.cloud:443' \
  --from-literal=elastic_api_key='<your-api-key>'

helm upgrade --install opentelemetry-kube-stack open-telemetry/opentelemetry-kube-stack \
  --namespace opentelemetry-operator-system \
  --values 'https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v9.3.2/deploy/helm/edot-collector/kube-stack/managed_otlp/values.yaml' \
  --version '0.12.4'

Path B - Elastic Agent (standalone): This path uses the elastic/elastic-agent Helm chart. The default manifest includes resource limits that may not be appropriate for production. Review the Scaling Elastic Agent on Kubernetes guide before deploying.

helm repo add elastic https://helm.elastic.co/ && \
helm install elastic-agent elastic/elastic-agent \
  --version 9.3.2 \
  -n kube-system \
  --set outputs.default.url=https://<your-endpoint>.es.elastic.cloud:443 \
  --set outputs.default.type=ESPlainAuthAPI \
  --set outputs.default.api_key=$(echo "<your-base64-api-key>" | base64 -d) \
  --set kubernetes.enabled=true

Step 3 - Enable the alert rule templates. Go to Observability > Alerts in Kibana. The Kubernetes templates are in the rule library. Enable the templates relevant to your environment, set thresholds, and connect your notification channel.

Step 4 - Let the ML modules warm up. After 24 to 48 hours, anomaly detection modules establish baselines and begin surfacing pattern-based deviations. Longer running jobs usually produce better baselines. Find results in the ML Anomaly Explorer, linked from the Kubernetes dashboards.

Steps 5, 6, and 7 - Agentic content will be covered in Part 2 (forthcoming), Kubernetes observability with Elastic: Agentic Investigations.

What's next

The next step is the layer that runs investigation workflows when an alert fires. That includes skills that encode investigation logic, tools that expose facts like ML state and topology, and MCP apps that render outputs in places like Claude Desktop or VS Code. These technical preview capabilities are available today and will be covered in Part 2 (forthcoming), Kubernetes observability with Elastic: Agentic Investigations.

If you are running Kubernetes on Elastic today, tell us which investigation steps you repeat manually on every incident. Tell us which remediations you would trust a workflow to propose. You can join the Elastic Community Discussion here.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In five years working with Enterprise accounts at Elastic, I have heard the same challenge again and again:

"We have several network monitoring tools, and we would love to correlate all of them into one platform."

For many organizations, the barrier to true correlation isn't a lack of data, but where that data lives. Frequently, we see SNMP metrics, flow data, and logs isolated in purpose-built silos or dashboards. Without a unified data store and a proper correlation engine, piecing together the full narrative — from a topology change to a performance degradation — becomes a manual, time-consuming puzzle.

When an incident happens, engineers become human correlation engines — manually jumping between systems, copying timestamps, cross-referencing device names, and trying to piece together what actually happened. A simple question like "Did this interface failure impact application performance?" requires querying multiple tools and mentally correlating the results.

The real cost isn't the tool licenses — it's the time lost during critical incidents.

This lab is my answer to a fundamental question: Can Elastic become the unified foundation that actually correlates network data?

More importantly, it demonstrates that Elastic is fully ready for network operations — capable of ingesting diverse telemetry and using AI to correlate relationships, identify root causes, and resolve issues in seconds instead of hours.

The Problem: Network Observability is Broken

Let me paint a typical scenario I encounter with enterprise network teams:

The Fragmented Reality:

• No single source of truth
• Manual correlation during incidents (15-30 minutes per event)
• Fragmented teams (network vs. platform engineers)
• Limited automation capabilities
• No AI-powered analysis

When a link goes down at 2 AM:

• Notice the alert - 2 minutes
• Log into monitoring tool to see the metric - 3 minutes
• Switch to traffic analyzer to check impact - 5 minutes
• Open log management to search for related messages - 10 minutes
• Manually correlate timestamps across systems - 8 minutes
• Create a ticket and copy context from multiple tools - 8 minutes

Time to initial diagnosis: 36 minutes

This workflow is expensive, error-prone, and doesn't scale.

The Vision: Elastic as a Unified Network Observability Platform

What if you could:

• Collect SNMP metrics, NetFlow, traps, and topology data in one platform
• Correlate network events with application performance automatically
• Generate executive dashboards without separate BI tools
• Use AI to analyze incidents in seconds, not hours
• Trigger alerting from network events

This is what this lab aims to demonstrate.

What I Built: A Production-Grade Network Simulation

To demonstrate how Elastic unifies network data, I needed a realistic environment that generates real-world telemetry. Enter Containerlab — a Docker-based solution that enables us to create a network simulation framework.

Lab Architecture

I simulated a Service Provider core network with:

• 7 FRR routers forming an OSPF Area 0 mesh
• 2 Ubuntu hosts for additional use cases
• 2 Layer 2 switches for access layer segmentation
• 3 telemetry collectors feeding Elastic Cloud

Total containers: 14

Deployment time: 12-15 minutes (fully automated)

Full deployment instructions and topology details are available in the GitHub repository README.

The Three Telemetry Pipelines: Proving Multi-Source Correlation

What makes this lab production-ready is its hybrid observability approach — proving that Elastic can unify disparate network data sources.

| Pipeline | Data Type | Collection Method | Collector | Use Case | | :---- | :---- | :---- | :---- | :---- | | SNMP Metrics | Interface stats, system health, LLDP topology | Active polling | OTEL Collector | Capacity planning, trend analysis | | NetFlow | Traffic flows | Push-based export | Elastic Agent | Top talkers, security investigation | | SNMP Traps | Interface up/down events | Event-driven | Logstash | Real-time incident detection |

This unified architecture proves Elastic can replace multiple specialized network monitoring tools with a single platform.

The Power of Correlation: One Platform, One Query

When a network incident occurs, you need to answer questions like:

• Which interface failed? (SNMP metrics)
• What traffic was affected? (NetFlow)
• What was the sequence of events? (SNMP traps)
• Which devices are downstream? (LLDP topology)

The Problem: modern tools offer separate modules glued together, forcing users to navigate different spaces for different sets of data.

The Reality: You still have to pivot. You see a spike in the Metrics module, but to see why, you have to open the Logs module and manually align the time picker. The data lives in different tables or backends, making true correlation impossible without human intervention.

The Elastic Difference: One Store, One Language, One AI

Elastic makes it simple. Whether it's an SNMP counter (metric), a NetFlow record (flow), or a Syslog message (log), it is all stored in a unified datastore powered by the Elasticsearch engine. This allows users to easily search across multiple datasets in a single query.

FROM logs-*
| WHERE host.name == "csr23" AND interface.name == "eth1"

Time required: 3 seconds

Furthermore, as you will see later, the exact location of the data becomes agnostic to the user when leveraging the AI Assistant.

Data Transformation: From Cryptic OIDs to Actionable Intelligence

Raw SNMP traps are notoriously difficult to interpret at a glance. In our current lab setup, the data arrives looking like this:

OID: 1.3.6.1.6.3.1.1.5.3
ifIndex: 2
ifDescr: eth1

While traditional Network Management Platforms (NMPs) handle OID translation natively, bringing that clarity into Elastic requires a specific configuration.

In this initial lab, we are intentionally working with this raw data to demonstrate how AI assistants can interpret these events even without pre-existing context.

However, the strategy for the next phase of this project is to implement Elasticsearch Ingest Pipelines. This will allow us to map raw OIDs to human-readable names. This step is crucial for bridging the gap between Network tools and Application Observability platforms, allowing network events to be instantly correlated with application errors and infrastructure logs.

The Target State

Once the pipeline is implemented in the next lab, we will transform that raw trap into searchable, meaningful data:

{
  "event.action": "interface-down",
  "host.name": "csr23",
  "interface.name": "eth1",
  "interface.oper_status_text": "Link Down"
}

The result:

• Human-readable fields
• Searchable dimensions for filtering
• Context for automation rules and dashboards
• Correlation keys for joining with metrics and flows

In our next blog post, we will walk through building the ingest pipeline that performs this transformation — step by step.

Intelligent Alerting: From Noise to Actionable Intelligence

Traditional network monitoring relies on simple threshold alerts — "interface down," "high CPU." These alerts flood your inbox but provide zero context about root cause, impact, or remediation.

The Lab's Approach: ES|QL + AI Assistant

1. Semantic Detection with ES|QL

Instead of generic threshold alerts, the lab uses ES|QL to detect specific event patterns:

FROM logs-snmp.trap-prod
| WHERE snmp.trap_oid == "1.3.6.1.6.3.1.1.5.3"
| KEEP @timestamp, host.name, interface.name, message

2. Automatic AI-Powered Investigation

When the alert triggers, it invokes the Observability AI Assistant with a structured investigation prompt that:

• Performs immediate triage (which device, which interface, when)
• Assesses OSPF impact and traffic rerouting
• Correlates with other recent failures
• Generates severity assessment and recommended actions

The Transformation

| Traditional Alerting | Intelligent Alerting (Elastic) | | :---: | :---: | | Email: "Interface down on csr23" | Structured analysis with device context | | Manual investigation: 20-30 min | AI-automated investigation: 90 seconds | | Engineer correlates across tools | Automatic cross-source correlation | | No business impact assessment | Severity + recommended actions included |

Accelerating Incident Response with the Elastic AI Assistant

This is where the Elastic AI Assistant demonstrates its operational value — moving beyond passive data collection to actively interpret and explain network events in real-time

When an engineer views a trap document in Discover and asks:

"Explain this log message"

The AI Assistant provides comprehensive analysis including:

• What happened: Plain-language explanation of the SNMP trap
• Device context: Router role, interface purpose, network position
• Impact analysis: OSPF neighbor status, traffic rerouting assessment
• Root cause possibilities: Physical layer, link layer, administrative causes
• Recommended actions: Immediate steps, investigation queries, validation checks
• Severity assessment: Business and technical impact rating

Manual Triage vs. AI-Assisted Investigation

| Before | After (Elastic AI) | | :---- | :---- | | Google the OID → 5 min | Click "Explain this log" → 20 seconds | | Open network diagram → 3 min | Topology context auto-provided | | Query multiple tools → 10 min | Cross-source correlation instant | | Assess business impact → 5 min | Impact analysis auto-generated | | Total: ~28 minutes | Total: ~20 seconds |

The Value Proposition: One Platform, One Data Model, One AI

What This Lab Demonstrates

Elastic provides:

• One unified platform for metrics, logs, flows
• One data model (SemConv) for consistent correlation
• One search interface (Kibana) for all network data
• One AI assistant that understands all your network telemetry
• AI-powered alerting with automated investigation

Business Impact

Efficiency Gains:

• 85% reduction in MTTR (36 min → 5 min for initial diagnosis)
• 90% reduction in manual correlation time
• Junior engineers gain access to AI-powered expert analysis

Operational Benefits:

• Network engineers focus on strategy, not tool-switching
• Cross-functional collaboration in one platform
• Reduced tool sprawl and management overhead

Lessons Learned

After building this lab, several key insights emerged regarding how network data fits into the broader observability ecosystem:

1. Extending Observability to the Network

Elastic is already the gold standard for high-volume logs and application traces. This lab demonstrates that the same engine seamlessly handles network telemetry without needing a separate, siloed tool.

• Scale: The same architecture that ingests petabytes of application logs easily handles millions of interface counters.
• Structure: Native support for complex nested documents allows for rich SNMP trap data (variable bindings) without flattening or losing context.
• Speed: Real-time search applies equally to network events, enabling sub-second troubleshooting.

2. OpenTelemetry Semantic Conventions (SemConv) as the Universal Translator

The power isn't just in storing the data, but in standardizing it. By mapping SNMP and NetFlow to the OpenTelemetry Semantic Conventions (SemConv), network data finally speaks the same language as the rest of the stack.

• Unified Search: Query across firewall logs, server metrics, and switch telemetry in a single search bar.
• Instant Visualization: Pre-built dashboards work immediately because the field names are standardized.
• Cross-Domain Correlation: Easily correlates a spike in application latency with a specific interface saturation event.

3. AI Assistants Thrive on Context

While the AI in this lab was powerful on its own, the experiment highlighted a critical realization: an AI Assistant becomes exponentially more effective when coupled with a specific Knowledge Base.

Context is King: The AI delivers better root cause analysis when provided with rich metadata, such as device roles and topology maps. Without it, the advice remains generic.

Pro Tip (and What’s Next):

To get organization-specific advice rather than generic suggestions, you need to feed the AI your documentation.

• The Goal: Create a Knowledge Base containing device roles, network topology diagrams, and troubleshooting procedures.
• The Next Step: In my next blog post, I will demonstrate exactly how to do this — connecting a Knowledge Base to the AI Assistant to enable fully context-aware troubleshooting.

Conclusion: Completing the Observability Picture

Elastic is already widely recognized as the standard for Application and Security observability. The goal of this lab wasn't to ask if Elastic can handle networking, but to demonstrate the immense value of bringing network data into that existing ecosystem.

The verdict is clear: Elastic acts as that unified foundation. It effectively breaks down the silo between Network Engineering and the rest of IT.

This isn't just about consolidating dashboards or replacing legacy tools. It is about establishing the Elasticsearch AI Platform as the single source of truth where network telemetry sits right alongside application and infrastructure data.

By treating network data as a first-class citizen in the observability stack, we unlock automated correlation, AI-assisted investigation, and the speed required to resolve incidents before they impact the business. The capabilities are in place, and the foundation is solid — Elastic is ready to unify your network with the rest of your digital business.

Ready to Try It Yourself?

Check out github.com/DeBaker1974/Containerlab-OSPF

The repository includes:

• Complete deployment scripts (12-15 minute automated setup)
• Pre-configured telemetry pipelines
• Kibana dashboards
• Alert rules with AI Assistant integration
• Detailed README

Not ready to build? Try Elastic Serverless: Start a free 14-day trial and explore AI-powered observability with your own data.

Special thanks to the Containerlab and FRRouting communities for their incredible open-source tools, and to Sheriff Lawal (CCIE, CISSP), Sr. Manager, Solutions Architecture at Elastic, for mentoring on this project.

One of the main components of Elastic's Synthetic Monitoring is the ability to create user journeys, which can be done with or without code. There is a Synthetics agent,, a CLI tool that guides you through the process of creating both heartbeat monitors and user journeys and deploying your code to Elastic Observability. If you are using code to create user journeys, you are using Playwright under the hood with some additional configuration to make it easier to work with Elastic Observability.

To automatically create user journeys using TypeScript, you can create Playwright tests based on a prompt using Warp, an AI-assisted terminal, Gemini 2.5 Pro, and MCP. This application was built using Python and FastMCP, which wraps the synthetic agent to deploy browser tests to Elastic automatically. This blog post will guide you through how the application works, how to use it, and its development process. You can find the complete code on GitHub.

Solution overview

Currently, this solution is set up to run inside Warp as an MCP server; however, you can also use another client, such as Claude Desktop or Cursor. From there, you create a Python script using FastMCP, which allows you to create functions that are callable by an LLM. Within Warp, you can make a configuration file in JSON that enables you to point to your Python script and pass in all the environment variables you are working with. From there, you'll want to toggle agent mode and ask a question about creating synthetic testing or call the MCP function directly. There are many options for which LLM you can select, be sure to check out Warp's documentation to learn more about the options available.

After that, you should ask a question about creating synthetic testing or call the MCP function you are looking for. The following three functions can be used:

• diagnose_warp_mcp_config Used for debugging environment variable issues that may arise. This function likely won't be needed unless there is an issue with your configuration.

• create_and_deploy_browser_test Will automatically create Playwright tests if given the test name, the URL you want to test, and a schedule. This approach uses a template-based method, rather than a machine learning-based method, and all the tests it outputs will appear similar.

• llm_create_and_deploy_test_from_prompt Similar to create_and_deploy_browser_test, but the main difference is that it uses an LLM to create tests based on a prompt you give it. The tests should reflect the prompt you provided. To run this function you'll provide a test name, URL, prompt, and schedule.

Why create this solution as an MCP server?

The reason this was developed as an MCP server, as opposed to just a standalone script or a standard CLI, is that it can be structured and interacted with in a more conversational manner. It enables an LLM to generate dynamic Playwright testing while maintaining consistent arguments, environment variables, and responses to ensure accuracy and reliability. Thus, it becomes a reliable workflow that other agents or developers can compose with additional tools. In other words, the MCP layer turns your LLM-based test authoring into a standardized, reusable capability instead of a one-off script. To learn more about the direction of MCP, be sure to check out our article on the topic.

Implementation considerations

When creating a solution like this one, one thing to be mindful of is your use of tokens. An early version of this solution took approximately twenty minutes to create synthetic tests and ultimately led to severe rate-limiting.

Another issue faced during the building process was striking a balance between creating a template that facilitates the creation of a Playwright script and having an LLM create Playwright scripts based on prompts that didn't feel cookie-cutter. While using a more LLM approach an issue faced was that the scripts often didn't work or were based on parameters that didn't exist and a more templated approach was more reliable but felt repetitive. The final version of this solution attempted to balance this by using elements of the template while adjusting the LLM parameter of temperature, which controls the randomness or creativity of a large language model's output.

While testing this solution, a failing test also emerged that required navigating past a pop-up. In more complex cases, this may serve as a building block that requires additional domain knowledge to create a complete passing Playwright test.

How to get started

Prerequisites

• The version of Python that is used is Python 3.12.1 but you can use any version of Python higher than 3.10.
• This application uses Elastic Observability version 9.1.2, but you can use any version of Elastics Observability that is higher than 8.10. You can also use Elastic Cloud Serverless as well.
• You will also need an OpenAI API key to use the LLM capabilities of this application. You will want to configure an environment variable for your OpenAI API Key, which you can find on the API keys page in OpenAI's developer portal.

Step 1: Install the packages and clone the repository

In order for this MCP server to run locally you will need to install the the following packages:

pip install fastmcp openai
npm install -g playwright @elastic/synthetics

You will use FastMCP 2.0 to create the MCP server, and OpenAI to generate tests based on prompts that you provide. Additionally, you will want to clone the repository to obtain a local copy of the server.

Step 2: Set up a configuration file in Warp

Inside of Warp, you will want to go to the side panel, where it says MCP servers and where it says “add”.

After that, you will be prompted to add a JSON configuration file that should resemble the following. Be sure to add your own Kibana URL, update the correct path, and include your own keys and tokens.

{
 "elastic-synthetics": {
   "command": "python",
   "args": ["elastic_synthetics_server.py"],
   "env": {
     "PYTHONPATH": ".",
     "ELASTIC_KIBANA_URL": "https://your-kibana-url.elastic-cloud.com",
     "ELASTIC_API_KEY": "your-api-key-here",
     "ELASTIC_PROJECT_ID": "mcp-synthetics-demo",
     "ELASTIC_SPACE": "default",
     "ELASTIC_AUTO_PUSH": "true",
     "ELASTIC_USE_JAVASCRIPT": "false",
     "ELASTIC_INSTALL_DEPENDENCIES": "true",
     "OPENAI_API_KEY": "sk-your-openai-key",
     "LLM_MODEL": "gpt-4o"
   },
   "working_directory": "/path/to/your/file",
   "start_on_launch": true 
   }
}

Step 3: Ask a question or call the tools directly

Now that you've set up locally, you will want to toggle agent mode and select the LLM you wish to use. The reason why Gemini-Pro-2.5 was chosen for this blog post is that it provides a straightforward answer, while other LLMs selected returned a very lengthy response.

To start using the MCP tools, from your MCP server, you can ask a question that contains the test name, URL, prompt, and schedule.

You can also call the directly by typing llm_create_and_deploy_test_from_prompt() and the program will prompt you for the relevant details:

Inside Kibana, you should see your monitor listed if you click under Applications and select Monitors listed under Synthetics. You can also find a link to your monitor in the response of your MCP tool.

What's Going On Inside

This code sample consists of three primary functions, which are MCP tools that you can call from your MCP client, including diagnose_warp_mcp_config, create_and_deploy_browser_test and llm_create_and_deploy_test_from_prompt.

Debugging environment issues

There were various issues that came up while creating this application around environment variable loading, so there was a need to create an MCP that could be called depending on errors that may be present.

The tool diagnose_warp_mcp_config kicks off with a decorator @mcp.tool() which allows it to be called and listed in the list of available tools. This tool is designed to help debug issues with Elastic-specific environment variables for troubleshooting purposes. First, it loads in the environment variables and looks for the Elastic specific variables, after it does some security masking so it doesn't show any variables and hides sensitive information like API keys in the output, showing only the first eight characters followed by "…". This tool determines if the minimum required credentials (Kibana URL and API Key) are present to proceed with deployment and provides a report letting you know to address any issues that may exist.

@mcp.tool()
def diagnose_warp_mcp_config() -> Dict[str, Any]:
   """Diagnose Warp MCP environment configuration for Elastic Synthetics"""
   try:
       env_vars = load_env_from_warp_mcp()

       # Check for required variables
       kibana_url = env_vars.get('ELASTIC_KIBANA_URL') or env_vars.get('KIBANA_URL')
       api_key = env_vars.get('ELASTIC_API_KEY') or env_vars.get('API_KEY')
       project_id = env_vars.get('ELASTIC_PROJECT_ID') or env_vars.get('PROJECT_ID')
       space = env_vars.get('ELASTIC_SPACE') or env_vars.get('SPACE', 'default')

       # Mask sensitive values for display
       masked_vars = {}
       for key, value in env_vars.items():
           if 'API_KEY' in key or 'TOKEN' in key:
               masked_vars[key] = f"{value[:8]}..." if value and len(value) > 8 else "***"
           else:
               masked_vars[key] = value

       deployment_ready = bool(kibana_url and api_key)

       return safe_json_response({
           "status": "success",
           "environment_variables": masked_vars,
           "required_check": {
               "kibana_url": bool(kibana_url),
               "api_key": bool(api_key),
               "project_id": bool(project_id),
               "space": bool(space)
           },
           "deployment_ready": deployment_ready,
           "recommendations": [
               "Environment variables detected" if env_vars else "No environment variables found",
               "Kibana URL configured" if kibana_url else "Missing ELASTIC_KIBANA_URL or KIBANA_URL",
               "API Key configured" if api_key else "Missing ELASTIC_API_KEY or API_KEY",
               "Ready for deployment" if deployment_ready else "Missing required credentials"
           ]
       })

   except Exception as e:
       return safe_json_response({
           "status": "error",
           "error": str(e),
           "error_type": type(e).__name__
       })

Creating synthetic tests based on a template

While developing this solution to generate tests based on a prompt, the process wasn't always smooth. Early versions encountered issues with accuracy, hallucinations, and the creation of loops. To make progress, a version that relied on creating a test template to verify the mechanics of the solution, such as whether the test could pass and be deployed to Elastic correctly, was a logical next step.

This solution automates the entire process of creating a synthetic browser test that will regularly check if a website is working correctly, then deploys it to Elastic Observability Synthetics. Similar to diagnose_warp_mcp_config, the MCP tool create_and_deploy_browser_test starts with the decorator @mcp.tool() and checks to make sure that the proper environment variables are loaded.

From there, it creates a TypeScript test file that is based on templates and generates dynamic test steps based on the target website's characteristics, including navigating to the website, verifying the page title exists, checking page load performance, taking a screenshot, verifying page content is visible, and finally saves the test file in a synthetic_tests directory.

Finally, it wraps Elastic's CLI tool @elastic/synthetics to push the test to Kibana, allowing you to set which geographic locations to run tests from, how often to run the test, and the project and workspace settings.

You check out the full code for this MCP tool here.

Creating synthetic tests based on a prompt

While creating browser tests based on a templated approach is a good starting point, it felt generic and cookie-cutter. But it made a helpful structure to build an LLM-based function on top of.

The MCP tool llm_create_and_deploy_test_from_prompt begins by ensuring that basic parameters, including locations, schedule, and directories, are listed. Additionally, it aims to learn more about the target website to inform the AI and initialize the OpenAI client and model, which is GPT-4o.

After setting up the LLM, it converts natural language requests into actual Playwright test code, then cleans and validates the AI-generated code to prevent issues like injection attacks or malformed syntax. It draws inspiration from the templated approach, wrapping AI-generated steps within a proven, reliable test framework template. Finally, it deploys the test to Elastic in a similar manner to the previous tool.

You can find the code for this tool here.

Conclusion and next steps

Synthetic monitoring in Elastic Observability makes it easy to test complete user journeys and keep your site reliable, with simple setup and a Playwright integration. A tool like this can provide a starting point for tests that you can iterate on after.

A solution like this is just the start of an MCP implementation that automatically generates Playwright tests for you and can be expanded in the future to include heartbeat monitors, utilize the Playwright MCP server, or consider experimenting with Claude for Chrome to create synthetic testing.

Check out more articles on Observability Labs on Infrastructure Monitoring

Elastic’s AI Assistant helps improve analysis for SREs, DevOps, Devs, and others. In a single window using natural language questions, you can analyze using not only general information but combine it with things like:

• Issues from internal GitHub repos, Jira, etc.

• Documents from internal wiki sites from Confluence, etc.

• Customer issues from your support service

• And more

In this blog, we will walk you through how to:

1. Ingest an external GitHub repository (OpenTelemetry demo repo) with code and issues into Elastic. Apply Elastic Learned Sparse EncodeR (ELSER) and store it in a specific index for the AI Assistant.

2. Ingest internal GitHub repository with runbook information into Elastic. Apply ELSER and store the processed data in a specific index for the AI Assistant.

3. Use these two indices when analyzing issues for the OpenTelemetry demo in Elastic using the AI Assistant.

3 simple questions using GitHub data with AI Assistant

Before we walk through the steps for setting up data from GitHub, let’s review what an SRE can do with the AI Assistant and GitHub repos.

We initially connect to GitHub using an Elastic GitHub connector and ingest and process two repos: the OpenTelemetry demo repo (public) and an internal runbook repo (Elastic internal).

With these two loaded and parsed by ELSER, we ask the AI Assistant some simple questions generally asked during analysis.

How many issues are open for the OpenTelemetry demo?

Since we ingested the entire repo (as of April 26, 2024) with a doc count of 1,529, we ask it a simple question regarding the total number of issues that are open. We specifically tell the AI Assistant to search our internal index to ensure the LLM knows to ask Elastic to search its internal index for the total number of issues.

Are there any issues for the Rust based shippingservice?

Elastic’s AI Assistant uses ELSER to traverse the loaded GitHub repo and finds the open issue against the shippingservice (which is the following issue at the time of writing this post).

Is there a runbook for the Cartservice?

Since we loaded an internal GitHub repo with a few sample runbooks, the Elastic AI Assistant properly finds the runbook.

As we go through this blog, we will talk about how the AI Assistant finds these issues using ELSER and how you can configure it to use your own GitHub repos.

Retrieval augmented generation (RAG) with Elastic AI Assistant

Elastic has the most advanced RAG-based AI Assistant for both Observability and Security. It can help you analyze your data using:

• Your favorite LLM (OpenAI, Azure OpenAI, AWS Bedrock, etc.)

• Any internal information (GitHub, Confluence, customer issues, etc.) you can either connect to or bring into Elastic’s indices

The reason Elastic’s AI Assistant can do this is because it supports RAG, which helps retrieve internal information along with LLM-based knowledge.

Adding relevant internal information for an SRE into Elastic:

• As data comes in, such as in your GitHub repository, ELSER is applied to the data, and embeddings (weights and tokens into a sparse vector field) are added to capture semantic meaning and context of the data.

• This data (GitHub, Confluence, etc.) is processed with embeddings and is stored in an index that can be searched by the AI Assistant.

When you query the AI Assistant for information:

• The query goes through the same inference process as the ingested data using ELSER. The input query generates a “sparse vector,” which is used to find the most relevant highly ranked information in the ingested data (GitHub, Confluence, etc.).

• The retrieved data is then combined with the query and also sent over to the LLM, which will then add its own knowledge base information (if there is anything to add), or it might ask Elastic (via function calls) to analyze, chart, or even search further. If a function call is made to Elastic and a response is provided, it will be added by the LLM to its response.

• The results will be the most contextual based answer combining both LLM and anything relevant from your internal data.

Application, prerequisites, and config

If you want to try the steps in this blog, here are some prerequisites:

• An Elastic Cloud account — sign up now

• OpenTelemetry demo running and connected to Elastic (APM documentation)

• Whatever internal GitHub repo you want to use with some information that is useful for analysis (In our walk through, we will be using a GitHub repo that houses runbooks for different scenarios when Elastic does demos).

• Account with your favorite or approved LLM (OpenAI, Azure OpenAI, AWS Bedrock)

Adding the GitHub repos to Elastic

The first step is to set up the GitHub connector and connect to your GitHub repo. Elastic has several connectors from GitHub, Confluence, Google Drive, Jira, AWS S3, Microsoft Teams, Slack, and more. So while we will go over the GitHub connector in this blog, don’t forget about other connectors.

Once you select the GitHub connector and give it a name, you need to add two items:

• GitHub token

• The URL open-telemetry/opentelemetry-demo

Next, add it to an index in the wizard.

Create a pipeline and process the data with ELSER

In order to add the embeddings we discussed in the section above, we need to add the following to the connector:

• Create a pipeline in the configuration wizard.

• Create a custom pipeline.

• Add the ML inference pipeline.

• Select ELSERv2 ML Model to add the embeddings.

• Select the fields that need to be evaluated as part of the inference pipeline.

• Test and save the inference pipeline and the overall pipeline.

Sync the data

Now that the pipeline is created, you need to start to sync the github repo. As the documents from the github repo come in, they will go through the pipeline and embeddings will be added.

Embeddings

Once the pipeline is set up, sync the data in the connector. As the GitHub repository comes in, the inference pipeline will process the data as follows:

• As data comes in from your GitHub repository, ELSER is applied to the data, and embeddings (weights and tokens into a sparse vector field) are added to capture semantic meaning and context of the data.

• This data is processed with embeddings and is stored in an index that can be searched by the AI Assistant.

When you look at the OpenTelemetry GitHub documents that were ingested, you will see how the weights and token are added to the predicted_value field in the index.

These embeddings will now be used to find the most contextually relevant document when the user asks the AI Assistant a query, which might use this.

Check if AI Assistant can use the index

Elastic’s AI Assistant uses ELSER to traverse the loaded Github repo and finds the open issue against the shippingservice. (which is the following issue at the time of writing this post).

Based on the response, we can see that the AI Assistant can now use the index to find the issue and use it for further analysis.

Conclusion

You’ve now seen how easy Elastic’s RAG-based AI Assistant is to set up. You can bring in documents from multiple locations (GitHub, Confluent, Slack, etc.). We’ve shown the setup for GitHub and OpenTelemetry. This internal information can be useful in managing issues, accelerating resolution, and improving customer experiences. Check out our other blogs on how the AI Assistant can help SREs do better analysis, lower MTTR, and improve operations overall:

• Analyzing OpenTelemetry apps with Elastic AI Assistant and APM

• The Elastic AI Assistant for Observability escapes Kibana!

• Getting started with the Elastic AI Assistant for Observability and Microsoft Azure OpenAI

• Elastic 8.13: GA of Amazon Bedrock in the Elastic AI Assistant for Observability

• Enhancing SRE troubleshooting with the AI Assistant for Observability and your organization's runbooks

• Context-aware insights using the Elastic AI Assistant for Observability

• Getting started with the Elastic AI Assistant for Observability and Amazon Bedrock

Try it out

Existing Elastic Cloud customers can access many of these features directly from the Elastic Cloud console. Not taking advantage of Elastic on cloud? Start a free trial.

All of this is also possible in your environments. Learn how to get started today.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

The Elastic AI Assistant significantly enhances the process, not only in identifying but also in resolving issues. This is further enhanced by Elastic’s new Service Level Objective (SLO) capability, allowing you to streamline your entire site reliability engineering (SRE) process from detecting potential issues to enhancing the overall customer experience.

In this blog, we will demonstrate how you, as an SRE, can detect issues in a service equipped with OpenTelemetry. We will explore problem identification using Elastic APM, Elastic’s AIOps capabilities, and the Elastic AI Assistant.

We will illustrate this using the OpenTelemetry demo, with a feature flag (cartService) that is activated.

Our walkthrough will encompass two scenarios:

1. When the SLO for cart service becomes noncompliant, we will analyze the error through Elastic APM. The Elastic AI Assistant will assist by providing a runbook and a GitHub issue to facilitate issue analysis.

2. Should the SLO for the cart service be noncompliant, we will examine the trace that indicates a high failure rate. We will employ AIOps for failure correlation and the AI Assistant to analyze logs and Kubernetes metrics directly from the Assistant.

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up the configuration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).

• We used the OpenTelemetry Demo. Directions for using Elastic with OpenTelemetry Demo are here.

• Additionally you will need to connect your AI Assistant to your favorite LLM. We used Azure OpenAI GPT-4.

• We also ran the OpenTelemetry Demo on Kubernetes, specifically on GKE.

SLO noncompliance

Elastic APM recently released the SLO (Service Level Objectives) feature in 8.12. This feature enables setting measurable performance targets for services, such as availability, latency, traffic, errors, and saturation or define your own. Key components include:

• Defining and monitoring SLIs (Service Level Indicators)

• Monitoring error budgets indicating permissible performance shortfalls

• Alerting on burn rates showing error budget consumption

We set up two SLOs for cart service:

• Availability SLO , which monitors its availability by ensuring that transactions succeed. We set up the feature flag in the OpenTelemetry application, which generates an error for EmptyCart transactions 10% of the time.

• Latency SLO to ensure transactions are not going below a specific latency, which will reduce customer experiences.

Because of the OTel cartservice feature flag, the availability SLO is triggered, and within the SLO details, we see that over a seven-day period the availability is well below our target of 99.9, at 95.5. Additionally all the error budget that was available is also exhausted.

With SLO, you can easily identify when issues with customer experience occur, or when potential issues with services arise before they become potentially worse.

Scenario 1: Analyzing APM trace and logs with AI Assistant

Once the SLO is found as non-compliant, we can dive into cart service to investigate in Elastic APM. The following walks through the set of steps you can take in Elastic APM and how to use the AI Assistant to analyze the issue:

From the video, we can see that once in APM, we took the following steps.

1. Investigated the trace EmptyCart, which was experiencing larger than normal failure rates.

2. The trace showed a significant number of failures, which also resulted in slightly larger latency.

3. We used AIOps failure correlation to identify the potential component causing the failure, which correlated to a field value of FailedPrecondition.

4. While filtering on that value and reviewing the logs, we still couldn’t understand what this meant.

5. This is where you can use Elastic’s AI Assistant to further your understanding of the issue.

AI Assistant helped us analyze the following:

1. It helped us understand what the log message meant and that it was related to the Redis connection failure issue.

2. Because we couldn’t connect to Redis, we asked the AI Assistant to give us the metrics for the Redis Kubernetes pods.

3. We learned there were two pods for Redis from the logs over the last two hours.

4. However, we also learned that the memory of one seems to be increasing.

5. It seems that Redis restarted (hence the second pod), and with this information we could dive deeper into what happened to Redis.

You can see how quickly we could correlate a significant amount of information, logs, metrics, and traces through the AI Assistant and Elastic’s APM capabilities. We didn’t have to go through multiple screens to hunt down information.

Scenario 2: Analyzing APM error with AI Assistant

Once the SLO is found as noncompliant, we can dive into cart service to investigate in Elastic APM. The following walks through the set of steps you can take in Elastic APM and use the AI Assistant to analyze the issue:

From the video, we can see that once in APM, we took the following steps:

1. We noticed a specific error for the APM service.

2. We investigated this in the error tab, and while we see it’s an issue with connection to Redis, we still need more information.

3. The AI Assistant helps us understand the stacktrace and provides some potential causes for the error and ways to diagnose and resolve it.

4. We also asked it for a runbook, created by our SRE team, which gives us steps to work through this particular issue.

But as you can see, AI Assistant provides us not only with information about the error message but also how to diagnose it and potentially resolve it with an internal runbook.

Achieving operational excellence, optimal performance, and reliability

We’ve shown how an OpenTelemetry instrumented application (OTel demo) can be analyzed using Elastic’s features, especially the AI Assistant coupled with Elastic APM, AIOps, and the latest SLO features. Elastic significantly streamlines the process of identifying and resolving issues within your applications.

Through our detailed walkthrough of two distinct scenarios, we have seen how Elastic APM and the AI Assistant can efficiently analyze and address noncompliance with SLOs in a cart service. The ability to quickly correlate information, logs, metrics, and traces through these tools not only saves time but also enhances the overall effectiveness of the troubleshooting process.

The use of Elastic's AI Assistant in these scenarios underscores the value of integrating advanced AI capabilities into operational workflows. It goes beyond simple error analysis, offering insights into potential causes and providing actionable solutions, sometimes even with customized runbooks. This integration of technology fundamentally changes how SREs approach problem-solving, making the process more efficient and less reliant on manual investigation.

Overall, the advancements in Elastic’s APM, AIOps capabilities, and the AI Assistant, particularly in handling OpenTelemetry data, represent a significant step forward in operational excellence. These tools enable SREs to not only react swiftly to emerging issues but also proactively manage and optimize the performance and reliability of their services, thereby ensuring an enhanced customer experience.

Try it out

Existing Elastic Cloud customers can access many of these features directly from the Elastic Cloud console. Not taking advantage of Elastic on cloud? Start a free trial.

• Build better Service Level Objectives (SLOs) from logs and metrics
• Elastic Observability 8.12: GA for AI Assistant, SLO, and Mobile APM support
• Native Observability support in Elastic Observability
• Context-aware insights using the Elastic AI Assistant for Observability

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

Enhanced troubleshooting for SREs

Site reliability engineers (SRE) in large organizations often face challenges in locating necessary information for troubleshooting alerts, monitoring systems, or deriving insights due to scattered and potentially outdated resources. This issue is particularly significant for less experienced SREs who may require assistance even with the presence of a runbook. Recurring incidents pose another problem, as the on-call individual may lack knowledge about previous resolutions and subsequent steps. Mature SRE teams often invest considerable time in system improvements to minimize "fire-fighting," utilizing extensive automation and documentation to support on-call personnel.

Elastic® addresses these challenges by combining generative AI models with relevant search results from your internal data using RAG. The Observability AI Assistant's internal Knowledge Base, powered by our semantic search retrieval model ELSER, can recall information at any point during a conversation, providing RAG responses based on internal knowledge.

This Knowledge Base can be enriched with your organization's information, such as runbooks, GitHub issues, internal documentation, and Slack messages, allowing the AI Assistant to provide specific assistance. The Assistant can also document and store specific information from an ongoing conversation with an SRE while troubleshooting issues, effectively creating runbooks for future reference. Furthermore, the Assistant can generate summaries of incidents, system status, runbooks, post-mortems, or public announcements.

This ability to retrieve, summarize, and present contextually relevant information is a game-changer for SRE teams, transforming the work from chasing documents and data to an intuitive, contextually sensitive user experience.The Knowledge Base (see requirements) serves as a central repository of Observability knowledge, breaking documentation silos and integrating tribal knowledge, making this information accessible to SREs enhanced with the power of LLMs.

Your LLM provider may collect query telemetry when using the AI Assistant. If your data is confidential or has sensitive details, we recommend you verify the data treatment policy of the LLM connector you provided to the AI Assistant.

In this blog post, we will cover different ways to enrich your Knowledge Base (KB) with internal information. We will focus on a specific alert, indicating that there was an increase in logs with “502 Bad Gateway” errors that has surpassed the alert’s threshold.

How to troubleshoot an alert with the Knowledge Base

Before the KB has been enriched with internal information, when the SRE asks the AI Assistant about how to troubleshoot an alert, the response from the LLM will be based on the data it learned during training; however, the LLM is not able to answer questions related to private, recent, or emerging knowledge. In this case, when asking for the steps to troubleshoot the alert, the response will be based on generic information.

However, once the KB has been enriched with your runbooks, when your team receives a new alert on “502 Bad Gateway” Errors, they can use AI Assistant to access the internal knowledge to troubleshoot it, using semantic search to find the appropriate runbook in the Knowledge Base.

In this blog, we will cover different ways to add internal information on how to troubleshoot an alert to the Knowledge Base:

1. Ask the assistant to remember the content of an existing runbook.

2. Ask the Assistant to summarize and store in the Knowledge Base the steps taken during a conversation and store it as a runbook.

3. Import your runbooks from GitHub or another external source to the Knowledge Base using our Connector and APIs.

After the runbooks have been added to the KB, the AI Assistant is now able to recall the internal and specific information in the runbooks. By leveraging the retrieved information, the LLM could provide more accurate and relevant recommendations for troubleshooting the alert. This could include suggesting potential causes for the alert, steps to resolve the issue, preventative measures for future incidents, or asking the assistant to help execute the steps mentioned in the runbook using functions. With more accurate and relevant information at hand, the SRE could potentially resolve the alert more quickly, reducing downtime and improving service reliability.

Your Knowledge Base documents will be stored in the indices .kibana-observability-ai-assistant-kb-*. Have in mind that LLMs have restrictions on the amount of information the model can read and write at once, called token limit. Imagine you're reading a book, but you can only remember a certain number of words at a time. Once you've reached that limit, you start to forget the earlier words you've read. That's similar to how a token limit works in an LLM.

To keep runbooks within the token limit for Retrieval Augmented Generation (RAG) models, ensure the information is concise and relevant. Use bullet points for clarity, avoid repetition, and use links for additional information. Regularly review and update the runbooks to remove outdated or irrelevant information. The goal is to provide clear, concise, and effective troubleshooting information without compromising the quality due to token limit constraints. LLMs are great for summarization, so you could ask the AI Assistant to help you make the runbooks more concise.

Ask the assistant to remember the content of an existing runbook

The easiest way to store a runbook into the Knowledge Base is to just ask the AI Assistant to do it! Open a new conversation and ask “Can you store this runbook in the KB for future reference?” followed by pasting the content of the runbook in plain text.

The AI Assistant will then store it in the Knowledge Base for you automatically, as simple as that.

Ask the Assistant to summarize and store the steps taken during a conversation in the Knowledge Base

You can also ask the AI Assistant to remember something while having a conversation — for example, after you have troubleshooted an alert using the AI Assistant, you could ask to "remember how to troubleshoot this alert for next time." The AI Assistant will create a summary of the steps taken to troubleshoot the alert and add it to the Knowledge Base, effectively creating runbooks for future reference. Next time you are faced with a similar situation, the AI Assistant will recall this information and use it to assist you.

In the following demo, the user asks the Assistant to remember the steps that have been followed to troubleshoot the root cause of an alert, and also to ping the Slack channel when this happens again. In a later conversation with the Assistant, the user asks what can be done about a similar problem, and the AI Assistant is able to remember the steps and also reminds the user to ping the Slack channel.

After receiving the alert, you can open the AI Assistant chat and test troubleshooting the alert. After investigating an alert, ask the AI Assistant to summarize the analysis and the steps taken to root cause. To remember them for the next time, we have a similar alert and add extra instruction like to warn the Slack channel.

The Assistant will use the built-in functions to summarize the steps and store them into your Knowledge Base, so they can be recalled in future conversations.

Open a new conversation, and ask what are the steps to take when troubleshooting a similar alert to the one we just investigated. The Assistant will be able to recall the information stored in the KB that is related to the specific alert, using semantic search based on ELSER, and provide a summary of the steps taken to troubleshoot it, including the last indication of informing the Slack channel.

Import your runbooks stored in GitHub to the Knowledge Base using APIs or our GitHub Connector

You can also add proprietary data into the Knowledge Base programmatically by ingesting it (e.g., GitHub Issues, Markdown files, Jira tickets, text files) into Elastic.

If your organization has created runbooks that are stored in Markdown documents in GitHub, follow the steps in the next section of this blog post to index the runbook documents into your Knowledge Base.

The steps to ingest documents into the Knowledge Base are the following:

Ingest your organization’s knowledge into Elasticsearch

Option 1: Use the Elastic web crawler . Use the web crawler to programmatically discover, extract, and index searchable content from websites and knowledge bases. When you ingest data with the web crawler, a search-optimized Elasticsearch® index is created to hold and sync webpage content.

Option 2: Use Elasticsearch's Index API . Watch tutorials that demonstrate how you can use the Elasticsearch language clients to ingest data from an application.

Option 3: Build your own connector. Follow the steps described in this blog: How to create customized connectors for Elasticsearch.

Option 4: Use Elasticsearch Workplace Search connectors . For example, the GitHub connector can automatically capture, sync, and index issues, Markdown files, pull requests, and repos.

• Follow the steps to configure the GitHub Connector in GitHub to create an OAuth App from the GitHub platform.

• Now you can connect a GitHub instance to your organization. Head to your organization’s Search > Workplace Search administrative dashboard, and locate the Sources tab.

• Select GitHub (or GitHub Enterprise) in the Configured Sources list, and follow the GitHub authentication flow as presented. Upon the successful authentication flow, you will be redirected to Workplace Search and will be prompted to select the Organization you would like to synchronize.

• After configuring the connector and selecting the organization, the content should be synchronized and you will be able to see it in Sources. If you don’t need to index all the available content, you can specify the indexing rules via the API. This will help shorten indexing times and limit the size of the index. See Customizing indexing.

• The source has created an index in Elastic with the content (Issues, Markdown Files…) from your organization. You can find the index name by navigating to Stack Management > Index Management , activating the Include hidden Indices button on the right, and searching for “GitHub.”

• You can explore the documents you have indexed by creating a Data View and exploring it in Discover. Go to Stack Management > Kibana > Data Views > Create data view and introduce the data view Name, Index pattern (make sure you activate “Allow hidden and system indices” in advanced options), and Timestamp field:

• You can now explore the documents in Discover using the data view:

Reindex your internal runbooks into the AI Assistant’s Knowledge Base Index, using it's semantic search pipeline

Your Knowledge Base documents are stored in the indices .kibana-observability-ai-assistant-kb-*. To add your internal runbooks imported from GitHub to the KB, you just need to reindex the documents from the index you created in the previous step to the KB’s index. To add the semantic search capabilities to the documents in the KB, the reindex should also use the ELSER pipeline preconfigured for the KB, .kibana-observability-ai-assistant-kb-ingest-pipeline.

By creating a Data View with the KB index, you can explore the content in Discover.

You execute the query below in Management > Dev Tools , making sure to replace the following, both on “_source” and “inline”:

• InternalDocsIndex : name of the index where your internal docs are stored
• text_field : name of the field with the text of your internal docs
• timestamp : name of the field of the timestamp in your internal docs
• public : (true or false) if true, makes a document available to all users in the defined Kibana Space (if is defined) or in all spaces (if is not defined); if false, document will be restricted to the user indicated in
• (optional) space : if defined, restricts the internal document to be available in a specific Kibana Space
• (optional) user.name : if defined, restricts the internal document to be available for a specific user
• (optional) "query" filter to index only certain docs (see below)

POST _reindex
{
    "source": {
        "index": "<InternalDocsIndex>",
        "_source": [
            "<text_field>",
            "<timestamp>",
            "namespace",
            "is_correction",
            "public",
            "confidence"
        ]
    },
    "dest": {
        "index": ".kibana-observability-ai-assistant-kb-000001",
        "pipeline": ".kibana-observability-ai-assistant-kb-ingest-pipeline"
    },
    "script": {
        "inline": "ctx._source.text=ctx._source.remove(\"<text_field>\");ctx._source.namespace=\"<space>\";ctx._source.is_correction=false;ctx._source.public=<public>;ctx._source.confidence=\"high\";ctx._source['@timestamp']=ctx._source.remove(\"<timestamp>\");ctx._source['user.name'] = \"<user.name>\""
    }
}

You may want to specify the type of documents that you reindex in the KB — for example, you may only want to reindex Markdown documents (like Runbooks). You can add a “query” filter to the documents in the source. In the case of GitHub, runbooks are identified with the “type” field containing the string “file,” and you could add that to the reindex query like indicated below. To add also GitHub Issues, you can also include in the query “type” field containing the string “issues”:

"source": {
        "index": "<InternalDocsIndex>",
        "_source": [
            "<text_field>",
            "<timestamp>",
            "namespace",
            "is_correction",
            "public",
            "confidence"
        ],
    "query": {
      "terms": {
        "type": ["file"]
      }
    }

Great! Now that the data is stored in your Knowledge Base, you can ask the Observability AI Assistant any questions about it:

Conclusion

In conclusion, leveraging internal Observability knowledge and adding it to the Elastic Knowledge Base can greatly enhance the capabilities of the AI Assistant. By manually inputting information or programmatically ingesting documents, SREs can create a central repository of knowledge accessible through the power of Elastic and LLMs. The AI Assistant can recall this information, assist with incidents, and provide tailored observability to specific contexts using Retrieval Augmented Generation. By following the steps outlined in this article, organizations can unlock the full potential of their Elastic AI Assistant.

Start enriching your Knowledge Base with the Elastic AI Assistant today and empower your SRE team with the tools they need to excel. Follow the steps outlined in this article and take your incident management and alert remediation processes to the next level. Your journey toward a more efficient and effective SRE operation begins now.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

• Managing fundamental AWS services such as Amazon EC2, Amazon RDS, Amazon VPC, and NAT gateway
• Data can be ingested into Elastic observability using a serverless forwarder or Amazon Kinesis Data Firehose
• Ingesting and analyzing AWS VPC Flow logs

One of the more heavily used AWS container services is Amazon ECS (Elastic Container Service). While there is a trend toward using Fargate to simplify the setup and management of ECS clusters, many users still prefer using Amazon ECS with EC2 instances. It may not be as straightforward or efficient as AWS Fargate, but it offers more control over the underlying infrastructure.

In the most recent blog, we explored how Elastic Observability helps manage Amazon ECS with Fargate. However, this blog will review how to manage an Amazon ECS cluster with EC2 instances using Elastic Observability instead.

In general, when setting up Amazon ECS-based clusters with EC2, you may or may not have access to the EC2 instances. This determines what you can use with Elastic Observability in monitoring your EC2-based ECS cluster. Hence, there are two components you can use in monitoring the EC2-based ECS cluster with Elastic Observability:

As you can see in the diagram above, the two components are:

1. Baseline setup __ : The elastic agent running the AWS integration is configured to obtain ECS metrics and logs from cloud watch. This agent runs on an instance that is not part of the ECS cluster because it allows you to see ALL ECS clusters and other AWS Services, such as EKS, RDS, and EC2.

2. Additional setup: If you have access to the EC2 instances in the ECS cluster, then you can run Elastic’s docker integration in each EC2 instance. This gives you significantly more details on the containers than AWS container insights. And it does not require AWS Cloudwatch, which can be fairly costly.

Using either just the baseline or the additional setup, you will have to set up AWS CloudWatch Container Insights for the ECS cluster. However, the docker integration with the additional setup can provide additional information to the AWS CloudWatch Container Insights.

Hence, we will review how you can monitor the various components of an EC2-based ECS cluster:

• EC2 instances in the ASG group
• ECS services running in the ECS cluster
• ECS tasks (containers)

Also, we will review how you can obtain metrics and logs from the ECS cluster with and without AWS Cloudwatch. We’ll show you how to use:

• AWS CloudWatch Container Insights (from Cloudwatch)
• Docker metrics (non-Cloudwatch)
• Amazon ECS logs via Cloudwatch

Prerequisites and configuration

If you plan on following this blog, here are some of the components and details we used to set up the configuration:

• An account on Elastic Cloud and a deployed stack (see instructions here) — ensure that you have both.
• A nginx container and a stress container — we will use these two basic containers to help highlight the load on the Elastic ECS Cluster.
• An ECS EC2 Cluster in an Auto Scaling Group — ensure you have access in order to load up the Elastic agent on the EC2 instances, or you can create an AMI and use that as the baseline image for your ECS cluster.
• An EC2 instance anywhere in your account that is not part of the ECS cluster and has public access (to send metrics and logs)

What will you see in Elastic Observability once it's all set up?

If you utilize the baseline configuration with ECS EC2 cluster configured with AWS CloudWatch Container Insights configured, the Elastic Agent configured with the following Elastic agent integrations:

• ECS integration
• EC2 integration
• AWS Cloudwatch Integration with metrics and logging

Then you will be able to get the following information in Elastic dashboards:

• Containers in the cluster (AWS CloudWatch Container Insights via Elastic Agent and AWS Cloudwatch integration)
• Services in the cluster (AWS CloudWatch Container Insights via Elastic Agent and AWS Cloudwatch integration)
• CPU and memory utilization of the ECS Cluster (Elastic Agent with ECS integration)
• EC2 CPU and memory utilization of the instance in the cluster (Elastic Agent with EC2 integration)
• CPU and memory utilization per container (via AWS CloudWatch Container Insights via Elastic Agent and AWS Cloudwatch integration)

If the additional configuration using Elastic agents with docker integration per ECS EC2 instance is used, you will be able to get a direct feed of metrics via docker. The following metrics can be viewed:

Let’s see how to set this all up.

Setting it all up

Over the next few steps, I’ll walk through:

• Getting an account on Elastic Cloud
• Bringing up an ECS EC2 cluster and potentially setting up your own AMI
• Setting up the containers nginx and a stress container
• Setting up the Elastic agent with docker container integration on the ECS EC2 instances
• Setting up the Elastic agent with AWS, Cloudwatch, and ECS integrations on an independent EC2 instance

Step 1: Create an account on Elastic Cloud

Follow the instructions to get started on Elastic Cloud.

Step 2: Set up an ECS Cluster with EC2 instances

When creating a cluster, you have two options when setting it up using the console:

• Create a new ASG group where you will only be allowed to use the preloaded set of Amazon Linux (2 or 2023) based AMIs
• Set up your own ASG Cluster prior to setting up the ECS Cluster and select this from the options. This option will give you more control over what Linux version and the ability to add things like Elastic agents in the AMI used for the instances in the ASG.

Regardless of either option, you will need to turn on Container Insights (see the bottom part of the image below).

Once the cluster is setup, you can go to AWS Cloudwatch where you should see Container Insights for your cluster:

Step 3: Set up Elastic agent with docker integration

Next, you will need to add an Elastic agent to each one of the instances. In the Elastic cloud, set up an Elastic policy with the docker and system integrations as such:

Next, add an agent for the policy, then copy the appropriate install script (in our case it was Linux since we were running Amazon Linux 2), and run it on every EC2 instance in the cluster:

Once this is added you should see agents in the fleet. Each agent will be on each EC2 instance:

If you decide to set up an ECS EC2 cluster with your own ASG and don’t use Amazon Lunix AMIs (2 or 2023 version), you will have to:

• Pick your base image to base an AMI on
• Add an ECS agent and register each instance to the AMI base image manually
• Add the Elastic agent — standalone version — this step will require you to configure your Elastic endpoint and API key (or simply add the script in the “add agent” part of the configuration above when using the UI)
• Create the AMI once all the above components are added
• Use the newly created AMI in creating the ASG for ECS cluster

Step 4: Set up an Elastic agent with the AWS integration

From the integrations tab in Elastic Cloud, select AWS integration and select add agent. You will then have to walk through the configuration of the AWS integration.

At a minimum, ensure that you have the following configuration options turned on:

This will ensure that not only EC2 metrics and logs are ingested but that all CloudWatch metrics and logs are also ingested. ECS metrics and logs are stored in CloudWatch.

If you want to ensure only logs from the specific ECS cluster are ingested, you can also restrict what to ingest by several parameters. In our setup, we are collecting only logs from Log Group with a prefix of /aws/ecs/containerinsights/EC2BasedCluster/.

Once this policy is set up, add an agent like in Step 1.

However, this agent needs to be added to an EC2 instance which is independent of the ECS cluster.

Once installed, this agent will help pull in:

• All EC2 instance metrics across your account (which can be adjusted in the integration policy)
• Ingest AWS CloudWatch Container Insights from ECS
• ECS metrics such as:
• aws.ecs.metrics.CPUReservation.avg
• aws.ecs.metrics.CPUUtilization.avg
• aws.ecs.metrics.GPUReservation.avg
• aws.ecs.metrics.MemoryReservation.avg
• aws.ecs.metrics.MemoryUtilization.avg
• More - see the full list here

Step 5: Setting up services and containers

In running this configuration, we used nginx and a stress container before we go into the task.

In order to initiate service and containers on ECS, you will need to set up a task for each of these containers. But more importantly, you will need to ensure that the roles for both of the following:

"taskRoleArn": "arn:aws:iam::xxxxx:role/ecsTaskExecutionRol"executionRoleArn":,

"arn:aws:iam::xxxxx:role/ecsTaskExecutionRole",

have the following permissions:

Most importantly, you should ensure that this permission is added:

AmazonEC2ContainerServiceforEC2Role

It will ensure containers can be brought up on the EC2 instances in the cluster.

Once you have the right permissions, then set up the following tasks.

Here is the task JSON for NGINX:

{
  "family": "NGINX",
  "containerDefinitions": [
    {
      "name": "nginx",
      "image": "nginx: latest",
      "cpu": 0,
      "portMappings": [
        {
          "name": "nginx-80-tcp",
          "containerPort": 80,
          "hostPort": 80,
          "protocol": "tcp",
          "appProtocol": "http"
        }
      ],
      "essential": true,
      "environment": [],
      "environmentFiles": [],
      "mountPoints": [],
      "volumesFrom": [],
      "ulimits": [],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-create-group": "true",
          "awslogs-group": "/ecs/",
          "awslogs-region": "us-west-2",
          "awslogs-stream-prefix": "ecs"
        },
        "secretOptions": []
      }
    }
  ],
  "taskRoleArn": "arn:aws:iam::xxxxxx:role/ecsTaskExecutionRole",
  "executionRoleArn": "arn:aws:iam::xxxxx:role/ecsTaskExecutionRole",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["EC2"],
  "cpu": "256",
  "memory": "512",
  "runtimePlatform": {
    "cpuArchitecture": "X86_64",
    "operatingSystemFamily": "LINUX"
  }
}

Here is the task JSON for stress container:

{
  "family": "stressLoad",
  "containerDefinitions": [
    {
      "name": "stressLoad",
      "image": "containerstack/alpine-stress",
      "cpu": 0,
      "memory": 512,
      "memoryReservation": 512,
      "portMappings": [],
      "essential": true,
      "entryPoint": ["sh", "-c"],
      "command": [
        "/usr/local/bin/stress --cpu 2 --io 2 --vm 1 --vm-bytes 128M --timeout 6000s"
      ],
      "environment": [],
      "mountPoints": [],
      "volumesFrom": [],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-create-group": "true",
          "awslogs-group": "/ecs/",
          "awslogs-region": "us-west-2",
          "awslogs-stream-prefix": "ecs"
        }
      }
    }
  ],
  "taskRoleArn": "arn:aws:iam::xxxxx:role/ecsTaskExecutionRole",
  "executionRoleArn": "arn:aws:iam::xxxxx:role/ecsTaskExecutionRole",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["EC2"],
  "cpu": "256",
  "memory": "512",
  "runtimePlatform": {
    "cpuArchitecture": "X86_64",
    "operatingSystemFamily": "LINUX"
  }
}

Once you have defined the tasks, ensure you bring up each service (one for each task) with the launch type of EC2:

You should have two services running now.

Step 6: Check on metrics and logs in Elastic Cloud

Go to Elastic Cloud and ensure that you are getting metrics and logs from the ECS Cluster. First, check to see if you are receiving metrics by viewing the built-in dashboard called [Metrics Docker] Overview.

With some work on this dashboard by adding in container insight metrics and docker metrics, you should be able to see:

If you only have the ECS integration and the Elastic agent in Step 2, then you will need to create a new dashboard:

This dashboard can be set up with the following metrics:

• Containers in the cluster (containerInsights via Elastic Agent and AWS Cloudwatch integration). Set up a TSVB panel using the following metric: aws.dimensions.ClusterName : "EC2BasedCluster" with aws.containerinsights.metrics.TaskCount.max
• Services in the cluster (containerInsights via Elastic Agent and AWS Cloudwatch integration). Use the following configuration to setup the chart:

• CPU and memory utilization of the ECS Cluster (Elastic Agent with ECS integration). Use the following configuration to set up both CPU and memory utilization charts:

• EC2 CPU and storage utilization of the instance in the cluster (Elastic Agent with EC2 integration). Use the following configuration to set up both CPU and memory utilization charts:

• (Not shown): CPU and memory utilization per container (via containerInsights via Elastic Agent and AWS Cloudwatch integration)

Step 7: Look at logs from your ECS cluster

Since we set up AWS CloudWatch logs collection in Step 2, we can view these logs in Discover by filtering on the logs group arn /aws/ecs/containerinsights/EC2BasedCluster/.

Summary

I hope you’ve gotten an appreciation for how Elastic Observability can help your AWS monitoring ECS service metrics. Here’s a quick recap of lessons and what you learned:

• Elastic Observability supports ingesting and analysis of AWS ECS service metrics and the corresponding EC2 metrics through the AWS integration on the Elastic Agent. It’s easy to set up ingest from AWS Services via the Elastic Agent.
• Elastic Observability can also get container metrics via the Docker integration running on Elastic agents on each of the EC2 instances in the ECS EC2 auto scaling group.
• Elastic has multiple out-of-the-box (OOTB) AWS service dashboards that can be used as baselines to get your own customized view.

Ready to get started? Start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.